It's been no secret that I think the lack of soft skills in the security space is one of our biggest problems. While usually I usually only write all about the world's problems and how to fix them here, during RSA I'm going to take a somewhat different approach.
I'm giving a talk on Friday titled Why Won't Anyone Listen to Us?
I'm going to talk about how a security person can talk to a normal person without turning them against us. We're a group that doesn't like talking to anyone, even each other. We need to start talking to people. I'm not saying we should stand around and accept abuse, I am saying the world wants help with security. We're not really in a place to give it because we don't like people. But they need our help, most of them know it even!
We've all had countless interactions where we give someone good, real advice, and they just completely ignore us. It's infuriating sometimes. Part of the problem is we're not talking to people, we're throwing verbal information at them, and they ignore it. They listen to Oprah, if she told them about two factor auth everyone would be using it by the end of the week!
That's just it, they listen to Oprah. They're going to listen to anyone who talks to them in a way they understand. If it's not us, it will be someone else, probably someone we don't want talking about security.
I can't teach you to like people (there are limits to my abilities), but I can help teach you how to talk to them. And of course a talk like this will need to have plenty of fun sprinkled in. How to talk to someone while being very important, can also be an extremely boring topic.
I touched on some of this during my Devconf.cz talk.
Red Hat is also putting on a Breakfast on Wednesday morning. I'm going to keynote it (I'll keep it short and sweet for those of you attending, there's nothing worse than a speaker at 8am who talks too much).
A coworker, Richard Morrell, is running a podcast from RSA called Locked Down. Be sure to give it a listen. I may even be able to convince him to let me on his show.
I have no doubt the RSA conference will be a great time. If you're there come find me, Red Hat has a booth, North Expo #N3038. Come say hi, or not if you don't like talking to people ;)
There or not, feel free to start a conversation on Twitter. I'm @joshbressers
Sunday, February 28, 2016
Tuesday, February 23, 2016
Thinking about glibc and Heartbleed, how do fix things
After my last blog post Change direction, increase speed! (or why glibc changes nothing) it really got me thinking about how can we start to fix some of this. The sad conclusion is that nothing can be fixed in the short term. Rather than trying to make up some nonsense about how to fix this, I want to explain what's happening and why this can't be fixed anytime soon.
Let's look at Heartbleed first.
There was a rather foul flaw found in OpenSSL, after Heartbleed the Linux Foundation collected a lot of money to help work on core infrastructure projects. If we look at the state of things it basically hasn't changed outside of money moving around. OpenSSL cannot be fixed for a number of reason.
Let's look at Heartbleed first.
There was a rather foul flaw found in OpenSSL, after Heartbleed the Linux Foundation collected a lot of money to help work on core infrastructure projects. If we look at the state of things it basically hasn't changed outside of money moving around. OpenSSL cannot be fixed for a number of reason.
- Old codebase
- Backward compatibility
- Difficult API
- It is "general purpose"
The reality is that the only way to get what could be considered a safe library, you would have to throw everything out and start over with some very specific ideas in mind. Things like sun-setting algorithms didn't exist when OpenSSL was created. There is no way you're going to get even a small number of projects to move from using OpenSSL to some new "better" library. It would have to be so much better they couldn't ignore it. As anyone who has ever written software knows, you don't build a library like that over night. I think 5 years would be a conservative estimate for double digit adoption rates.
While I'm picking on OpenSSL here, the story is the same in virtually every library and application that exists. OpenSSL isn't special, it just gets a lot of attention.
Let's think about glibc.
Glibc is the C library used by most Linux distributions. If the Kernel is the heart, glibc is the soul. Nothing can even exist without this library. Glibc even strives to be POSIX compliant, for good reason. POSIX has given us years of compatibility and cooperation.
Glibc probably has plenty more horrible bugs hiding in the code. It's wicked complex and really large. If you ever need some nightmare fuel, look at the glibc source code. Everything we do in C relies on a libc being around, glibc doesn't have that luxury.
Replacing libc is probably out of the question, it's just not something practical. So let's think about something like golang. What if everything was written using golang? It's not totally insane, there are substantial benefits. It's not as fast as C though, that will be the argument most use. Golang will probably never beat C, the things that makes it safer also makes it slower. But now if we think about replacing UNIX utilities with golang, why would we want to do that? Why not throw out all the mistakes UNIX made and do something else?
Now we're back to the legacy and compatibility arguments. Linux has more than twenty years of effort put into it. You can't just replace that. Even if you had the best team in the world I bet 10 years would be wishful thinking for having half the existing features.
So what does this mean? It means we don't know where to start yet. We are trying to solve our problems using what we know and the tools that exist. We have to solve this using new tools and new thinking. The way we fix security is by doing something nobody has ever thought of before. In 1920 the concept of the Internet didn't exist, people couldn't imagine how to even solve some of the problems we can easily solve using it. Don't try to solve our problems with what you know. Solve the problems using new ideas, find out what you don't know, that's where the solution lives.
Sunday, February 21, 2016
Change direction, increase speed! (or why glibc changes nothing)
The glibc issue has had me thinking. What will we learn from this?
I'm pretty sure the answer is "nothing", which then made me wonder why this is.
The conclusion I came up with is we are basically the aliens from space invaders. Change direction, increase speed! While this can give the appearance of doing something, we are all very busy all the time. It's not super useful when you really think about it. Look at Shellshock, Heartbleed, GHOST, LOGJAM, Venom, pick an issue with a fancy name. After the flurry of news stories and interviews, did anything change, or did everyone just go back to business as usual? Business as usual pretty much.
Dan Kaminski explains glibc nicely and has some advice. But let's look at this honestly. Is anything going to change? No. Dan found a serious DNS issue back in the day. Did we fix DNS or did we bandage it up as best as we could? We bandaged it. What Dan found was without a doubt as bad or worse than this glibc issue, nothing changed.
I've said this before, I'll say it again. I'm going to say it at RSA next week. We don't know how to fix this. We think we do, you're thinking about it right now, about how we can fix everything! We just have to do that one ... Except you can't. We don't really know what's wrong. Security bugs aren't the problem, they are the result of the problem. We can't fix all the security bugs. I'd be surprised if we've even fixed 10% of security bugs that exist. Even mitigation technologies aren't going to get us there (they are better than constantly fixing bugs, but that's a story for another day).
It's like being obsessed about your tire pressure when there is a hole in the tire. If you only worry about one detail, the tunnel vision makes you miss what's actually going on. Our tires are losing air faster than we can fix them, so we're looking for a bigger pump instead of new tires.
We say things all the time about not using C anymore, or training developers, or writing better documentation. There's nothing wrong with these ideas exactly, but the fact is they've all been tried more than once and none of them work. If we started the largest developer education program ever, if we made every developer sit through a week of training. I bet it would be optimistic if our bug rate would decrease by 5%. Think about that for a while.
We first have to understand our problem. We have lots of solutions, solutions to problems that don't really exist. Solutions without problems tend to turn into new problems. We need to understand our security problem. It's probably more like hundreds or thousands of problems. Every group, every company, every person has different problems. We understand none of them.
We start by listening. We're not going to fix any of this with code. We need to see what's happening, some big picture, some in the weeds. Today we show up, yell at people (if they're lucky), then we leave. We don't know what's really happening. We don't tell anyone what they need to know. We don't even know what they need to know. The people we're not talking to know what the problems are though. They don't know they know, we just have to give them time to explain it to us.
If you're at RSA next week, come talk to me. If not, hit me up on twitter @joshbressers
I'm pretty sure the answer is "nothing", which then made me wonder why this is.
The conclusion I came up with is we are basically the aliens from space invaders. Change direction, increase speed! While this can give the appearance of doing something, we are all very busy all the time. It's not super useful when you really think about it. Look at Shellshock, Heartbleed, GHOST, LOGJAM, Venom, pick an issue with a fancy name. After the flurry of news stories and interviews, did anything change, or did everyone just go back to business as usual? Business as usual pretty much.
Dan Kaminski explains glibc nicely and has some advice. But let's look at this honestly. Is anything going to change? No. Dan found a serious DNS issue back in the day. Did we fix DNS or did we bandage it up as best as we could? We bandaged it. What Dan found was without a doubt as bad or worse than this glibc issue, nothing changed.
I've said this before, I'll say it again. I'm going to say it at RSA next week. We don't know how to fix this. We think we do, you're thinking about it right now, about how we can fix everything! We just have to do that one ... Except you can't. We don't really know what's wrong. Security bugs aren't the problem, they are the result of the problem. We can't fix all the security bugs. I'd be surprised if we've even fixed 10% of security bugs that exist. Even mitigation technologies aren't going to get us there (they are better than constantly fixing bugs, but that's a story for another day).
It's like being obsessed about your tire pressure when there is a hole in the tire. If you only worry about one detail, the tunnel vision makes you miss what's actually going on. Our tires are losing air faster than we can fix them, so we're looking for a bigger pump instead of new tires.
We say things all the time about not using C anymore, or training developers, or writing better documentation. There's nothing wrong with these ideas exactly, but the fact is they've all been tried more than once and none of them work. If we started the largest developer education program ever, if we made every developer sit through a week of training. I bet it would be optimistic if our bug rate would decrease by 5%. Think about that for a while.
We first have to understand our problem. We have lots of solutions, solutions to problems that don't really exist. Solutions without problems tend to turn into new problems. We need to understand our security problem. It's probably more like hundreds or thousands of problems. Every group, every company, every person has different problems. We understand none of them.
We start by listening. We're not going to fix any of this with code. We need to see what's happening, some big picture, some in the weeds. Today we show up, yell at people (if they're lucky), then we leave. We don't know what's really happening. We don't tell anyone what they need to know. We don't even know what they need to know. The people we're not talking to know what the problems are though. They don't know they know, we just have to give them time to explain it to us.
If you're at RSA next week, come talk to me. If not, hit me up on twitter @joshbressers
Thursday, February 18, 2016
glibc for humans
Unless you've been living under a rock, you've heard about the latest glibc issue.
CVE-2015-7547 - glibc stack-based buffer overflow in getaddrinfo()
It's always hard to understand some of these issues, so I'm going to do my best to explain it using simple language. Making security easy to understand is something I've been talking about for a long time now, it's time to do something about it.
What is it?
The fundamental problem here is that glibc has a bug that could allow a DNS response from an attacker to run the command of that attacker's choosing on your system. The final goal of course would be to become the root user.
The problem is that this glibc function is used by almost everything that talks to the network. In today's hyperconnected world, this means basically everything is vulnerable to this bug because almost everything can connect to the network. As of this writing we have not seen this attack being used on the Internet. Just because there are no known attacks is no reason to relax though, constant vigilance is key for issues like this.
Am I vulnerable?
If you run Linux (most distributions use glibc), and you haven't installed an update from your vendor, yes, you are vulnerable.
Are there workarounds?
No, there is no way to stop this issue. You have to install an update to glibc. Even the stack protector technology that is built into gcc and glibc will not stop this bug. While it is a stack overflow bug, the stack protector checks do not run before the exploit would gain control.
What about containers, VMs, or other confinement technology?
It is possible that a container, VM, or other technology such as SELinux could limit the possible damage from this bug. However it affects so many binaries on the system it should be expected that an attacker able to gain access to one applications could continue to exploit this bug to eventually become root and take over the entire machine.
Do I only need to be worried if I run a webserver or mailserver?
As stated previously, this bug affects virtually everything that talks to the network. Even if you think your webserver or mailserver are safe, everything from bash to your ssh client will use this library. Updating glibc is the only way to ensure you'll be OK.
What if I run my own DNS server?
This point is currently under investigation. It is thought that it may be possible for a bad DNS request to be able to make it through a DNS server to a vulnerable host. Rather than find out, you should update your glibc.
What about ...
No, just update your glibc :)
Do you have other questions? Ask me on twitter and I'll be sure to update this article if I know the answer.
@joshbressers
CVE-2015-7547 - glibc stack-based buffer overflow in getaddrinfo()
It's always hard to understand some of these issues, so I'm going to do my best to explain it using simple language. Making security easy to understand is something I've been talking about for a long time now, it's time to do something about it.
What is it?
The fundamental problem here is that glibc has a bug that could allow a DNS response from an attacker to run the command of that attacker's choosing on your system. The final goal of course would be to become the root user.
The problem is that this glibc function is used by almost everything that talks to the network. In today's hyperconnected world, this means basically everything is vulnerable to this bug because almost everything can connect to the network. As of this writing we have not seen this attack being used on the Internet. Just because there are no known attacks is no reason to relax though, constant vigilance is key for issues like this.
Am I vulnerable?
If you run Linux (most distributions use glibc), and you haven't installed an update from your vendor, yes, you are vulnerable.
Are there workarounds?
No, there is no way to stop this issue. You have to install an update to glibc. Even the stack protector technology that is built into gcc and glibc will not stop this bug. While it is a stack overflow bug, the stack protector checks do not run before the exploit would gain control.
What about containers, VMs, or other confinement technology?
It is possible that a container, VM, or other technology such as SELinux could limit the possible damage from this bug. However it affects so many binaries on the system it should be expected that an attacker able to gain access to one applications could continue to exploit this bug to eventually become root and take over the entire machine.
Do I only need to be worried if I run a webserver or mailserver?
As stated previously, this bug affects virtually everything that talks to the network. Even if you think your webserver or mailserver are safe, everything from bash to your ssh client will use this library. Updating glibc is the only way to ensure you'll be OK.
What if I run my own DNS server?
This point is currently under investigation. It is thought that it may be possible for a bad DNS request to be able to make it through a DNS server to a vulnerable host. Rather than find out, you should update your glibc.
What about ...
No, just update your glibc :)
Do you have other questions? Ask me on twitter and I'll be sure to update this article if I know the answer.
@joshbressers
Sunday, February 7, 2016
Devconf.cz
I spent last week at Devconf in the Czech Republic. I didn't have time to write anything new and compelling, but I did give a talk about why everything seems to be on fire.
https://www.youtube.com/watch?v=zmDm7J7V7aw
I explore what's going on right now, why do things look like they're on fire, and how we can start to fix this. Our problem isn't technology, it's the people. We're good at technology problems, we're bad at people problems.
Give the talk a listen. Let me know what you think, I hope to peddle this message as far and wide as possible.
Join the conversation, hit me up on twitter, I'm @joshbressers
https://www.youtube.com/watch?v=zmDm7J7V7aw
I explore what's going on right now, why do things look like they're on fire, and how we can start to fix this. Our problem isn't technology, it's the people. We're good at technology problems, we're bad at people problems.
Give the talk a listen. Let me know what you think, I hope to peddle this message as far and wide as possible.
Join the conversation, hit me up on twitter, I'm @joshbressers
Subscribe to:
Posts (Atom)