If you've managed to find this blog and are wondering where the updates are, it's found a new home. You can find the new posts here
https://opensourcesecurity.io/blog/
You can find the RSS feed here
https://opensourcesecurity.io/feed/
As always, feel free to hit me up on Twitter with questions or comments
@joshbressers
Tuesday, February 6, 2018
Sunday, September 24, 2017
Measuring security: Part 2 - The cost of doing business
If you've not read my last post on measuring security you probably should. It talks about how to measure the security of things that make money. That post is mostly focused on things like products that directly generate revenue. This time we're going to talk about a category I'm calling the cost of doing business.
The term "cost of doing business" is something I made up so I could group these ideas in some sensible way. At least sensible to me. You probably can't use this with other humans in a discussion, they won't know what you're talking about. If I had a line graph of spending I would put revenue generating on one side, the purse cost centers on the other side. The cost of doing business is somewhere in the middle. These are activities that directly support whatever it is the organization does to make new money. Projects and solutions that don't directly make money themselves but do directly support things being built that make money.
The cost of doing business includes things like compliance, sending staff to meetings, maybe regulatory requirements. Things that don't directly generate revenue but you can't move forward if you don't do these things. There's not a lot of options in many cases. If you don't have PCI compliance, you can't process payments, you can't make any money, and the company won't last long. If you don't attend certain meetings nobody can get any work done. Regulated industry must follow their requirements or the company can often just be shut down. Sometimes there are things we have to do, even if we don't want to do them.
In the next post we'll talk about what I call "infrastructure", these are the things that are seen as cost centers and often a commodity service (like electricity or internet access). I just want to clarify the difference. Infrastructure is something where you have choice or can decide not to do it with a possible negative (or positive) consequence. Infrastructure is what keep the lights on at a bare minimum. Cost of doing business must be done to get yourself to the next step in a project, there is no choice, which changes what we measure and how we measure it.
The Example
Let's pick on PCI compliance as it's pretty easy to understand example. If you don't do this it's quite likely your company won't survive, assuming you need to process card payments. If you're building a new web site that will process payments, you have to get through PCI compliance, there is no choice, and the project cannot move forward until this is complete. The goal now isn't so much measuring the return on an investment as it is being a good steward of the resources given to us. PCI requirements and audits are not cheap. If you are seen as making poor decisions and squandering your resources it's quite likely the business will get grumpy with you.
Compliance and security aren't the same thing. There is some overlap but it must be understood that you can be compliant and still get hacked. The overlap of compliance is a great thing to focus on for measuring what we do. Did your compliance program make you more secure? Can you show how another group used a compliance requirement to make something better? What if something compliance required saved some money on how the network was architected? There are a lot of side benefits to pay attention to. Make sure you note the things that are improvements, even if they aren't necessarily security improvements.
I've seen examples where compliance was used to justify 2 factor authentication (2FA) in an organization, There are few things more powerful than 2FA that you can deploy. Showing compliance helped move an initiative like this forward, and also showing how the number of malicious logs drops substantially is a powerful message. Just turning on 2FA isn't enough. Make sure you show why it's better, how the attacks are slowed or stopped. Make sure you can show there were few issues for users (the people who struggle will complain loudly). If there is massive disruption for your users, figure out why you didn't know this would happen, someone screwed something up that means. It's important to measure the good and the bad. We rarely measure failure which is a problem. Nobody has a 100% success rate, learn from your failure.
What about attending a meeting or industry conference? Do you just go, file the expense report, and do nothing? That sounds like a waste of time and money. Make sure you have concrete actions. Write down what happened, why it was important you were there, how you made the situation better, and what you're going to do next. How did the meeting move your project forward? Did you learn something new, or make some plans that will help in the future? Make sure the person paying your bills sees this. Make them happy to be providing you the means to keep the business moving forward.
The Cost
The very first step we have to consider when we want to measure what we're doing is to do your homework and understand cost. Not just upfront cost but cost of machines, disk, people, services, anything you need to keep the business moving forward. If there are certain requirements needed for a solution make sure you understand and document it. If a certain piece of software or service has to be used show why. Show what part of the business can function because of the cost you're providing. Remember this is going to be specific requirements you can't escape. These are not commodity services and solutions. And of course the goal is to move forward.
If you inherit an existing solution take a good look at everything, make sure you know exactly what the resource cost of the solution is. The goal here isn't always to show a return on investment, but to show that the current solution makes sense. Just because something costs less money doesn't mean it's cheaper. If your cut rate services will put the project in jeopardy you're going to be in trouble someday. Be able to show this is a real threat. It's possible a decision will be made to take on this threat, but that's not always your choice. Always be able to answer the questions "if we do this what happens" and "if we don't do this what happens".
Conclusion
This topic is tricky. I keep thinking about it and even as I wrote this post it changed quite a lot from what I started to write. If you have something that makes money it's easy to justify investment. If you have something that's a pure cost center it's easy to minimize cost. This middle ground is tricky. How do you show value for something you have to do but isn't directly generating revenue? If you work for a forward looking business you probably won't have to spend a ton of time getting these projects funded. Growing companies understand the cost of doing business.
I have seen some companies that aren't growing as quickly fail to see value in the cost of doing business. There's nothing wrong with this sometimes, but as a security leader your job is to make your leadership understand what isn't happening because of this lack of investment. Sometimes if you keep a project limping along, barely alive, you end up causing a great deal of damage to the project and your staff. If leadership won't fund something, it means they don't view it as important and neither should you. If you think it is important, you need to sell it to your leadership. Sometimes you can't and won't win though, and then you have to be willing to let it go.
The term "cost of doing business" is something I made up so I could group these ideas in some sensible way. At least sensible to me. You probably can't use this with other humans in a discussion, they won't know what you're talking about. If I had a line graph of spending I would put revenue generating on one side, the purse cost centers on the other side. The cost of doing business is somewhere in the middle. These are activities that directly support whatever it is the organization does to make new money. Projects and solutions that don't directly make money themselves but do directly support things being built that make money.
The cost of doing business includes things like compliance, sending staff to meetings, maybe regulatory requirements. Things that don't directly generate revenue but you can't move forward if you don't do these things. There's not a lot of options in many cases. If you don't have PCI compliance, you can't process payments, you can't make any money, and the company won't last long. If you don't attend certain meetings nobody can get any work done. Regulated industry must follow their requirements or the company can often just be shut down. Sometimes there are things we have to do, even if we don't want to do them.
In the next post we'll talk about what I call "infrastructure", these are the things that are seen as cost centers and often a commodity service (like electricity or internet access). I just want to clarify the difference. Infrastructure is something where you have choice or can decide not to do it with a possible negative (or positive) consequence. Infrastructure is what keep the lights on at a bare minimum. Cost of doing business must be done to get yourself to the next step in a project, there is no choice, which changes what we measure and how we measure it.
The Example
Let's pick on PCI compliance as it's pretty easy to understand example. If you don't do this it's quite likely your company won't survive, assuming you need to process card payments. If you're building a new web site that will process payments, you have to get through PCI compliance, there is no choice, and the project cannot move forward until this is complete. The goal now isn't so much measuring the return on an investment as it is being a good steward of the resources given to us. PCI requirements and audits are not cheap. If you are seen as making poor decisions and squandering your resources it's quite likely the business will get grumpy with you.
Compliance and security aren't the same thing. There is some overlap but it must be understood that you can be compliant and still get hacked. The overlap of compliance is a great thing to focus on for measuring what we do. Did your compliance program make you more secure? Can you show how another group used a compliance requirement to make something better? What if something compliance required saved some money on how the network was architected? There are a lot of side benefits to pay attention to. Make sure you note the things that are improvements, even if they aren't necessarily security improvements.
I've seen examples where compliance was used to justify 2 factor authentication (2FA) in an organization, There are few things more powerful than 2FA that you can deploy. Showing compliance helped move an initiative like this forward, and also showing how the number of malicious logs drops substantially is a powerful message. Just turning on 2FA isn't enough. Make sure you show why it's better, how the attacks are slowed or stopped. Make sure you can show there were few issues for users (the people who struggle will complain loudly). If there is massive disruption for your users, figure out why you didn't know this would happen, someone screwed something up that means. It's important to measure the good and the bad. We rarely measure failure which is a problem. Nobody has a 100% success rate, learn from your failure.
What about attending a meeting or industry conference? Do you just go, file the expense report, and do nothing? That sounds like a waste of time and money. Make sure you have concrete actions. Write down what happened, why it was important you were there, how you made the situation better, and what you're going to do next. How did the meeting move your project forward? Did you learn something new, or make some plans that will help in the future? Make sure the person paying your bills sees this. Make them happy to be providing you the means to keep the business moving forward.
The Cost
The very first step we have to consider when we want to measure what we're doing is to do your homework and understand cost. Not just upfront cost but cost of machines, disk, people, services, anything you need to keep the business moving forward. If there are certain requirements needed for a solution make sure you understand and document it. If a certain piece of software or service has to be used show why. Show what part of the business can function because of the cost you're providing. Remember this is going to be specific requirements you can't escape. These are not commodity services and solutions. And of course the goal is to move forward.
If you inherit an existing solution take a good look at everything, make sure you know exactly what the resource cost of the solution is. The goal here isn't always to show a return on investment, but to show that the current solution makes sense. Just because something costs less money doesn't mean it's cheaper. If your cut rate services will put the project in jeopardy you're going to be in trouble someday. Be able to show this is a real threat. It's possible a decision will be made to take on this threat, but that's not always your choice. Always be able to answer the questions "if we do this what happens" and "if we don't do this what happens".
Conclusion
This topic is tricky. I keep thinking about it and even as I wrote this post it changed quite a lot from what I started to write. If you have something that makes money it's easy to justify investment. If you have something that's a pure cost center it's easy to minimize cost. This middle ground is tricky. How do you show value for something you have to do but isn't directly generating revenue? If you work for a forward looking business you probably won't have to spend a ton of time getting these projects funded. Growing companies understand the cost of doing business.
I have seen some companies that aren't growing as quickly fail to see value in the cost of doing business. There's nothing wrong with this sometimes, but as a security leader your job is to make your leadership understand what isn't happening because of this lack of investment. Sometimes if you keep a project limping along, barely alive, you end up causing a great deal of damage to the project and your staff. If leadership won't fund something, it means they don't view it as important and neither should you. If you think it is important, you need to sell it to your leadership. Sometimes you can't and won't win though, and then you have to be willing to let it go.
Monday, September 11, 2017
Measuring security: Part 1 - Things that make money
If you read my previous post on measuring security, you know I broke measuring into three categories. I have no good reason to do this other than it's something that made sense to me. There are without question better ways to split these apart, I'm sure there is even overlap, but that's not important. What actually matters is to start a discussion on measuring what we do. The first topic is about measuring security that directly adds to revenue such as a product or service.
Revenue
The concept of making money is simple enough. You take a resource such as raw materials, money, even people in some instances. Usually it's all three. You take these resources then transform them into something new and better. The new creation is then turned into money, or revenue, for your
business. If you have a business that doesn't make more money than it spends you have a problem. If you have a business that doesn't make any money you have a disaster.
This is easy enough to understand, but let's use a grossly simplified example to make sure we're all on the same page. Let's say you're making widgets. I suppose since this is a security topic we should call them BlockWidgetChain. In our fictional universe you spend $10 on materials and people. Make sure you can track how much something costs, you should be able to determine how much of that $10 is materials and how much is people. You then you sell the BlockWidgetChain for $20. That means you spent $10 to make $20. This should make sense to anyone who understands math (or maths for you English speakers).
Now let's say you have a competitor who makes BlockChainWidgets. They're the same thing basically, but they have no idea how much it costs them to make BlockChainWidgets. They know if they charge more than $20 they can't compete because BlockWidgetChains cost $20. Their solution is to charge $20 and hope the books work out.
I've not only described the business plan for most startups but also a company that's almost certainly in trouble. You have to know how much you spend on resources. If you spend more than you're charging for the product that's a horrible business model. Most of security works like this unfortunately. We have no idea how much a lot of what we do costs, we certainly don't know how much value it adds to the bottom line. In many instances we cannot track spending in a meaningful way.
Measuring security
So now we're on to the idea of measuring security in an environment where the security is responsible for making money. Something like security features in a product. Maybe even a security product in some instances. This is the work that pays my bills. I've been working on product security for a very long time. If you're part of your product team (which you should be, product security doesn't belong anywhere else, more on that another day) then you understand the importance of having features that make a product profitable and useful. For example I would say SSO is a must have in today's environment. If you don't have this feature you can't be as effective in the market. But adding and maintaining features isn't free. If you spend $30 and sell it for $20, you'd make more money just by staying in bed. Sometimes the most profitable decision is to not do something.
Go big or go home
The biggest mistake we like to make is doing too much. It's easy to scope a feature too big. At worst you end up failing completely, at best you end up with what you should have scoped in the first place. But you spend a lot more on failure before you end up where you should have been from the start.
Let's use SSO as our example here. If you were going to scope the best SSO solution in the world, your product would be using SAML, OAuth, PKI, Kerberos, Active Directory, LDAP, and whatever else you manage to think of on planning day. This example is pretty clearly over the top, but I bet a lot of new SSO system scope SAML and OAuth at the same time. The reality is you only need one to start. You can add more later. Firstly having a small scope is important. It shows you want to do one thing and do it well instead of doing 3 things badly. There are few features that are useful in a half finished state. Your sales team has no desire to show off a half finished product.
How to decide
But how do we decide which feature to add? The first thing I do is look at customer feedback. Do the customers clearly prefer one over the other? Setup calls with them, go on visits. Learn what they do and how they do it. If this doesn't give you a clear answer, the next question is always "which feature would sell more product". In the case of something like SAML vs OAuth there might not be a good answer. If you're some sort of cloud service OAuth means you can let customers auth against Google and Facebook. That would probably result in more users.
If you're focused on a lot of on-prem solutions, SAML might be more used. It's even possible SSO isn't what customers are after once you start to dig. I find it's best to make a mental plan of how things should look, then make sure that's not what gets built because whatever I think of first is always wrong ;)
But how much does it cost?
Lastly if there's not a good way to show revenue for a feature, you can look at investment cost. The amount of time and money something will take to implement can really help when deciding what to do. If a feature will take years to develop, that's probably not a feature you want or need. Most industries will be very different in a few years. The expectations of today won't be the expectations of tomorrow.
For example if SAML will take three times as long as OAuth to implement. And both features will result in the same number of sales. OAuth will have a substantially larger return on investment as it's much cheaper to implement. A feature doesn't count for anything until it's on the market. Half done or in development are the same as "doesn't exist". Make sure you track time as part of your costs. Money is easy to measure, but people and time are often just as important.
I really do think this is the easiest security category to measure and justify. That could be because I do it every day, but I think if you can tie actual sales back to security features you'll find yourself in a good place. Your senior leadership will think you're magic if you can show them if they invest resources in X they will get Y. Make sure you track the metrics though. It's not enough to meet expectations, make an effort to exceed your expectations. There's nothing leadership likes better than someone who can over-deliver on a regular basis.
I see a lot of groups that don't do any of this. They wander in circles sometimes adding security features that don't matter, often engineering solutions that customers only need or want 10% of. I'll never forget when I first looked at actual metrics on new features and realized something we wanted to add was going to have a massive cost and generate zero additional revenue (it may have actually detracted in future product sales). On this day I saw the power in metrics. Overnight my group became heroes for saving everyone a lot of work and headaches. Sometimes doing nothing is the most valuable action you can take.
Revenue
The concept of making money is simple enough. You take a resource such as raw materials, money, even people in some instances. Usually it's all three. You take these resources then transform them into something new and better. The new creation is then turned into money, or revenue, for your business. If you have a business that doesn't make more money than it spends you have a problem. If you have a business that doesn't make any money you have a disaster.
This is easy enough to understand, but let's use a grossly simplified example to make sure we're all on the same page. Let's say you're making widgets. I suppose since this is a security topic we should call them BlockWidgetChain. In our fictional universe you spend $10 on materials and people. Make sure you can track how much something costs, you should be able to determine how much of that $10 is materials and how much is people. You then you sell the BlockWidgetChain for $20. That means you spent $10 to make $20. This should make sense to anyone who understands math (or maths for you English speakers).
Now let's say you have a competitor who makes BlockChainWidgets. They're the same thing basically, but they have no idea how much it costs them to make BlockChainWidgets. They know if they charge more than $20 they can't compete because BlockWidgetChains cost $20. Their solution is to charge $20 and hope the books work out.
I've not only described the business plan for most startups but also a company that's almost certainly in trouble. You have to know how much you spend on resources. If you spend more than you're charging for the product that's a horrible business model. Most of security works like this unfortunately. We have no idea how much a lot of what we do costs, we certainly don't know how much value it adds to the bottom line. In many instances we cannot track spending in a meaningful way.
Measuring security
So now we're on to the idea of measuring security in an environment where the security is responsible for making money. Something like security features in a product. Maybe even a security product in some instances. This is the work that pays my bills. I've been working on product security for a very long time. If you're part of your product team (which you should be, product security doesn't belong anywhere else, more on that another day) then you understand the importance of having features that make a product profitable and useful. For example I would say SSO is a must have in today's environment. If you don't have this feature you can't be as effective in the market. But adding and maintaining features isn't free. If you spend $30 and sell it for $20, you'd make more money just by staying in bed. Sometimes the most profitable decision is to not do something.
Go big or go home
The biggest mistake we like to make is doing too much. It's easy to scope a feature too big. At worst you end up failing completely, at best you end up with what you should have scoped in the first place. But you spend a lot more on failure before you end up where you should have been from the start.
Let's use SSO as our example here. If you were going to scope the best SSO solution in the world, your product would be using SAML, OAuth, PKI, Kerberos, Active Directory, LDAP, and whatever else you manage to think of on planning day. This example is pretty clearly over the top, but I bet a lot of new SSO system scope SAML and OAuth at the same time. The reality is you only need one to start. You can add more later. Firstly having a small scope is important. It shows you want to do one thing and do it well instead of doing 3 things badly. There are few features that are useful in a half finished state. Your sales team has no desire to show off a half finished product.
How to decide
But how do we decide which feature to add? The first thing I do is look at customer feedback. Do the customers clearly prefer one over the other? Setup calls with them, go on visits. Learn what they do and how they do it. If this doesn't give you a clear answer, the next question is always "which feature would sell more product". In the case of something like SAML vs OAuth there might not be a good answer. If you're some sort of cloud service OAuth means you can let customers auth against Google and Facebook. That would probably result in more users.
If you're focused on a lot of on-prem solutions, SAML might be more used. It's even possible SSO isn't what customers are after once you start to dig. I find it's best to make a mental plan of how things should look, then make sure that's not what gets built because whatever I think of first is always wrong ;)
But how much does it cost?
Lastly if there's not a good way to show revenue for a feature, you can look at investment cost. The amount of time and money something will take to implement can really help when deciding what to do. If a feature will take years to develop, that's probably not a feature you want or need. Most industries will be very different in a few years. The expectations of today won't be the expectations of tomorrow.
For example if SAML will take three times as long as OAuth to implement. And both features will result in the same number of sales. OAuth will have a substantially larger return on investment as it's much cheaper to implement. A feature doesn't count for anything until it's on the market. Half done or in development are the same as "doesn't exist". Make sure you track time as part of your costs. Money is easy to measure, but people and time are often just as important.
I really do think this is the easiest security category to measure and justify. That could be because I do it every day, but I think if you can tie actual sales back to security features you'll find yourself in a good place. Your senior leadership will think you're magic if you can show them if they invest resources in X they will get Y. Make sure you track the metrics though. It's not enough to meet expectations, make an effort to exceed your expectations. There's nothing leadership likes better than someone who can over-deliver on a regular basis.
I see a lot of groups that don't do any of this. They wander in circles sometimes adding security features that don't matter, often engineering solutions that customers only need or want 10% of. I'll never forget when I first looked at actual metrics on new features and realized something we wanted to add was going to have a massive cost and generate zero additional revenue (it may have actually detracted in future product sales). On this day I saw the power in metrics. Overnight my group became heroes for saving everyone a lot of work and headaches. Sometimes doing nothing is the most valuable action you can take.
Monday, September 4, 2017
The father of modern security: B. F. Skinner
A lot of what we call security is voodoo. Most of it actually.
What I mean with that statement is our security process is often based on ideas that don't really work. As an industry we have built up a lot of ideas and processes that aren't actually grounded in facts and science. We don't understand why we do certain things, but we know that if we don't do those things something bad will happen! Will it really happen? I heard something will happen. I suspect the answer is no, but it's very difficult to explain this concept sometimes.
I'm going to start with some research B. F. Skinner did as my example here. The very short version is that Skinner did research on pigeons. He had a box that delivered food at random intervals. The birds developed rituals that they would do in order to have their food delivered. If a pigeon decided that spinning around would cause food to be delivered, it would continue to spin around, eventually the food would appear reinforcing the nonsensical behavior. The pigeon believed their ritual was affecting how often the food was delivered. The reality is nothing the pigeon did affected how often food was delivered. The pigeon of course didn't know this, they only knew what they experienced.
My favorite example to use next to this pigeon experiment is the password policies of old. A long time ago someone made up some rules about what a good password should look like. A good password has letters, and numbers, and special characters, and the name of a tree in it. How often we should change a password was also part of this. Everyone knows you should change passwords as often as possible. Two or three times a day is best. The more you change it the more secure it is!
Today we've decided that all this advice was terrible. The old advice was based on voodoo. It was our ritual that kept us safe. The advice to some people seemed like a fair idea, but there were no facts backing it up. Lots of random characters seems like a good idea, but we didn't know why. Changing your password often seemed like a good idea, but we didn't know why. This wasn't much different than the pigeon spinning around to get more food. We couldn't prove it didn't not work, so we kept doing it because we had to do something.
Do you know why we changed all of our password advice? We changed it because someone did the research around passwords. We found out that very long passwords using real words is substantially better than a nonsense short password. We found out that people aren't good at changing their passwords every 90 days. They end up using horrible passwords and adding a 1 to the end. We measured the effectiveness of these processes and understood they were actually doing the opposite of what we wanted them to do. Without question there are other security ideas we do today that fall into this category.
Even though we have research showing this password advice was terrible we still see a lot of organizations and people who believe the old rituals are the right way to keep passwords safe. Sometimes even when you prove something to someone they can't believe it. They are so invested in their rituals that they are unable to imagine any other way of existing. A lot of security happens this way. How many of our rules and processes are based on bad ideas?
How to measure
Here's where it gets real. It's easy to pick on the password example because it's in the past. We need to focus on the present and the future. You have an organization that's full of policy, ideas, and stuff. How can we try to make a dent in what we have today? What matters? What doesn't work, and what's actually harmful?
I'm going to split everything into 3 possible categories. We'll dive deeper into each in future posts, but we'll talk about them briefly right now.
Things that make money
Number one is things that make money. This is something like a product you sell, or a website that customers use to interact with your company. Every company does something that generates revenue. Measuring things that fit into this category is really easy. You just ask "Will this make more, less, or the same amount of money?" If the answer is less you're wasting your time. I wrote about this a bit a long time ago, the post isn't great, but the graphic I made is useful, print it out and plot your features on it. You can probably start asking this question today without much excitement.
Cost of doing business
The next category is what I call cost of doing business. This would be things like compliance or being a part of a professional organization. Sending staff to conferences and meetings. Things that don't directly generate revenue but can have a real impact on the revenue. If you don't have PCI compliance, you can't process payments, you have no revenue, and the company won't last long. Measuring some of these is really hard. Does sending someone to Black Hat directly generate revenue? No. But it will create valuable connections and they will likely learn new things that will be a benefit down the road. I guess you could think of these as investments in future revenue.
My thoughts on how to measure this one is less mature. I think about these often. I'll elaborate more in a future post.
Infrastructure
The last category I'm going to call "infrastructure". This one is a bit harder to grasp what makes sense. It's not unlike the previous question though. In this case we ask ourselves "If I stopped doing this what bad thing would happen?" Now I don't mean movie plot bad thing. Yeah if you stopped using your super expensive keycard entry system a spy from a competitor could break in and steal all your secrets using an super encrypted tor enabled flash drive, but they probably won't. This is the category where you have to consider the cost of an action vs the cost of not doing an action. Not doing things will often have a cost, but doing things also has a cost.
Return on investment is the name of the game here. Nobody likes to spend money they don't have to. This is why cloud is disrupting everything. Why pay for servers you don't need when you can rent only what you do need?
I have some great stories for this category, be sure to come back when I publish this followup article.
The homework for everyone now is to just start thinking about what you do and why you do it. If you don't have a good reason, you need to change your thinking. Changing your thinking is really hard to do as a human though. Many of us like to double down on our old beliefs when presented with facts. Don't be that person, keep an open mind.
What I mean with that statement is our security process is often based on ideas that don't really work. As an industry we have built up a lot of ideas and processes that aren't actually grounded in facts and science. We don't understand why we do certain things, but we know that if we don't do those things something bad will happen! Will it really happen? I heard something will happen. I suspect the answer is no, but it's very difficult to explain this concept sometimes.
I'm going to start with some research B. F. Skinner did as my example here. The very short version is that Skinner did research on pigeons. He had a box that delivered food at random intervals. The birds developed rituals that they would do in order to have their food delivered. If a pigeon decided that spinning around would cause food to be delivered, it would continue to spin around, eventually the food would appear reinforcing the nonsensical behavior. The pigeon believed their ritual was affecting how often the food was delivered. The reality is nothing the pigeon did affected how often food was delivered. The pigeon of course didn't know this, they only knew what they experienced.My favorite example to use next to this pigeon experiment is the password policies of old. A long time ago someone made up some rules about what a good password should look like. A good password has letters, and numbers, and special characters, and the name of a tree in it. How often we should change a password was also part of this. Everyone knows you should change passwords as often as possible. Two or three times a day is best. The more you change it the more secure it is!
Today we've decided that all this advice was terrible. The old advice was based on voodoo. It was our ritual that kept us safe. The advice to some people seemed like a fair idea, but there were no facts backing it up. Lots of random characters seems like a good idea, but we didn't know why. Changing your password often seemed like a good idea, but we didn't know why. This wasn't much different than the pigeon spinning around to get more food. We couldn't prove it didn't not work, so we kept doing it because we had to do something.
Do you know why we changed all of our password advice? We changed it because someone did the research around passwords. We found out that very long passwords using real words is substantially better than a nonsense short password. We found out that people aren't good at changing their passwords every 90 days. They end up using horrible passwords and adding a 1 to the end. We measured the effectiveness of these processes and understood they were actually doing the opposite of what we wanted them to do. Without question there are other security ideas we do today that fall into this category.
Even though we have research showing this password advice was terrible we still see a lot of organizations and people who believe the old rituals are the right way to keep passwords safe. Sometimes even when you prove something to someone they can't believe it. They are so invested in their rituals that they are unable to imagine any other way of existing. A lot of security happens this way. How many of our rules and processes are based on bad ideas?
How to measure
Here's where it gets real. It's easy to pick on the password example because it's in the past. We need to focus on the present and the future. You have an organization that's full of policy, ideas, and stuff. How can we try to make a dent in what we have today? What matters? What doesn't work, and what's actually harmful?
I'm going to split everything into 3 possible categories. We'll dive deeper into each in future posts, but we'll talk about them briefly right now.
Things that make money
Number one is things that make money. This is something like a product you sell, or a website that customers use to interact with your company. Every company does something that generates revenue. Measuring things that fit into this category is really easy. You just ask "Will this make more, less, or the same amount of money?" If the answer is less you're wasting your time. I wrote about this a bit a long time ago, the post isn't great, but the graphic I made is useful, print it out and plot your features on it. You can probably start asking this question today without much excitement.
Cost of doing business
The next category is what I call cost of doing business. This would be things like compliance or being a part of a professional organization. Sending staff to conferences and meetings. Things that don't directly generate revenue but can have a real impact on the revenue. If you don't have PCI compliance, you can't process payments, you have no revenue, and the company won't last long. Measuring some of these is really hard. Does sending someone to Black Hat directly generate revenue? No. But it will create valuable connections and they will likely learn new things that will be a benefit down the road. I guess you could think of these as investments in future revenue.
My thoughts on how to measure this one is less mature. I think about these often. I'll elaborate more in a future post.
Infrastructure
The last category I'm going to call "infrastructure". This one is a bit harder to grasp what makes sense. It's not unlike the previous question though. In this case we ask ourselves "If I stopped doing this what bad thing would happen?" Now I don't mean movie plot bad thing. Yeah if you stopped using your super expensive keycard entry system a spy from a competitor could break in and steal all your secrets using an super encrypted tor enabled flash drive, but they probably won't. This is the category where you have to consider the cost of an action vs the cost of not doing an action. Not doing things will often have a cost, but doing things also has a cost.
Return on investment is the name of the game here. Nobody likes to spend money they don't have to. This is why cloud is disrupting everything. Why pay for servers you don't need when you can rent only what you do need?
I have some great stories for this category, be sure to come back when I publish this followup article.
The homework for everyone now is to just start thinking about what you do and why you do it. If you don't have a good reason, you need to change your thinking. Changing your thinking is really hard to do as a human though. Many of us like to double down on our old beliefs when presented with facts. Don't be that person, keep an open mind.
Wednesday, August 30, 2017
Security ROI isn't impossible, we suck at measuring
As of late I've been seeing a lot of grumbling that security return on investment (ROI) is impossible. This is of course nonsense. Understanding your ROI is one of the most important things you can do as a business leader. You have to understand if what you're doing makes sense. By the very nature of business, some of the things we do have more value than other things. Some things even have negative value. If we don't know which things are the most important, we're just doing voodoo security.
H. James Harrington once said
Anyone paying attention to the current state of security will probably shed a tear over that statement. The foundation of the statement results in this truth: we can't control or improve the state of security today. As much as we all like to talk about what's wrong with security and how to fix it. The reality is we don't really know what's broken, which of course means we have no idea how to fix anything.
Measuring security isn't impossible, it's just really hard today. It's really hard because we don't really understand what security is in most instances. Security isn't one thing, it's a lot of little things that don't really have anything to do with each other but we clump them together for some reason. We like to build teams of specialized people and call them the security team. We pretend we're responsible for a lot of little unrelated activities but we often don't have any real accountability. The reality is this isn't a great way to do something that actually works, it's a great way to have a lot smart people fail to live up to their true potential. The best security teams in the world today aren't good at security, they're just really good at marketing themselves so everyone thinks they're good at security.
Security needs to be a part of everything, not a special team that doesn't understand what's happening outside their walls. Think for a minute what an organization would look like if we split groups up by what programming language they knew. Now you have all the python people in one corner and all the C guys in the other corner. They'll of course have a long list of reasons why they're smarter and better than the other group (we'll ignore the perl guys down in the basement). Now if there is a project that needs some C and some python they would have to go to each group and get help. Bless the soul of anyone who needs C and python working together in their project. You know this would just be a massive insane turf war with no winner. It's quite likely the project would never work because the groups wouldn't have a huge incentive to work together. I imagine you can see the problem here. You have two groups that need to work together without proper incentive to actually work together.
Security is a lot like this. Does having a special secure development group outside of the development group make sense? Why does it make sense to have a security operations group that isn't just part of IT? If you're not part of a group do you have an incentive for the group to succeed? If I can make development's life so difficult they can't possibly succeed that's development's problem, not yours. You have no incentive to be a reasonable member of the team. The reality is you're not a member of the team at all. Your incentive is to protect your own turf, not help anyone else.
I'm going to pick on Google's Project Zero for a minute here. Not because they're broken, but because they're really really good at what they do. Project zero does research into how to break things, then they work with the project they broke to make it better. If this was part of a more traditional security thinking group, Project Zero would do research, build patches, then demand everyone uses whatever it is they built and throw a tantrum if they don't. This would of course be crazy, unwelcome, and a waste of time. Project Zero has a razor focus on research. More importantly though they work with other groups when it's time to get the final work done. Their razor focus and ability to work with others gives them a pretty clear metric they can see. How many flaws did they find? How many got fixed? How many new attack vectors did they create? This is easy to measure. Of course some groups won't work with them, but in that case they can publish their advisories and move on. There's no value in picking long horrible fights.
So here's the question you have to ask yourself. How much of what you do directly affects the group you're a part of? I don't mean things like enforcing compliance, compliance is a cost like paying for electricity, think bigger here about things that generate revenue. If you're doing a project with development, do your decisions affect them or do they affect you? If your decisions affect development you probably can't measure what you do. You can really only measure things that affect you directly. Even if you think you can measure someone else, you'll never be as good as they are. And honestly, who cares what someone else is doing, measure yourself first.
It's pretty clear we don't actually understand what we like to call "security" because we have no idea how to measure it. If we did understand it, we could measure it. According to H. James Harrington we can't fix what we can't measure it. I think given everything we've seen over the past few years, this is quite accurate. We will never fix our security problems without first measuring our security ROI.
I'll spend some time in the next few posts discussing how to measure what we do with actual examples. It's not as hard as it sounds.
H. James Harrington once said
Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.
Anyone paying attention to the current state of security will probably shed a tear over that statement. The foundation of the statement results in this truth: we can't control or improve the state of security today. As much as we all like to talk about what's wrong with security and how to fix it. The reality is we don't really know what's broken, which of course means we have no idea how to fix anything.Measuring security isn't impossible, it's just really hard today. It's really hard because we don't really understand what security is in most instances. Security isn't one thing, it's a lot of little things that don't really have anything to do with each other but we clump them together for some reason. We like to build teams of specialized people and call them the security team. We pretend we're responsible for a lot of little unrelated activities but we often don't have any real accountability. The reality is this isn't a great way to do something that actually works, it's a great way to have a lot smart people fail to live up to their true potential. The best security teams in the world today aren't good at security, they're just really good at marketing themselves so everyone thinks they're good at security.
Security needs to be a part of everything, not a special team that doesn't understand what's happening outside their walls. Think for a minute what an organization would look like if we split groups up by what programming language they knew. Now you have all the python people in one corner and all the C guys in the other corner. They'll of course have a long list of reasons why they're smarter and better than the other group (we'll ignore the perl guys down in the basement). Now if there is a project that needs some C and some python they would have to go to each group and get help. Bless the soul of anyone who needs C and python working together in their project. You know this would just be a massive insane turf war with no winner. It's quite likely the project would never work because the groups wouldn't have a huge incentive to work together. I imagine you can see the problem here. You have two groups that need to work together without proper incentive to actually work together.
Security is a lot like this. Does having a special secure development group outside of the development group make sense? Why does it make sense to have a security operations group that isn't just part of IT? If you're not part of a group do you have an incentive for the group to succeed? If I can make development's life so difficult they can't possibly succeed that's development's problem, not yours. You have no incentive to be a reasonable member of the team. The reality is you're not a member of the team at all. Your incentive is to protect your own turf, not help anyone else.
I'm going to pick on Google's Project Zero for a minute here. Not because they're broken, but because they're really really good at what they do. Project zero does research into how to break things, then they work with the project they broke to make it better. If this was part of a more traditional security thinking group, Project Zero would do research, build patches, then demand everyone uses whatever it is they built and throw a tantrum if they don't. This would of course be crazy, unwelcome, and a waste of time. Project Zero has a razor focus on research. More importantly though they work with other groups when it's time to get the final work done. Their razor focus and ability to work with others gives them a pretty clear metric they can see. How many flaws did they find? How many got fixed? How many new attack vectors did they create? This is easy to measure. Of course some groups won't work with them, but in that case they can publish their advisories and move on. There's no value in picking long horrible fights.
So here's the question you have to ask yourself. How much of what you do directly affects the group you're a part of? I don't mean things like enforcing compliance, compliance is a cost like paying for electricity, think bigger here about things that generate revenue. If you're doing a project with development, do your decisions affect them or do they affect you? If your decisions affect development you probably can't measure what you do. You can really only measure things that affect you directly. Even if you think you can measure someone else, you'll never be as good as they are. And honestly, who cares what someone else is doing, measure yourself first.
It's pretty clear we don't actually understand what we like to call "security" because we have no idea how to measure it. If we did understand it, we could measure it. According to H. James Harrington we can't fix what we can't measure it. I think given everything we've seen over the past few years, this is quite accurate. We will never fix our security problems without first measuring our security ROI.
I'll spend some time in the next few posts discussing how to measure what we do with actual examples. It's not as hard as it sounds.
Monday, August 28, 2017
Helicopter security
After my last post about security spending, I was thinking about how most security teams integrate into the overall business (hint: they don't). As part of this thought experiment I decided to compare traditional security to something that in modern times has come to be called helicopter parenting.A helicopter parent is someone who won't let their kids do anything on their own. These are the people you hear about who follow their child to college, to sports practice. They yell at teachers and coaches for not respecting how special the child is. The kids are never allowed to take any risks because risk is dangerous and bad. If they climb the tree, while it could be a life altering experience, they could also fall and get hurt. Skateboarding is possibly the most dangerous thing anyone could ever do! We better make sure nothing bad can ever happen.
It's pretty well understood now that this sort of attitude is terrible for the children. They must learn to do things on their own, it's part of the development process. Taking risks and failing is an extremely useful exercise. It's not something we think about often, but you have to learn to fail. Failure is hard to learn. The children of helicopter parents do manage to learn one lesson they can use in their life, they learn to hide what they do from their parents. They get extremely good at finding way to get around all their rules and restrictions. To a degree we all had this problem growing up. At some point we all wanted to do something our parents didn't approve of, which generally meant we did it anyway, we just didn't tell our parents. Now imagine a universe where your parents let you do NOTHING, you're going to be hiding literally everything. Nobody throughout history has ever accepted the fact that they can do nothing, they just make sure the authoritarian doesn't know about it. Getting caught is still better than doing nothing much of the time.
This brings us to traditional security. Most security teams don't try to work with the business counterparts. Security teams often think they can just tell everyone else what to do. Have you ever heard the security team ask "what are you trying to do?" Of course not. They always just say "don't do that" or maybe "do it this way" then move on to tell the next group how to do their job. They don't try to understand what you're doing and why you are doing it. It's quite literally not their job to care what you're doing, which is part of the problem. Things like phishing tests are used to belittle, not teach (they have no value as teaching tools, but we won't discuss that today). Many of the old school security teams see their job as risk aversion, not risk management. They are helicopter security teams.
Now as we know from children, if you prevent someone from doing anything they don't become your obedient servant, they go out of their way to make sure the authority has no idea what's going on. This is basically how shadow IT became a thing. It was far easier to go around the rules than work with the existing machine. Helicopter security is worse than nothing. At least with nothing you can figure out what's going on by asking questions and getting honest answers. In a helicopter security environment information is actively hidden because truth will only get you in trouble.
Can we fix this?
I don't know the answer to this question. A lot of tech people I see (not just security) are soldiers from the last war. With the way we see cloud transforming the universe there are a lot of people who are still stuck in the past. We often hear it's hard to learn new things but it's more than that. Technology, especially security, never stands still. It used to move slow enough you could get by for a few years on old skills, but we're in the middle of disruptive change right now. If you're not constantly questioning your existing skills and way of thinking you're already behind. Some people are so far behind they will never catch up. It's human nature to double down on the status quo when you're not part of the change. Helicopter security is that doubling down.
It's far easier to fight change and hope your old skills will remain useful than it is to learn a new skill. Everything we see in IT today is basically a new skill. Today the most useful thing you can know is how to learn quickly, what you learned a few months ago could be useless today, it will probably be useless in the near future. We are actively fighting change like this in security today. We try to lump everything together and pretend we have some sort of control over it. We never really had any control, it's just a lot more obvious now than it was before. Helicopter security doesn't work, no matter how bad you want it to.
The Next Step
The single biggest thing we need to start doing is measure ourselves. Even if you don't want to learn anything new you can at least try to understand what we're doing today that actually works, which things sort of work, and of course the things that don't work at all. In the next few posts I'm going to discuss how to measure security as well as how to avoid voodoo security. It's a lot harder to justify helicopter security behavior once we understand which of our actions work and which don't.
Tuesday, August 22, 2017
Spend until you're secure
I was watching a few Twitter conversations about purchasing security last week and had yet another conversation about security ROI. This has me thinking about what we spend money on. In many industries we can spend our way out of problems, not all problems, but a lot of problems. With security if I gave you a blank check and said "fix it", you couldn't. Our problem isn't money, it's more fundamental than that.
Spend it like you got it
First let's think about how some problems can be solved with money. If you need more electricity capacity, or more help during a busy time, or more computing power, it's really easy to add capacity. You need more compute power, you can either buy more computers or just spend $2.15 in the cloud. If you need to dig a big hole, for a publicity stunt on Black Friday, you just pay someone to dig a big hole. It's not that hard.
This doesn't always work though, if you're building a new website, you probably can't buy your way to success. If a project like this falls behind it can be very difficult to catch back up. You can however track progress which I would say is at least a reasonable alternative. You can move development to another group or hire a new consultant if the old one isn't living up to expectations.
More Security
What if we need "more" security. How can we buy our way into more security for our organization? I'd start by asking the question can we show any actual value for our current security investment? If you stopped spending money on security tomorrow do you know what the results would be? If you stopped buying toilet paper for your company tomorrow you can probably understand what will happen (if you have a good facilities department I bet they already know the answer to this).
This is a huge problem in many organizations. If you don't know what would happen if you lowered or increased your security spending you're basically doing voodoo security. You can imagine many projects and processes as having a series of inputs that can be adjusted. Things like money, time, people, computers, the list could go on. You can control these variables and have direct outcomes on the project. More people could mean you can spend less money on contractors, more computers could mean less time spent on rendering or compiling. Ideally you have a way to find the optimal levels for each of these variables resulting in not only a high return on investment, but also happier workers as they can see the results of their efforts.
We can't do this with security today because security is too broad. We often don't know what would happen if we add more staff, or more technology.
Fundamental fundamentals
Spend it like you got it
First let's think about how some problems can be solved with money. If you need more electricity capacity, or more help during a busy time, or more computing power, it's really easy to add capacity. You need more compute power, you can either buy more computers or just spend $2.15 in the cloud. If you need to dig a big hole, for a publicity stunt on Black Friday, you just pay someone to dig a big hole. It's not that hard.
This doesn't always work though, if you're building a new website, you probably can't buy your way to success. If a project like this falls behind it can be very difficult to catch back up. You can however track progress which I would say is at least a reasonable alternative. You can move development to another group or hire a new consultant if the old one isn't living up to expectations.
More Security
What if we need "more" security. How can we buy our way into more security for our organization? I'd start by asking the question can we show any actual value for our current security investment? If you stopped spending money on security tomorrow do you know what the results would be? If you stopped buying toilet paper for your company tomorrow you can probably understand what will happen (if you have a good facilities department I bet they already know the answer to this).
This is a huge problem in many organizations. If you don't know what would happen if you lowered or increased your security spending you're basically doing voodoo security. You can imagine many projects and processes as having a series of inputs that can be adjusted. Things like money, time, people, computers, the list could go on. You can control these variables and have direct outcomes on the project. More people could mean you can spend less money on contractors, more computers could mean less time spent on rendering or compiling. Ideally you have a way to find the optimal levels for each of these variables resulting in not only a high return on investment, but also happier workers as they can see the results of their efforts.
We can't do this with security today because security is too broad. We often don't know what would happen if we add more staff, or more technology.
Fundamental fundamentals
So this brings us to why we can't spend our way to security. I would argue there are two real problems here. The first being "security" isn't a thing. We pretend security is an industry that means something but it's really a lot of smaller things we've clumped together in such a way that ensures we can only fail. I see security teams claim to own anything that has the word security attached to it. They claim ownership of projects and ideas, but then they don't actually take any actions because they're too busy or lack the skills to do the work. Just because you know how to do secure development doesn't automatically make you an expert at network security. If you're great at network security it doesn't mean you know anything about physical security. Security is a lot of little things, we have to start to understand what those are and how to push responsibility to respective groups. Having a special application security team that's not part of development doesn't work. You need all development teams doing things securely.
The second problem is we don't measure what we do. How many security teams tell IT they have to follow a giant list of security rules, but they have no idea what would happen if one or more of those rules were rolled back? Remember when everyone insisted we needed to use complex passwords? Now that's considered bad advice and we shouldn't make people change their passwords often. It's also a bad idea to insist they use a variety of special characters now. How many millions have been wasted on stupid password rules? The fact that we changed the rules without any fanfare means there was no actual science behind the rules in the first place. If we even tried to measure this I suspect we would have known YEARS ago that it was a terrible idea. Instead we just kept doing voodoo security. How many more of our rules do you think will end up being rolled back in the near future because they don't actually make sense?
If you're in charge of a security program the first bit of advice I'd give out is to look at everything you own and get rid of whatever you can. Your job isn't to do everything, figure out what you have to do, then do it well. One project well done is far better than 12 half finished. The next thing you need to do is figure out how much whatever you do costs, and how much benefit it creates. If you can't figure out the benefit, you can probably stop doing it today. If it costs more than it saves, you can stop that too. We must have a razor focus if we're to understand what our real problems are. Once we understand the problems we can start to solve them.
Subscribe to:
Posts (Atom)