Measuring security: Part 1 - Things that make money

If you read my previous post on measuring security, you know I broke measuring into three categories. I have no good reason to do this other than it's something that made sense to me. There are without question better ways to split these apart, I'm sure there is even overlap, but that's not important. What actually matters is to start a discussion on measuring what we do. The first topic is about measuring security that directly adds to revenue such as a product or service.

Revenue
The concept of making money is simple enough. You take a resource such as raw materials, money, even people in some instances. Usually it's all three. You take these resources then transform them into something new and better. The new creation is then turned into money, or revenue, for your
business. If you have a business that doesn't make more money than it spends you have a problem. If you have a business that doesn't make any money you have a disaster.

This is easy enough to understand, but let's use a grossly simplified example to make sure we're all on the same page. Let's say you're making widgets. I suppose since this is a security topic we should call them BlockWidgetChain. In our fictional universe you spend $10 on materials and people. Make sure you can track how much something costs, you should be able to determine how much of that $10 is materials and how much is people. You then you sell the BlockWidgetChain for $20. That means you spent $10 to make $20. This should make sense to anyone who understands math (or maths for you English speakers).

Now let's say you have a competitor who makes BlockChainWidgets. They're the same thing basically, but they have no idea how much it costs them to make BlockChainWidgets. They know if they charge more than $20 they can't compete because BlockWidgetChains cost $20. Their solution is to charge $20 and hope the books work out.

I've not only described the business plan for most startups but also a company that's almost certainly in trouble. You have to know how much you spend on resources. If you spend more than you're charging for the product that's a horrible business model. Most of security works like this unfortunately. We have no idea how much a lot of what we do costs, we certainly don't know how much value it adds to the bottom line. In many instances we cannot track spending in a meaningful way.

Measuring security
So now we're on to the idea of measuring security in an environment where the security is responsible for making money. Something like security features in a product. Maybe even a security product in some instances. This is the work that pays my bills. I've been working on product security for a very long time. If you're part of your product team (which you should be, product security doesn't belong anywhere else, more on that another day) then you understand the importance of having features that make a product profitable and useful. For example I would say SSO is a must have in today's environment. If you don't have this feature you can't be as effective in the market. But adding and maintaining features isn't free. If you spend $30 and sell it for $20, you'd make more money just by staying in bed. Sometimes the most profitable decision is to not do something.

Go big or go home
The biggest mistake we like to make is doing too much. It's easy to scope a feature too big. At worst you end up failing completely, at best you end up with what you should have scoped in the first place. But you spend a lot more on failure before you end up where you should have been from the start.

Let's use SSO as our example here. If you were going to scope the best SSO solution in the world, your product would be using SAML, OAuth, PKI, Kerberos, Active Directory, LDAP, and whatever else you manage to think of on planning day. This example is pretty clearly over the top, but I bet a lot of new SSO system scope SAML and OAuth at the same time. The reality is you only need one to start. You can add more later. Firstly having a small scope is important. It shows you want to do one thing and do it well instead of doing 3 things badly. There are few features that are useful in a half finished state. Your sales team has no desire to show off a half finished product.

How to decide
But how do we decide which feature to add? The first thing I do is look at customer feedback. Do the customers clearly prefer one over the other? Setup calls with them, go on visits. Learn what they do and how they do it. If this doesn't give you a clear answer, the next question is always "which feature would sell more product". In the case of something like SAML vs OAuth there might not be a good answer. If you're some sort of cloud service OAuth means you can let customers auth against Google and Facebook. That would probably result in more users.

If you're focused on a lot of on-prem solutions, SAML might be more used. It's even possible SSO isn't what customers are after once you start to dig. I find it's best to make a mental plan of how things should look, then make sure that's not what gets built because whatever I think of first is always wrong ;)

But how much does it cost?
Lastly if there's not a good way to show revenue for a feature, you can look at investment cost. The amount of time and money something will take to implement can really help when deciding what to do. If a feature will take years to develop, that's probably not a feature you want or need. Most industries will be very different in a few years. The expectations of today won't be the expectations of tomorrow.

For example if SAML will take three times as long as OAuth to implement. And both features will result in the same number of sales. OAuth will have a substantially larger return on investment as it's much cheaper to implement. A feature doesn't count for anything until it's on the market. Half done or in development are the same as "doesn't exist". Make sure you track time as part of your costs. Money is easy to measure, but people and time are often just as important.

I really do think this is the easiest security category to measure and justify. That could be because I do it every day, but I think if you can tie actual sales back to security features you'll find yourself in a good place. Your senior leadership will think you're magic if you can show them if they invest resources in X they will get Y. Make sure you track the metrics though. It's not enough to meet expectations, make an effort to exceed your expectations. There's nothing leadership likes better than someone who can over-deliver on a regular basis.

I see a lot of groups that don't do any of this. They wander in circles sometimes adding security features that don't matter, often engineering solutions that customers only need or want 10% of. I'll never forget when I first looked at actual metrics on new features and realized something we wanted to add was going to have a massive cost and generate zero additional revenue (it may have actually detracted in future product sales). On this day I saw the power in metrics. Overnight my group became heroes for saving everyone a lot of work and headaches. Sometimes doing nothing is the most valuable action you can take.