Quantifying Risk: How do you measure success in security?

Every quarter, a CISO has to stand up in front of their leadership and attempt to put a price on something that didn’t happen. When your job is all about avoiding costly incidents and mistakes, it’s hard to put a dollar value on your work, and even harder to justify your request for budget and resources. At HackerOne’s recent Security@ conference, Slack and Hyatt’s CISOs sat down for a chat about their challenges and the hacks they use to quantify risk: 

The challenge: 

Larkin Ryder, CISO of Slack: “I’m lucky, Slack is a company that understands the value of security and that customer data security is a top priority so our team has enjoyed both alignment and support. With that, however, comes the challenge of living up to expectations and that I’m spending every dollar well. I make sure I am spending that money with the best metrics and attention to value in mind and returning the value to the organization.”

Benjamin Vaughn, CISO of Hyatt: “We’ve had great success with building support for the security team and our programs by intrinsically linking our purpose to Hyatt’s purpose: ‘we care for people so they can be their best.’ We don’t have conversations about cutting costs, rather how we can provide more care.”

Benjamin: “I have always struggled to present what success looks like to the board. How do you decide what success or ROI is? How do we measure breaches that haven’t happened? We want to be able to show that bad things happen less often because we’re here, and when the bad things happen, their impact is lessened because we’re here.” 

Larkin: “We have the same problem: how do you measure something that didn’t happen? I was recently speaking to a former CISO at a banking and investment company who, when he joined, was excited that he’d be working with people who understand financial risk and that they would be able to help him apply that risk management to cyber, finding that magic formula to measure ROI. At the end of their work, they discovered the only metric that made sense was simply to measure how up to date their most vulnerable system was. The learning here is that simple measurements are the best.”

The solution:

When it comes to product security Larkin evaluates just three things:

1. How efficiently we can find vulnerabilities - how many resources it took to find it
2. How effectively we’re preventing vulnerabilities, which is hard to measure in absolutes so we present trends instead
3. How fast can we fix the vulnerability

Earlier this year, HackerOne also asked a selection of global CISOs how they measure the ROI of their cybersecurity efforts. There seems to be no size-fits-all solution with the results showing a real mix in approaches:

Cyber insurance is a tactic Benjamin is very familiar with:

Benjamin: “When I joined Hyatt, I worked closely with Risk Management, who manages  insurance purchasing for our properties, on Hyatt’s cyber insurance policy. The insurance marketplace is very equal; we get pricing based on the assessments of the same underwriters. We took the approach of going to all three major marketplaces for insurance and presented detailed presentations to the underwriters. We talked about the incidents we’ve had and what we’re doing to mitigate them. As a result, we now receive a very performant rate compared to others in the industry."

Making it stick

Larkin: “When I talk about security to a board of directors I’ve learned I need to translate into a language they understand. Stories are the best way to talk about incidents, and never fail to take advantage of a crisis! One story we shared with our board was of a recent hack that another cloud-based service experienced. We walked them through the attack chain and recovery process, and correlated with our own security program and how we were going to implement improvements.”

Benjamin: “I recently told Hyatt’s board about a HackerOne report we received where we disagreed with the hacker about how impactful their bug was. The hacker stood their ground and went on to prove that the defect was more important than we thought. We appreciate that since we know the bad guys wouldn’t argue, they’d just take advantage. We love these hackers because they bring us stuff that blows our minds on a regular basis.”

Both Larkin and Ben present the same 5 tactical things that they know their boards want to hear about:

1. Findings from pentests and audits
2. The incidents and what they have we done to follow up
3. How they’re driving down residual risk over time
4. Any changes to scope including expanding the operation or acquiring new tech 
5. Challenges for the year ahead