Some KPIs could include:
1. MTTD
2. MTTR
3. Coverage across endpoints
4. Number of incidents handled
5. Incident resolution rate%
ROI can become difficult to quantify sometimes. However, you may try to emphasize some of these points:
Cost Savings: Reduction in costs due to decreased incident response times and improved threat detection.
Financial ROI in XDR services: this can be calculated by comparing the cost of XDR with the savings from prevented breaches and incidents.
Great Answer! Is MTTR mean time to resolve or respond? I think both would be great.
alert precision (TP/(TP+FP)).
How about other products you have in place, and any way to visualize overlap? As in where did xdr catch stuff that would've otherwise not been caught? Or how did xdr compliment your other technologies to fill in gaps? Even if just a few walkthrough examples they might be instructive.
As you said, both respond and recover could be considered good metrics 👍
Alert precision is great for internal reporting within the team, but I think it would be negatively interpreted by external customers.
Other metrics depend on the other products you have in your environment. For example if you want to evaluate your FW you might measure: Uptime - throughput utilization- blocked packets ..etc
[Goodhart's law](https://www.splunk.com/en_us/blog/learn/goodharts-law.html)
If you're asking for metrics in this subreddit, you may want to get to know your audience here better and the Security in general too.
Metrics in Security suck. They are made up rubbish that almost never reflect the actual situation.
This might be true for a lot of people but I can say for sure we have instrumented our entire pipeline for alert precision, MTTD, all tracked by os and a bunch of more ways we break things down.l
I re-read my reply and it may had been overly harsh. Apologies. It struck a nerve. I have issue with basing so much Security around metrics, since one the facts of Security is that you **cannot** measure harm that didn't happen (or was prevented). Therefore all statements saying X measure prevents Y percentage of attacks are inherently inaccurate.
You can have trends in attacks, personal vendettas (a guy recently deleted a few hundred VMs), business rivalry, etc. Even if you gather data from your APIs being probed by external entities, you cannot be sure if the dropdown in the number of queries has anything to do with recent new measures taken.
The main goal of Security is to deter. As long as the effort to overcome Security mechanisms is greater that the effort put it, all is good. But every-single-Security-measure will fold if enough effort is leveraged against it.
^ this statement is *brutally* difficult to express in numbers. Almost impossible. If anyone attempts to correlate it with numbers, it's worse than playing roulette.
But how do you identify when the Security measure are sufficiently effective at deterring the attackers? That's a billion dollar question :-).
I understand this hasn't really been helping much with your original question, but hopefully it helped with broadening the perspective.
If I may to mention one thing about ROI - another incredibly hard mechanism to measure. If I may suggest, do not make decisions based on ROI in Security. As mentioned above, it's impossible to measure attacks that didn't happen (whether it's the effect they would have had if successfully completed or their frequency of successful attempts).
If you are unsure whether your organization should take a Security measure, I recommend asking: 'Is it acceptable that XYZ account gets hacked? Is it acceptable that Y internal information are leaked?' If no, take measures until you *feel* Secure (or hire people who have a good sense for this). There is a lot, a lot of feeling in Security :-).
Hi 👋 again, just needed some more insights as you seem to be very experienced in this area. May I know what are the most important things you would look into a health check report for an XDR product from a vendor? Would love to know from your perspective.
Since you are providing services then you may want to try and see if you can put a figure on costs avoided. IE: Our services done by our highly skilled people allows you to avoid having to hire 3.5 headcount.
Some KPIs could include: 1. MTTD 2. MTTR 3. Coverage across endpoints 4. Number of incidents handled 5. Incident resolution rate% ROI can become difficult to quantify sometimes. However, you may try to emphasize some of these points: Cost Savings: Reduction in costs due to decreased incident response times and improved threat detection. Financial ROI in XDR services: this can be calculated by comparing the cost of XDR with the savings from prevented breaches and incidents.
Great Answer! Is MTTR mean time to resolve or respond? I think both would be great. alert precision (TP/(TP+FP)). How about other products you have in place, and any way to visualize overlap? As in where did xdr catch stuff that would've otherwise not been caught? Or how did xdr compliment your other technologies to fill in gaps? Even if just a few walkthrough examples they might be instructive.
As you said, both respond and recover could be considered good metrics 👍 Alert precision is great for internal reporting within the team, but I think it would be negatively interpreted by external customers. Other metrics depend on the other products you have in your environment. For example if you want to evaluate your FW you might measure: Uptime - throughput utilization- blocked packets ..etc
Screws metrics. Does the product have decent alerts and a way to tune them?
Get the fuck out of my board room. (Joking, but seriously anon has to report to leadership, they want to see charts trending up and to the right)
[Goodhart's law](https://www.splunk.com/en_us/blog/learn/goodharts-law.html) If you're asking for metrics in this subreddit, you may want to get to know your audience here better and the Security in general too. Metrics in Security suck. They are made up rubbish that almost never reflect the actual situation.
This might be true for a lot of people but I can say for sure we have instrumented our entire pipeline for alert precision, MTTD, all tracked by os and a bunch of more ways we break things down.l
This is incredible piece of information. Thank you so much :)
I re-read my reply and it may had been overly harsh. Apologies. It struck a nerve. I have issue with basing so much Security around metrics, since one the facts of Security is that you **cannot** measure harm that didn't happen (or was prevented). Therefore all statements saying X measure prevents Y percentage of attacks are inherently inaccurate. You can have trends in attacks, personal vendettas (a guy recently deleted a few hundred VMs), business rivalry, etc. Even if you gather data from your APIs being probed by external entities, you cannot be sure if the dropdown in the number of queries has anything to do with recent new measures taken. The main goal of Security is to deter. As long as the effort to overcome Security mechanisms is greater that the effort put it, all is good. But every-single-Security-measure will fold if enough effort is leveraged against it. ^ this statement is *brutally* difficult to express in numbers. Almost impossible. If anyone attempts to correlate it with numbers, it's worse than playing roulette. But how do you identify when the Security measure are sufficiently effective at deterring the attackers? That's a billion dollar question :-). I understand this hasn't really been helping much with your original question, but hopefully it helped with broadening the perspective. If I may to mention one thing about ROI - another incredibly hard mechanism to measure. If I may suggest, do not make decisions based on ROI in Security. As mentioned above, it's impossible to measure attacks that didn't happen (whether it's the effect they would have had if successfully completed or their frequency of successful attempts). If you are unsure whether your organization should take a Security measure, I recommend asking: 'Is it acceptable that XYZ account gets hacked? Is it acceptable that Y internal information are leaked?' If no, take measures until you *feel* Secure (or hire people who have a good sense for this). There is a lot, a lot of feeling in Security :-).
Hahaha please do not apologise. I wish security leaders see this comment :D I must say super insightful, yes it can never be quantitative in security.
Hi 👋 again, just needed some more insights as you seem to be very experienced in this area. May I know what are the most important things you would look into a health check report for an XDR product from a vendor? Would love to know from your perspective.
Hi hi, sure I'll be more than happy to. But I'm currently on holidays. Please ping me after 8th July o/.
So kind of you good sir. Will surely reach out to you around July 10th. Have a good vacation and sorry to trouble during your holidays :)
Since you are providing services then you may want to try and see if you can put a figure on costs avoided. IE: Our services done by our highly skilled people allows you to avoid having to hire 3.5 headcount.