I am searching for ideas or experiment feedback on how to gather a sort of TM “NPS score” as a measure on how well or not we’re doing with our engineering teams. Hint: Sending MS Forms surveys don’t really work.
Looking past the “number of threat models performed”, “number of security work items opened” (and maybe never worked on), etc… how would you measure the actual value that is brought (or not) to various engineering teams as you educate/have them perform threat modeling?
As I am endeavoring in some development work to create a custom Azure DevOps extension for NFRs to bring stuff in-band of engineering teams (and ensure something more cyclic too), I have some rough ideas, but would like to open the question to the experts :)
Interesting question! If I understand what you mean, we essentially want to know: How do we measure the value that engineering teams get from threat modeling?
Depending on who’s doing the threat modeling, you might frame this two different ways:
If we work backwards from these questions, we need to understand what would engineers value in a threat model, and what brings negative-value. I would say that engineering teams would value having actionable security tasks at design time. And they would de-value unnecessary work.
If someone outside the team is doing the threat modeling and giving it to the engineering teams you could measure:
If the team themselves are creating the model, then they are going to be self-selecting the threats and work according to those two criteria as they’re building the model (they’re not going to knowingly create useless tasks for themselves). So the value should be obvious to them, in the same way that creating a user story is valuable because it helps them to design the functionality correctly. If it isn’t obvious, then perhaps you could measure time spent doing the threat model, vs estimated time to retrofit these security tasks into the application after it has gone live. This might be more relevant to the product owner, so that they can see the value of doing threat modeling early and often, vs having their product launch delayed by security concerns.
If you mean NPS, why not actually use NPS, and ask “Would you recommend threat modeling to a colleague?” 👇
Chiming in to what
@stephendv and @Adam Shostack said. Acceptance is also what I consider the key factor to measure. One can measure it in the acceptance of the methodology, here the answer to Adam’s question should give you a first hand indication closely watching the response and expressions by the team. Secondly, one can measure the acceptance of the results and the adaption/follow-up questions to conveyed information during the workshop. Is the team putting its head around the issue that have been jointly articulate or your guidance that you have been giving? Even further, I find it always most rewarding if a team is asking whether you could help them their newest feature or even asking whether you can enable them to do it themselves.
There is one more dimension that I would like to mention - having NPS (would maybe better call it a KPI) for management. There I would recommend not to highlight the number of opened security work item. When people realize it’s about your numbers, they will loose trust. An actual complex finding that would not be identifiable via security tools or the collaborative designed solution are perfect arguments. Additionally, I tend to also highlight the learnings that teams had, expressing the confidence that they will not do the same mistake again.
There is one more dimension that I would like to mention - having NPS (would maybe better call it a KPI) for management.
Just to be clear, NPS is A Thing. It’s why you get all those “Would you recommend us to a friend ...” questions with a 1-10 scale everywhere. Ask your marketing team about it. 😂
That said, you could (and probably should) use NPS as a KPI, and as with any KPI, you want multiples to avoid gaming.
Thanks all for your answers. Actually using NPS is surely going to be the first shot at that. Now, to get a proper “fill-out” ratio as in-band as possible to the developer ;-)