NPS for threat modeling

Interesting question!  If I understand what you mean, we essentially want to know: How do we measure the value that engineering teams get from threat modeling?

Depending on who’s doing the threat modeling, you might frame this two different ways:

• Is this threat modeling that I created for you valuable?
• Is this threat model that you created yourself valuable?

If we work backwards from these questions, we need to understand what would engineers value in a threat model, and what brings negative-value.  I would say that engineering teams would value having actionable security tasks at design time.  And they would de-value unnecessary work.

If someone outside the team is doing the threat modeling and giving it to the engineering teams you could measure:

• What % of the threats in the model are accepted by the engineering teams as tasks they need to work on?  100% means that everything in the model is useful and the teams choose to work on it.  0% means everything in the model is useless work, and teams choose not to work on it.

If the team themselves are creating the model, then they are going to be self-selecting the threats and work according to those two criteria as they’re building the model (they’re not going to knowingly create useless tasks for themselves).  So the value should be obvious to them, in the same way that creating a user story is valuable because it helps them to design the functionality correctly.  If it isn’t obvious, then perhaps you could measure time spent doing the threat model, vs estimated time to retrofit these security tasks into the application after it has gone live.  This might be more relevant to the product owner, so that they can see the value of doing threat modeling early and often, vs having their product launch delayed by security concerns.  

Chiming in to what @stephendv and @Adam Shostack said. Acceptance is also what I consider the key factor to measure. One can measure it in the acceptance of the methodology, here the answer to Adam’s question should give you a first hand indication closely watching the response and expressions by the team. Secondly, one can measure the acceptance of the results and the adaption/follow-up questions to conveyed information during the workshop. Is the team putting its head around the issue that have been jointly articulate or your guidance that you have been giving? Even further, I find it always most rewarding if a team is asking whether you could help them their newest feature or even asking whether you can enable them to do it themselves.

There is one more dimension that I would like to mention - having NPS (would maybe better call it a KPI) for management. There I would recommend not to highlight the number of opened security work item. When people realize it’s about your numbers, they will loose trust. An actual complex finding that would not be identifiable via security tools or the collaborative designed solution are perfect arguments. Additionally, I tend to also highlight the learnings that teams had, expressing the confidence that they will not do the same mistake again.

Thanks all for your answers. Actually using NPS is surely going to be the first shot at that. Now, to get a proper “fill-out” ratio as in-band as possible to the developer ;-)