Did we do a good job?

  • 26 September 2023
  • 5 replies
  • 101 views

Userlevel 2

I would say this is the most enigmatic of the 4 questions in the 4 question threat modelling framework.

After you’ve clearly defined what you are going to threat model, after you’ve analyzed the system for threats, after you’ve created a set of additional controls to mitigate those threats - did you do a good job?  How do you know?  What does “good” even mean?  Would you really ever finish a threat model  and then call it ugly?

It’d be fascinating to hear how people tackle this thorny question?


5 replies

Userlevel 1

I view that question more in the agile sense-- "what could we do better?" "How can we improve?" That turns it into more of an iterative process without a final answer.

 

Remember the TM exercise is usually done as a group (“Did *WE*”), so the dynamic is prone to inconsistencies you should check

 

When I teach TM I include the following checks:

  • Check each Threat has an assigned Risk
  • Check each Risk has at least an assigned Risk decision and how to manage it: how (change the system/feature to avoid it), who (assume/transfer to) or what (control)
  • Check the Controls are “fit for purpose” (subjective but it serves as a wrap-up of the risk  decisions)
  • Try to assign a Test to verify the effectivity of each Control (my personal dream would be to change “Try to assign” to “Assign”)

In addition you should update the original system diagram:

  • New dataflows discovered
  • Use cases considered
  • Users of each data
  • Implementation details discussed
  • Is it able to describe each user journey and “data journey”

And your own global TM assumptions and catalog of Treats, Threat Actors, Risks, Controls, Checks

At the end, the model (subjective):

  • Is it complete?
  • Is it precise?
  • Does it cover the security decisions we have adopted?
  • Does it allow you to start a new iteration/revision (refining) without additional data?

In “A Guide to Threat Modelling for Developers” (https://martinfowler.com/articles/agile-threat-modelling.html) Jim Gumbley said:

 

Adam Shostack, who has written extensively on threat modelling and has provided feedback on this guide takes credit for the three question structure. He also adds a fourth question "Did we do a good enough job?" I don't disagree with Adam that we need to reflect and improve on the outcomes of our threat modelling. However, I have omitted this question from the basic structure as I believe it can be addressed elsewhere. Iterating and improving based on feedback should be implicit in agile software development, particularly when we are threat modelling 'little and often'.”

For me this question is the most enigmatic and the most difficult to answer. 

As a rule of thumb, the first thing we need to be clear about is the type of system we need to protect. Depending on the level of criticality, we should adjust the time, efforts, and resources so the task is not endless. 

From here, my advice is to try to see things from different points of view. Different profiles can provide a different perspective that can help identify aspects that had not been taken into account.

Userlevel 2

I see the “Did we do a good job” more as how did we do. To understand what good looks like, would itself need some thought or planning and invariably, over the course of execution, things change right!? Agile approach therefore as mentioned would be a suitable approach and depending on the technology and processes being used greatly impacts the outcome. Considering these, if improvements have been identified, and there is scope for further improvements (next iteration), then we did a good job.

Reply


V2