I’d like to relate a few things I’ve noticed from this year’s Hackathon judging.
Most importantly, had a development team or security champion asked for my feedback on the vast majority of submissions, I’d be so delighted to have seen such fine work. Which made this year’s judging wickedly difficult.
Clearly, our capabilities are progressing significantly.
For one thing, there are far more resources available to help with threats.
It’s been a truism for a long time that one of the most difficult things for threat modelling newbies is to climb out of their well-known threat catalog (often quite limited) to consider the broad range of things “that can go wrong”, as
In fact, that’s why Adam wrote his last book, Threats: it’s hard to be comprehensive. One needs a near encyclopedic knowledge of threats, while also understanding to which technologies and attack surfaces particular threats apply (it’s most certainly not all!)
to conquer this challenge, many teams drew from our expanding set of threat resources, threat catalogs, threat libraries. Having wrestled for decades with my students’ and colleagues’ challenge in this arena, I’m thrilled that finally there’s real help. On a side note, let me suggest that beyond this hackathon, there are open source and commercial tools that can also assist with identifying relevant threats. I say, “use them!” But not for the Hackathon, please.
At the same time, I could find misses in every submission. There was no perfect. Som of those were technical, like mitigations that don’t address a particular threat, or not up-leveling variations of a threat or its applicability to multiple points on DevSecAI’s systems to a singl, combined threat, not 30 variations applied to multiple components (which makes parsing the model difficult).
I will point out that last year’s and this year’s panel of judges most certainly scored teams’ reflections and collaboration documentation. Sometimes, these scores can be the difference between placing and not! Attend to the non-technical. It matters in these Hackathons because threat modelling is a team sport.
Next year’s participants, don’t forget to reflect on your process and tell the judges about your collaboration efforts. We don’t grade on how much collaboration, but rather, whether you've thought about it, documented what happened, and done your best.
The same goes for prioritizing threats. In the real world (and I’ve got 1000’s of dev project’s models in my past), every dev team will demand priority because they won’t have the resources to do everything at once. Again, lack of prioritization points can make the difference between placing and not.
Thank you, every team, every participant for your thoughtful submissions. It has been a privilege and honour to be a judge once again. I’ll end by saying, “Whew! That was quite a bit of work.” We had to make some tough distinctions.
There was, as always with my dear friends,
