Brook's Hackathon Judging Notes, 2024

Brook's Hackathon Judging Notes, 2024
Userlevel 4

I’d like to relate a few things I’ve noticed from this year’s Hackathon judging.

Most importantly, had a development team or security champion asked for my feedback on the vast majority of submissions, I’d be so delighted to have seen such fine work. Which made this year’s judging wickedly difficult.

Clearly, our capabilities are progressing significantly.

For one thing, there are far more resources available to help with threats.

It’s been a truism for a long time that one of the most difficult things for threat modelling newbies is to climb out of their well-known threat catalog (often quite limited) to consider the broad range of things “that can go wrong”, as @Adam Shostack leads the process. Adam, and really many of us who teach and mentor others, talk about this difficulty repeatedly (and have for years).

In fact, that’s why Adam wrote his last book, Threats: it’s hard to be comprehensive. One needs a near encyclopedic knowledge of threats, while also understanding to which technologies and attack surfaces particular threats apply (it’s most certainly not all!)

to conquer this challenge, many teams drew from our expanding set of threat resources, threat catalogs, threat libraries. Having wrestled for decades with my students’ and colleagues’ challenge in this arena, I’m thrilled that finally there’s real help. On a side note, let me suggest that beyond this hackathon, there are open source and commercial tools that can also assist with identifying relevant threats. I say, “use them!” But not for the Hackathon, please.

At the same time, I could find misses in every submission. There was no perfect. Som of those were technical, like mitigations that don’t address a particular threat, or not up-leveling variations of a threat or its applicability to multiple points on DevSecAI’s systems to a singl, combined threat, not 30 variations applied to multiple components (which makes parsing the model difficult).

I will point out that last year’s and this year’s panel of judges most certainly scored teams’ reflections and collaboration documentation. Sometimes, these scores can be the difference between placing and not! Attend to the non-technical. It matters in these Hackathons because threat modelling is a team sport.

Next year’s participants, don’t forget to reflect on your process and tell the judges about your collaboration efforts. We don’t grade on how much collaboration, but rather, whether you've thought about it, documented what happened, and done your best.

The same goes for prioritizing threats. In the real world (and I’ve got 1000’s of dev project’s models in my past), every dev team will demand priority because they won’t have the resources to do everything at once. Again, lack of prioritization points can make the difference between placing and not.

Thank you, every team, every participant for your thoughtful submissions. It has been a privilege and honour to be a judge once again. I’ll end by saying, “Whew! That was quite a bit of work.” We had to make some tough distinctions.

There was, as always with my dear friends, @Kim Wuyts, @RobertHurlbut, and @AviD fantastic collaboration, amazing discussions that I hope will help me to improve my practice. You are, each of you, brilliant and amazing. You three are very special to me. It is such a pleasure to work with you. Thank you from the bottom of my heart.


2 replies

Badge +1

Grats to team 15!

Userlevel 1
Badge +1

Thanks, in fact I am having plans to check with my team captain to find out additional improvements onto our threat modeling submission, since we have mentioned threat modeling as a continuous process, but not sure if our team has time :). I would like to thank everyone’s support in helping us understand the prompt and our mentors were awesome @Michael Bernhardt and @Jholmes  helping us with our questions and guidance. The special workshops from @lfservin worth a mention here where we picked up the Gherkin syntax to address the threats, it helped us learn a lot together. Proud of our team work Team 15.