May 30, 2021
The Detriments of Hero Culture
Or, “Celebrating Superman and Ignoring Clark Kent”
What Is It?
“Hero culture” is the phrase I’m using to describe this phenomenon — the recognition and celebration of work done in the face of ongoing disaster, and the ignorance or disinterest in work done that would prevent it from happening in the first place. Where everyone hears about and cheers the fire fighting, working breaches or incidents under pressure… while prevention work goes on mostly unnoticed and under-rewarded.
This is certainly not the first time this broader phenomenon has been pointed out. The comic above is one of many illustrations doing so. Folks have written various analogies about it. The security version I used in the past was related to football since the leadership I worked around at that time were all fans and students of the game. I abstracted the work that security teams did as the Offensive Line of a team:
They’re involved on every single play, they protect the organization’s greatest asset(s), and the success of every play hinges on their performance. When they’re doing their job, you tend to forget they exist. When they mess up one play, all eyes are on them, dissecting a replay of the failure in slow motion. I think this comes pretty close to conveying a lot of security work, except if you’re involved in a breach there’s no play by play from Troy Aikman (thankfully).
But if the problem has already been described then why does this note exist? The very reason I’m writing this is that despite the previous coverage of the problem, it still goes unsolved nearly everywhere. And it needs to be addressed, lest we remain trapped forever in an endless loop of enjoyable memes illustrating the dilemma.
Where Is It?
Everywhere. Not just in security fields, and not just in tech. It’s human nature. This will become much more apparent in examples below.
Why Is It?
It’s concrete and feels important
Firefighting in terms of incident response is absolutely necessary. Once there is already a fire, it’s got to be put out. That is intuitive to everyone. But it is often also easy to measure and quantify that reactive work. For preventative work it often isn’t. In my previous note I described the Shift Left philosophy where the end goal is to shift left into preventing security and privacy issues and away from merely responding to them. Difficulty in quantifying preventative work was described there. As a people leader, it may be hard to compare to more quantifiable work come performance evaluation time. It may also be difficult to describe the impact of “boring” preventative work to folks lacking context whereas everyone just knows how important the incident response work is.
It’s more glamorous
Another part is that prevention is “unsexy” in comparison. One of the manifestations I’ve noticed for a while is the spectrum of security conferences and accepted talks within them. Offensive work, exploitation, etc. tend to be the talks that are packed at security cons and thus the most commonly accepted. Take a look at the archived talks, workshops, and trainings from past BlackHat, DefCon, ToorCon, REcon, etc. events. They also have the most cons and events dedicated to them. There are bug bounties, hacking events, and CTFs focused on them. Lots of people like to read Project Zero blog posts, myself included.
Building hardened or secure products doesn’t come close. It’s not sexy, it’s just the most important work. Even some of the offensive security folks who do the work sometimes forget that the very reason Red Teams exist is to improve the defensive capabilities of the Blue Team and the system or product.
It’s human nature?
As mentioned earlier, this is not just something we see in security. It’s not even something unique to tech or engineering. It’s inherent to humanity in a sense, and we can even pluck examples from film and literature. There is great overall tweet thread on this that gets into the intrinsic nature of this phenomenon. It includes a pretty memorable example from Superman II:
No one cares or notices when Clark Kent prevents a boy from falling in the first place by highlighting the risk and forcing action to get him off the rail. Then when the boy ends up falling over the edge, everyone hurrahs after Superman swoops in and catches him just before he falls to his impending doom.
It hammers the point home with multiple examples. Another is of a Captain of the Titanic who crashes into an iceberg and needs to save the passengers vs. one who avoids it in the first place, and which one of them garners the front page news and a book deal (I’m sure you can guess by now which is which). The thread also includes this succinct synopsis:
How Do We Fix It?
This topic comes up fairly frequently in different circles in tech, but the problem never seems to get burned down. We continue to reflect on the last performance cycle, notice all the people we recognized for putting down fires that needed to be extinguished, and think about this prevention problem all over again.
Having both technical and people leadership increase their awareness of the problem — the symptoms, the systems that encourage it, and the challenges in fixing it — is half the battle. Most of the rest of the battle is taking concrete action based on that awareness.
Leadership: Socialize and reward prevention work. Whatever vehicle you use to share folks’ work might do well to include a prevention congratulation in each “episode”. Avoid the trap of overvaluing reactive or “busy” work that comes with clean countable metrics over valuable preventative work that doesn’t. Again, not everything that is measurable is impactful and not everything that is impactful is easily measured.
“Individual Contributors”: Avoid self reduction. Don’t let yourself fall into the trap of thinking preventative work isn’t important. If there was no fire, no fanfare, potentially not many eyes on the work … folks may subconsciously deprioritize sharing it, talking about it, etc. Highlight and champion the work, even if you aren’t saving folks from the rubble of a tech-debt skyscraper that’s fallen.
Everyone: Encourage shifting left. Acknowledge need for fighting fires when they arise, but reward shfting left and preventing fires in various ways or incentives will stay aligned with waiting for fires and heroically fighting them. We all have to invest in shifting left or we’ll remain stuck on the right.
Prevention and avoidance isn’t sexy. Heroism is. But that brand of sexy doesn’t build safe, secure, and trustworthy technologies. If you’re in charge of recognizing and rewarding the work of people, focus on impact rather than urgency. Shift left towards prevention, and reward the work that gets you there. Rarely is anyone thanked for the work they did to prevent the disaster that didn’t happen. Well let’s all try to make it a little less rare.
