GSX Learnings: Yes, Even Security Should Embrace Failure

By Mike Gips, CPP

At GSX 2022, I moderated a panel called Learning From Failure. Featuring renowned security practitioners Jeff Slotnick (Setracon), Antoinette King (Credo Cyber Consulting), and Ricky Davis (RICE Security and Consulting), the presenters shared some of the most painful failures in their career, but also explained how those letdowns became professional turning points or led them to wisdom and success they wouldn’t have obtained otherwise. We then offered guidance on how to turn failure to success.

Other industries welcome, even embrace, failure. Not security. Amazon founder Jeff Bezos famously blessed innovation through failure when he said, “If you are going to take bold bets, they’re going to be experiments. And if they’re experiments, you don’t know ahead of time if they’re going to work. Experiments are by their very nature prone to failure.”

Take Amazon’s delivery drones. It’s been at almost 10 years since Amazon promised them, but recent crashes in testing have delayed the rollout. Fortunately, the crashes have not caused any injuries.

News like that makes us security practitioners practically tremble at the word failure. We’ve been trained that no news is good news. When we think of failure, we often go to worst case scenarios: an active assailant that got past our officers; a background check that didn’t flag a fraud artist in our midst; a hack that cost the company invaluable proprietary information and incalculable reputational loss. Or a delivery drone injuring a child.

Our panel emphasized that we don’t have to think about success and failure as all or none. Obviously, we don’t want to fail when the stakes are high—a TSA officer failing to detect explosives that take down a plane is obviously unacceptable—but as long as we limit the potential consequences, failure can be our friend. In fact, it could lift us up to greater levels than we would have reached otherwise. After all, Amazon’s drone crashes have been occurring during testing. That’s the time to fail, so their drones can soar higher, metaphorically, at least.

In our session, Jeff shared a profound learning experience from his days stationed in the U.S. Army in Europe. He was tasked with writing a nuclear spill response plan for an Army civilian military engineer, which he labored on over an IBM Selectric typewriter. Two days after he turned it in, the civilian called Jeff into his office and handed him the document—which was “bleeding green from his felt-tip marker,” Jeff recalls. “I threw the report on his desk and said, ‘If you think you can do it better, then you do it,’ and I turned to leave. He called me back in a tone I was not used to hearing from a civilian and read me the Riot Act.”

By swallowing his pride, Jeff transformed his career and life. “The skills he taught me in writing, management, and leadership have lasted a lifetime,” Jeff says, starting with three Army promotions. He uses those skills today to write standards, prepare reports, mentor executives, and lead teams. Best of all, decades later he remains friends with his one-time nemesis.

Antoinette’s failure came from the opposite problem: not having enough confidence. She told the audience about her entry into the security field as an installation technician. “Unbeknownst to me, it was highly unusual for a woman to be a technician pulling cable, installing devices, and building head ends,” she says. As the only woman on a typical job, she would blend in or try to become invisible. “For the next several years I did everything in my power not to be seen. This resulted in many missed opportunities.” She eventually realized that her differences made her valuable, and today she spends time mentoring women in technology and ensuring that they don’t minimize themselves like she did.

Probably the most inspirational parts of the session occurred when we invited audience members to share their stories of failure—whether they led to redemption or not. One attendee related how he had recently been turned down for a prestigious credential, but the feedback he received in the process showed him that he needed to evolve from an operational to strategic mindset. 

Ricky, Antoinette, and Jeff then discussed how to grow from setbacks, such as by acknowledging failure, accepting responsibility, pausing and reflecting, seeking advice and criticism, extracting lessons, keeping perspective, making incremental changes, staying positive, and taking care of yourself. They then turned toward a more clinical approach to overcoming failure, exploring topics such as process inadequacy, task challenge, process complexity, and hypothesis testing. 

Today’s security professionals are risk managers. Though we manage risk and usually don’t try to eliminate it, risk gives us anxiety. But we also know there is no reward without risk. So how do we adjust our risk tolerance to accept failure?

Akshay Bhargava, Chief Product Officer at Malwarebytes, developed a philosophy called Failing Toward Zero, and it can work for security professionals of all types. He writes that “Failing toward zero is a state in which each security incident leads to a successive reduction in future incidences of the same type.” In corporate security, this may mean reducing the number of tailgating incidents, security policy violations, or incidences of theft successively over time. It involves identifying the source or cause of the failure and remediating it, iteratively improving security and yielding better results. But be careful not to focus on the results alone. Sometimes good processes yield bad results and bad processes yield good results. It’s improving the process that’s key. In short, test, tweak, and test again.

Michael Gips, JD, CPP, CSyP, CAE, is the Principal of Global Insights in Professional Security, a consultancy focusing on security thought leadership, content, strategy, research, insights, and influence within the profession. Reach him on LinkedIn.