Ransomware and crippling cyberattacks are so pervasive they have permeated our social fabric down to the late night comedy. HBO’s Last Week Tonight with John Oliver covered the subject of ransomware and even how cybercriminals race their Lamborghinis through the streets of Moscow, lavishing their protection within the borders of their native country, Russia in this case.
The majority of these crippling cyberattacks begin with a simple phishing email or business email compromise (BEC) scam. While most companies provide some form of annual cybersecurity awareness training to reduce the likelihood of employees being hooked, they focus on overly simplistic lures taken from public events that fail to represent the real danger of targeted cyberattack campaigns.
What’s worse is they rely on disengaged metrics like employee attendance and simple test results as a proxy to measure business risk reduction. However, there is little to no connection between the two. In fact, this shortsighted approach creates a false sense of security which only exposes the business further to cyber risk.
Stepping back for a moment, the purpose of phishing and security awareness training (PSAT) is to drive behavioral changes amongst your employees and the corporate ecosystem to improve business resiliency through reduced exposure to related cyber risks, alleviated resource constraints, and increased ability to meet regulatory compliance requirements.
So, why would you look at attendance lists and the so-called “smiley sheets” to assess business resiliency? This is why most programs fail–they don’t represent the cyber risks, they don’t provide the tools and skills necessary for employees to identify cyberattacks, and they focus on the wrong measures.
Plus, think about the wasted effort and costs. Whether you see it or not, there is a cost associated with your employees sitting in the PSAT classroom sessions. That salary easily eclipses the direct expenses associated with PSAT training sessions, such as the cost of lunch or even the speaker.
So how do you know your PSAT program is working?
Successful PSAT programs move beyond the overly represented focus on compliance (or as I like to call it, the Ten Commandments of IT). They provide contextual phishing examples, empower employees to protect themselves and the business, and provide clear threat reporting mechanisms.
When it comes to measuring and reporting PSAT effectiveness there are two models you can explore: the SANS Security Awareness Roadmap and the Kirkpatrick model.
The SANS Security Awareness Roadmap
The SANS Security Awareness Roadmap provides a tiered approach to maturing your program similar to other maturity models, in that it helps your organization move from a state of disregard for cyber risk and towards a more integrated, team approach:
Level | Focus | Leadership Approach | Employee Behavior |
1 | No program | Leadership does not recognize the cyber risks nor value of PSAT | Employees are oblivious and do not discuss cybersecurity |
2 | Compliance | Leadership understands cybersecurity obligations and risks | Employees do not understand cybersecurity, and attend PSAT as unproductive obligation |
3 | Behavior | Leadership understands cybersecurity obligations and risks and funds the program | Employees understand the risk and can identify threats and proactively report potential cyber threats |
4 | Culture | Long-term commitment to PSAT with metrics-driven decision making | Employees are empowered to make recommendations and proactively seek training and report cyber threats. |
5 | Outcomes | Leadership aligns business objectives, cyber risk prioritization, and PSAT resources | Employees are empowered to make recommendations and proactively seek training and report cyber threats. |
This is a good framework that can help you determine where your company is in terms of maturity, and identify the gaps you need to fill to move up the safety stack.
The Kirkpatrick Model
The Kirkpatrick model is a long-standing standard for validating the efficacy of investments in training and learning. Like the SANS Security Awareness Roadmap, the Kirkpatrick model consists of four levels of maturity and indicators, focusing on Reaction, Learning, Behavior, and Results. Below, I’ve provided the levels and their focus, what they evaluate, and some specific examples of how to measure PSAT success.
Level | Focus | Evaluation | Examples to Measure PSAT Success |
1 | Reaction | The degree to which participants find the training favorable, engaging, and relevant to their jobs. | Subjective feedback forms to assess learner engagement, instructor performance, and content usability or format. Passive metrics collected from online learning systems like % or # of employees trained and pass/fail metrics.
|
2 | Learning | The degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training. | Open rates measure detection based on Inbox preview data. Click rates measure detection once opened and read. Surrender rates measure user credentials given over.
|
3 | Behavior | The degree to which participants apply what they learned during training when they are back on the job. | Report rates measure the number of suspicious lures reported using this mechanism. Engagement rates measure subsequent communications with the cybersecurity team. Policy rate measures the number of policy violations. Regulatory rates measure the number of regulatory or legal violations.
|
4 | Results | The degree to which targeted outcomes occur as a result of the training and the support and accountability package. | Losses to fraudulent financial transfers. Losses based on cyberattacks. Costs of data breaches. Operation savings based on optimizations or reduced workload.
|
All too often, program coordinators proudly display their “smiley sheet” results and tout the number of employees trained as a measure of their value. Yet these results only measure subjective response, which in no way indicates the trainees’ ability to identify a cyber threat and avoid falling for the cybercriminal’s intent.
When asked which metric I depend on most, I always say, “Engagement”. It’s the most qualitative measure and it tells you that you’ve got through to your employees.
Ask them why they reported the suspicious email to see what indicators triggered for them since this will tell you that your program works when it comes to empowering employees to protect the business:
Was it an “external” flag on an email that looks like it comes from an employee colleague?
Was it because the sender’s email signature or tone seemed insincere or unusual from a known colleague?
Was it that the sender’s ask was out of bounds?
The second metric I look at is the time employees proactively (unrequested) spend with the IT/Security team. This action shows passion, which in turn shows their devotion to cybersecurity and their desire to protect the business.
Make it personal and make it worth it
When it comes to engaging employees, metrics can measure your success but they normally don’t motivate attendance and participation. Nor do compliance requirements. We’ve all taken cybersecurity awareness courses with the class window in the background as we do our “real work”.
One of the best examples I’ve seen of a successful phishing and security awareness training program is a company that created tailored campaigns, including whimsical posters, swag, and a competition ladder of escalating phishing testing to crown a cyber champion. It’s an innovative approach that led to record attendance, motivated employees, and reduced operational costs. It also meant their company reduced the risk of phishing.
Remember, just because cybersecurity is a serious topic, doesn’t mean your company can’t have fun!
How our Phishing and Security Awareness Training program can help
Designing, developing, and executing a fun–yet, educational–PSAT program is hard. Checking a box for compliance is easy, but moving the needle takes more leverage and more force. The better the PSAT program, the more effort is required since the complexity increases.
eSentire’s Managed Phishing and Security Awareness Training puts your organization on the forefront of user protection against the latest social engineering cyberattacks.
Our end-to-end service alleviates the resources required to operationalize an effective user resiliency program. By leveraging software paired with dedicated social engineering experts, we ensure that your users are continuously tested and hardened against even the most sophisticated phishing attempts. Our robust phishing libraries consist of hundreds of templates, ensuring users are tested against real-world scenarios, not commoditized and easy to spot templates.
More importantly, our PSAT training modules are automatically assigned to users that fall victim to simulated testing, ensuring education is integrated at the moment of failure and drives sustained behavioral change. Our PSAT program generates measurable results, extending well beyond checking a compliance box. We not only help you meet regulatory requirements, but also ensure that your organization is resilient against the latest social engineering tactics.
To learn how eSentire’s Managed Phishing and Security Awareness Training can help drive behavioral change with your employees across your organization, book a meeting with a cybersecurity specialist today.
