Security awareness phishing simulation metrics Saudi institutions report to executives are often limited to one figure: the click rate, produced once or twice a year. That number is easy to collect and hard to defend. It does not show whether employees recognise threats as attacks evolve, and an annual cadence hides the trends a board actually needs to see.
This article sets out five metrics to review each quarter and a scenario-rotation pattern that keeps the measurements honest.
Why click rate alone fails
A click rate moves for reasons unrelated to skill. A simple template, a message sent at a quiet hour, or employees who have learned that "the simulation email looks like this" will all change the number. When staff memorise the template, the rate improves while real detection ability may not.
Annual measurement adds a second problem. A change in results cannot be linked to a training action, a new attack pattern or a staffing change, because too much time has passed between readings.
When an organisation needs this
Regulated institutions in the Kingdom work under expectations from SAMA, whose Cyber Security Framework addresses awareness and training, and from the NCA, whose Essential Cybersecurity Controls include awareness and training requirements. Both expect a programme that is run and evidenced, not merely scheduled.
Signs it is time to change the measurement approach:
- The same click-rate figure is reported every cycle and leadership has stopped asking questions about it.
- The simulation library has not changed in a year, and staff discuss the exercises openly.
- The same individuals fail repeatedly and nothing different happens to them.
- No one can say how quickly a suspicious message is reported once it arrives.
- Finance and executive-assistant roles have never been tested on payment-redirection scenarios.
The five quarterly metrics
1. Repeat-clicker cohort size
Track how many employees have failed more than one simulation within a defined window, and whether that cohort is shrinking. A stable or growing cohort means the training is not reaching the people who need it. Report the cohort by role type, not by name, and pair it with a targeted follow-up action.
2. Time-to-report improvement
Measure the time between delivery of a simulated message and the first report to the security team. Reporting speed matters in a real incident because it shortens the period in which a malicious message is live. Track the median and the slowest quartile, and compare each quarter with the last.
3. Scenario-variant performance delta
Run the same underlying lure in two or more variants, such as a different sender, tone, channel or pretext, and compare performance. A large gap between variants suggests employees are reacting to surface features rather than applying detection principles. A small gap suggests the learning is transferring.
4. Business-email-compromise recognition rate
Business email compromise rarely contains a link, so click rate cannot capture it. Test scenarios such as invoice changes, executive requests and supplier bank-detail updates, and measure how many recipients verify through a separate channel or report the message. This is the metric most relevant to finance and procurement teams.
5. Cross-department performance variance
Compare results across departments and look at the spread, not only the average. A good overall figure can conceal one function with consistently weak recognition. Variance points to where targeted content, manager involvement or role-specific scenarios are needed.
The scenario-rotation pattern
The aim of rotation is to change surface features while keeping the underlying detection principle constant. A workable pattern has four parts.
- Fixed principle, changing presentation. Each quarter, select a small set of detection principles, such as urgency, unusual payment instructions, or sender mismatch. Build new scenarios around each, varying brand, language, channel and difficulty.
- Anchor scenarios. Keep one or two comparable scenarios across quarters so that trend lines remain valid. Everything else rotates.
- Paired variants. Within a quarter, send variants of one lure to different, randomly assigned groups. This supplies the data for the variant delta metric.
- Staggered timing. Avoid sending to the whole organisation at once. Staggering reduces the chance that early recipients warn colleagues and distorts results.
Arabic and English versions should both be used where the workforce reads both, since recognition in one language does not guarantee it in the other.
What a good outcome looks like
A well-run programme produces a short quarterly report that an executive committee can read in minutes. It shows the repeat-clicker cohort falling, reporting times shortening, a narrowing gap between scenario variants, stronger business-email-compromise recognition in finance roles, and a smaller spread between departments. It also explains where results did not improve and what action follows. That candour is what separates a credible programme from compliance theatre.
Governance and ethics
Simulations should be approved by HR and legal functions, and results should be used to direct support, not to punish. Reporting by cohort and role protects employees and still gives leadership what it needs. Data from simulations is also personal data, so retention and access should follow the Personal Data Protection Law and the organisation's own policies.
How we approach it
Security Awareness at CISO Consulting is described as customized training programs to build a security-first culture across your organization. In practice that means starting from your own roles, risks and reporting obligations rather than a generic template, and aligning the measurement approach above with what your executives and regulators expect to see.
If your programme reports a click rate but cannot yet show whether staff are getting better at recognising real threats, our Security Awareness service is the place to start the conversation.
