Critical-infrastructure operators in the Kingdom increasingly ask a sharper question of their compliance programmes: does a high score mean the controls work? NCA ECC self-assessment control effectiveness testing is the discipline of answering that question with measurements rather than with documents. The pattern described here is one we commonly see in energy, utilities, petrochemical and transport-adjacent entities. It is presented as a typical pattern, not as the story of any single organization.
The pattern: "implemented" on paper, silent in the exercise
A typical operator prepares its self-assessment against the National Cybersecurity Authority (NCA) Essential Cybersecurity Controls. For the cybersecurity risk management domain, the team finds a risk methodology, a risk register, a monitoring procedure and an approved incident process. Each is dated, signed and version controlled. Each is rated "implemented".
Then a tabletop exercise is run with a realistic scenario: an attacker gains a foothold, moves laterally between network segments and stages data for exfiltration. The discussion exposes uncomfortable facts:
- Nobody can say how long the first suspicious authentication would sit unnoticed.
- Alerts exist, but analysts treat them as background noise.
- The escalation path on paper names roles that no longer exist or people who are not on call.
- Logs that would reconstruct the event are kept for a shorter period than the procedure states, or are not kept centrally at all.
None of this means the team was careless. It means the self-assessment asked whether a control existed, and the exercise asked whether it worked. These are different questions, and a scoring scale that does not separate them will overstate capability.
Why the gap arises
Three causes recur across sectors.
First, evidence requests drift toward artefacts. When a reviewer asks for proof, the easiest thing to produce is a policy or a screenshot of a tool configuration. Neither shows that the tool detects anything.
Second, ownership is split. Risk management sits with governance and compliance, while detection sits with security operations. The self-assessment is completed by the first group, and the operational reality lives with the second.
Third, controls are rated once, at a point in time. Detection rules age, staff change and networks are modified, but the rating stays where it was.
Four control-effectiveness tests
The following tests can be run by an internal team before a self-assessment is finalized. Each produces a measurable result that can be recorded as evidence and repeated.
1. Detection-time measurement
Define a small set of simulated behaviours that represent lateral movement and data exfiltration, such as an unusual administrative logon across segments or a large outbound transfer to an unfamiliar destination. Execute them in a controlled way, record the time of each action, and record the time the monitoring function first raised an alert and the time an analyst acknowledged it.
The result is a measured interval, not an assumption. Compare it with the detection and response targets your own procedures set. If no target exists, that is a finding in itself. If the measured time exceeds the target, or no alert fires, the control cannot honestly be rated as fully implemented.
2. False-positive rate
A detection that fires constantly is, in practice, a detection that is ignored. Take a representative sample of alerts from the relevant rules over a defined period and have analysts classify each as true positive, benign or unclear. Record the proportion of alerts that required no action and how long triage took.
This test answers whether the signal is usable. A rule with a very high noise level may technically detect the behaviour while making real detection unlikely. The remedy is usually tuning, suppression logic and better asset context rather than more tooling.
3. Escalation-path validation
Take the documented escalation path and walk it with a live, unannounced scenario. Contact the named roles through the stated channels and record who responded, how long it took, and whether each person knew what decision was theirs to make.
Typical failures include outdated contact lists, unclear authority to isolate a system in an operational environment, and no defined route to executive management or to notify the relevant regulator. Where operational technology is involved, confirm that the path includes plant or operations leadership, since containment decisions there carry safety consequences.
4. Evidence-retention check
Pick one simulated incident and attempt to reconstruct it from retained records alone. Identify which log sources are needed, confirm that each is collected centrally, confirm the time synchronization between them, and confirm that the retention period matches what your procedure and applicable requirements state.
If the reconstruction has gaps, the organization would also struggle to support an investigation, a regulator query or an internal review. This test also confirms that the evidence you cite in a self-assessment is real, retrievable and attributable.
Turning results into honest ratings
Once the four tests are run, the self-assessment can record more than a status. For each relevant control, note the test performed, the result, the date, the owner and the remediation action where the result fell short. A control with a policy but a failed detection test might be rated as partially implemented, with a dated plan.
This approach has two benefits. It gives the board a more reliable view of exposure, and it creates a baseline that can be repeated at each cycle. Improvement then becomes visible as a trend, such as shorter detection times or lower noise, rather than as a new document.
A few practical points help adoption:
- Run the tests with the people who operate the controls, not only those who document them.
- Agree the pass criteria before testing, so results are not reinterpreted afterwards.
- Coordinate simulations with operations teams, especially in environments where availability matters.
- Keep the outputs as evidence alongside the policies they relate to.
A note on ECC-2:2024
Organizations that complied with ECC-1:2018 often ask what changes under ECC-2:2024. The structure and several requirements changed. We map your existing compliance to the new controls so you only address what is actually new or different. This is also a good moment to introduce effectiveness testing, because the control mapping is already being revisited.
How we can help
Our NCA Essential Cybersecurity Controls service covers compliance assessment, remediation and self-assessment support, with readiness for ECC-2:2024 reviewed control by control. The assessment takes 6–10 weeks. It produces a control-by-control compliance status with evidence, a remediation plan with owners, dates and effort, and a baseline you can re-measure every year.
Deliverables include a compliance assessment report, a remediation plan and tracker, a policy and procedure set, an executive summary and support with the self-assessment submission. If your Domain 3 ratings have not yet been tested against real detection and response behaviour, that is a sensible place to start the conversation.
