Most fraud teams in the Kingdom can tell you how many alerts their monitoring system raised last quarter. Far fewer can show whether each rule is worth keeping. This article covers the SAMA CFF fraud rule tuning metrics that turn a quarterly review from an alert count into evidence: five measures, how to calculate them from fraud-queue data, and the point at which a rule should be rebuilt rather than adjusted.
A note on scope. SAMA's Counter-Fraud Framework expects institutions to keep their fraud controls effective and to review them periodically. We describe the review in practical terms and do not rely on a specific article number. Confirm how your own compliance function maps the evidence to the framework text.
Why alert counts are not enough
An alert count says how busy the team was. It does not say whether the rules found fraud, missed it, found it late, or consumed analyst time on customers who were never at risk. A reviewer reading a report that contains only volumes cannot tell a healthy rule from a failing one.
A defensible review needs data from three places: the alert queue, the confirmed-fraud register, and loss records. Each alert needs a disposition (confirmed fraud, false positive, or unresolved), a timestamp, and a link to the rule that raised it. Where fraud was confirmed without an alert, the case needs a record too. That last set of cases is the one most often missing.
The five metrics
1. Precision
Precision is the share of a rule's alerts that end as confirmed fraud.
Precision = confirmed-fraud alerts ÷ total alerts closed for the rule
Calculate it per rule, not for the whole system. Count only closed alerts, and report unresolved ones separately so they do not flatter or damage the figure.
2. Recall
Recall is the share of known fraud that the rule, or the rule set, caught.
Recall = confirmed fraud cases alerted ÷ all confirmed fraud cases in the same scope
The denominator comes from the confirmed-fraud register, including customer-reported cases and chargebacks. Without it, recall cannot be measured, which is why it is the metric most often skipped.
3. Detection lag
Detection lag is the time between the fraudulent transaction and the alert that flagged it. Measure the median and the slowest cases for each channel. A rule with good precision that fires hours after funds leave through an instant payment channel has limited protective value.
4. False-negative leakage
Leakage is the value and count of confirmed fraud that no rule alerted on, grouped by channel and typology. It shows where the rule set has gaps, for example in authorised push payment scams or account takeover, where transactions can look normal on their face.
5. Business-impact concentration
Impact concentration shows how much of the fraud loss prevented, and how much analyst effort spent, sits in a small number of rules. Rank rules by loss prevented and by alert-handling hours. Rules that consume large effort and prevent little are candidates for change. Rules that prevent most of the loss deserve the closest protection from careless tuning.
Calculating the metrics from queue data
A workable sequence for a quarterly cycle:
- Export all alerts for the quarter with rule identifier, channel, timestamp, disposition and closure time.
- Export the confirmed-fraud register for the same period, including cases found through customer complaints, chargebacks and investigations.
- Match each confirmed case to an alert where one exists. Cases with no match feed the leakage metric.
- Calculate precision, recall and detection lag per rule and per channel.
- Rank rules by loss prevented and by handling effort for the concentration view.
- Compare each figure with the previous quarter and with the targets your fraud committee has approved.
Keep the extraction queries and the matching logic documented. A reviewer should be able to reproduce your numbers.
When to tune and when to rebuild
The brief for this article refers to a threshold that triggers re-calibration. We do not offer a universal number, because the right alert-to-confirmed-fraud ratio depends on your products, channels and analyst capacity. Instead, set your own trigger and apply it consistently. Our suggested approach is to approve the trigger at committee level, record the reasoning, and test it against your own history.
Tuning a threshold is appropriate when:
- the rule logic still matches a live typology;
- precision or recall has drifted gradually;
- the problem is concentrated in a customer segment or amount band that a parameter change can isolate.
Rebuilding is the better choice when:
- the alert-to-confirmed-fraud ratio stays below your approved trigger after two consecutive threshold adjustments;
- the typology has changed, so the rule is looking for behaviour fraudsters no longer use;
- leakage shows that the fraud sits outside what the rule can see, for example missing data fields or an uncovered channel;
- detection lag is structural, caused by batch processing or data feeds rather than the threshold.
Repeated adjustment of a rule that fails these tests usually trades false positives for missed fraud without improving either.
Documenting the review
For each quarter, the report should hold the five metrics per rule and channel, the trend against prior quarters, decisions taken (keep, tune, rebuild, retire), the owner and date for each action, and evidence of back-testing before any change goes live. Approval records matter as much as the numbers, because they show governance rather than a technical exercise.
Signs it is time for outside help
- Your quarterly report contains alert volumes but not recall or leakage.
- Analysts report that most alerts on certain rules are closed without action.
- Fraud losses are rising in channels where rules have not changed for a long time.
- Confirmed-fraud cases cannot easily be matched back to alerts.
- Your vendor controls configuration and nobody internal can explain why a rule fires.
How we run this work
Our Fraud Detection and Transaction Monitoring Optimization service reviews and tunes the rules, scores and workflows of your fraud monitoring system across cards, mada, SARIE, instant payments and digital channels. It is delivered as a project of 6–10 weeks, mapped to SAMA CFF and PCI DSS.
The work follows four stages: Measure, Analyze, Tune and Embed. In the first we look at alert volumes, hit rates and losses by rule. We then analyze gaps against current typologies and channels. Tuning covers thresholds, logic and new scenarios, tested on history. Finally we embed procedures, metrics and handover so your team can repeat the review each quarter.
The deliverables are a detection coverage map, a tuned rule set with rationale, an alert quality metrics baseline and operating procedures for the fraud team. The scope includes rule and scenario effectiveness review, false-positive reduction, coverage of new typologies including APP and account-takeover fraud, and alert handling workflow and SLAs.
Two practical points often come up. We work with any rule-based or score-based system, and with your vendor where configuration access is restricted. And every change is back-tested on your history first, with false-positive reductions balanced against detection, so tuning is not allowed to increase losses unnoticed.
What a good outcome looks like
A good outcome is a quarterly report that a regulator, auditor or board member can follow: five metrics per rule, a documented trigger for rebuilding, and a clear record of decisions. The fraud team spends less time on alerts that go nowhere and more on cases that matter.
Next step
If your quarterly review still rests on alert counts, a baseline is the sensible starting point. Speak to CISO Consulting about Fraud Detection and Transaction Monitoring Optimization, and we will discuss how the Measure stage would apply to your monitoring environment.
