SAMA AML transaction monitoring model validation often starts at the wrong place. The team receives the scenario inventory, opens the threshold settings and begins measuring how many alerts fire and how many true events are missed. When results look poor, the finding reads as a weak scenario. Frequently the logic is sound and the inputs are not: customer-risk scores are blank, transaction attributes are mis-mapped, or a feed dropped fields during extraction.
This article gives a same-day checklist. It sets out five data-quality checks to run before any threshold accuracy testing, followed by three field-completeness baselines for judging whether missing customer-risk scores are distorting your false negatives. We do not assume any particular share of false negatives is caused by missing scores; that is something your own baselines will show.
Why data comes before thresholds
SAMA's AML/CTF framework expects institutions to operate monitoring that is risk-based and effective, and an independent validator will ask whether the monitoring works as intended. A scenario that cannot see a customer's risk rating, a counterparty country or a channel code cannot apply its logic as designed. Testing thresholds on that data measures the data, not the scenario.
Separating the two gives you cleaner findings, a more defensible tuning case and a shorter conversation with the inspector.
Part 1: Five data-quality checks
Run these in order. Record the result of each with the extract date and the population tested, so the evidence can be reproduced.
Group A: Source and lineage
- Reconcile record counts from source to monitoring engine. Compare transactions in the core banking or payment platform with those arriving in the monitoring system for the same period. Why it matters: silent drops in a feed mean some activity is never scored at all.
- Document lineage for every field a scenario uses. For each scenario, list the input fields and trace each to its source system and transformation. Why it matters: undocumented transformations are a common cause of values that look populated but mean something different.
Group B: Completeness and population
- Measure completeness for each scenario-critical field. Count null, blank, default and placeholder values (for example "N/A", "0000" or a generic date) separately from true values. Why it matters: placeholders pass a simple null check yet carry no information.
- Test customer-risk-score population across the active customer base. Confirm that each active customer has a current score, and check how scores are treated when missing: excluded, defaulted to low, or defaulted to a neutral value. Why it matters: a default of "low" quietly places unrated customers into the least sensitive thresholds.
Group C: Attribute accuracy
- Sample transaction attributes against source documents. Check amount, currency, counterparty, country, channel and transaction-type codes on a sample against the originating record. Why it matters: a mis-mapped channel or country code can move activity out of the scenario that was designed to catch it.
Part 2: Three field-completeness baselines
Once the five checks are complete, set three baselines. They tell you whether missing customer-risk scores are a material cause of missed detection in your environment.
- Baseline one: risk-score completeness by segment. Calculate the proportion of customers with a valid score for each segment, such as retail, corporate, correspondent and money-service business. Why it matters: gaps are rarely uniform, and a high-risk segment with weak population is the priority.
- Baseline two: scenario-critical field completeness at alert level and at event level. Compare completeness for the population that generated alerts with the population of known suspicious cases or reported activity. Why it matters: if known cases show more missing fields than alerted activity, missing inputs are a likely contributor to false negatives.
- Baseline three: threshold-segment assignment with and without defaults. Re-run a sample twice, once with the data as held and once with missing risk scores replaced by the correct rating from a manual review. Count the transactions that move to a different threshold band. Why it matters: this is direct evidence of how much missing scores change outcomes, replacing assumptions with a number you can defend.
Record these baselines before any threshold change. If you tune first, you lose the point of comparison.
How to use the results
- Data defects dominate: fix the feed or the mapping, rerun the baselines, then test thresholds.
- Defects are limited and quantified: proceed to threshold testing, and state the limitation in the validation report.
- Data is clean: weak performance now points to a real scenario or threshold gap, and tuning recommendations can rest on it.
Keep the checklist outputs in the model documentation. An inspector will want to see that data fitness was established before performance conclusions were drawn.
Where Part 1 and Part 2 often fail
- Treating a populated field as a correct field.
- Testing a single month that does not include onboarding peaks or remediation cycles.
- Relying on a data dictionary rather than tracing actual values.
- Leaving the default treatment of missing risk scores undocumented.
How CISO Consulting helps
Our AML Monitoring and Screening Model Validation service provides independent validation of transaction monitoring scenarios and name-screening tools, covering data quality, thresholds, fuzzy-matching performance and coverage of your risk assessment. The work follows four stages: data lineage, completeness and quality testing; scenario coverage mapped to your risk assessment; threshold and matching-quality testing; and a validation report with model documentation.
The data stage is the one described in this article. The project runs 6–10 weeks, and the deliverables are a validation report, tuning recommendations with evidence, and a model inventory and governance pack. The work is aligned to SAMA AML/CTF, FATF and sanctions requirements.
On access, we work on extracts in your environment or a secure copy, so data never leaves your control. Validation includes evidence-based tuning recommendations, and we can support implementation.
If you are preparing for independent validation or a SAMA inspection, contact CISO Consulting to discuss scope.
