A sanctions screening vendor selection method should answer one question before any contract is signed: how well does this engine match the names your institution actually holds? Most selections never answer it. Vendors present polished demonstrations on synthetic test data, and that data rarely reflects your name patterns, the transliteration challenges of Arabic names rendered in Latin script, or the demographics of your customers.
This article sets out a five-week approach for AML and Compliance Heads who are replacing or implementing screening technology. It is a structured way to measure accuracy on your own portfolio. It sits inside our wider AML Compliance Solution, which covers vendor-independent selection and implementation of AML technology.
A note on scope. The AML Compliance Solution as a whole runs as a project of 4–9 months, and follows four phases: define, select, implement and validate. The five weeks described here are a focused way of running the selection work, with the screening engine as the subject. The weekly structure below is the method we apply to that selection.
Why synthetic demonstrations mislead
A demonstration shows what an engine can do under favorable conditions. It does not show how it behaves with compound Arabic names, variant spellings of the same name, names with several valid transliterations, or the mix of individuals and entities in your book. Two engines that look equivalent on a vendor dataset can behave very differently on yours.
The consequence is practical. An engine that over-alerts floods your analysts with false positives. An engine that under-alerts misses true matches, which is a regulatory and reputational exposure. You cannot see either problem until you test with realistic data.
Week one: extract the requirements
What happens. We translate your obligations into testable requirements. The sources are the FATF standards on targeted financial sanctions and the SAMA AML/CTF requirements, together with the sanctions regimes your institution must apply: UN, local, and, where relevant to your business, OFAC and EU. We also record operational needs: which customer and payment types are screened, when screening runs, how alerts are escalated and how decisions are evidenced.
Who takes part on your side. The AML or Compliance Head, the sanctions or screening lead, a representative from operations or onboarding, and IT for integration constraints.
What you hold at the end. A requirements specification covering screening, and where you are also selecting wider AML technology, risk scoring, monitoring and case management. Each requirement is written so that it can be scored later.
Week two: prepare the test dataset
What happens. We build the test dataset from anonymized records of your customer portfolio. The aim is a representative sample: individuals and entities, common and uncommon name structures, and names that are known to cause transliteration difficulty. We then add known list entries and controlled variations of them, such as spelling changes, reordered name parts and omitted elements, so that true matches are known in advance. Knowing the correct answer is what makes accuracy measurable.
Who takes part on your side. Compliance, data owners and your data-protection function. Anonymization must be agreed before any data leaves your environment, and it should be handled in line with your data-protection obligations.
What you hold at the end. An approved, anonymized test dataset with documented construction rules, and a defined set of expected matches against which every engine will be judged.
Week three: test the vendors in parallel
What happens. Shortlisted vendors screen the same dataset against the same lists under the same conditions. Running them in parallel removes the advantage of a tailored demonstration and makes results directly comparable. Vendors configure their engines within agreed limits, and we record the settings used so results can be reproduced.
Who takes part on your side. Your screening lead observes the runs, and IT supports any environment or data-handling arrangements. Vendors are not given the expected-match key.
What you hold at the end. Raw, comparable result sets for each vendor and a record of the configuration each one used.
Week four: analyze false positives and false negatives
What happens. We compare each result set with the expected matches. Two questions are answered for every engine:
- False negatives: which true or deliberately varied matches did it fail to alert on, and why?
- False positives: how many alerts would your analysts have to clear that were not real matches, and what patterns drive them?
We look at causes as well as counts. An engine that misses a particular transliteration pattern may be fixable by tuning; one that misses it structurally may not be. Your analysts review a sample of alerts so the findings reflect how they would work in practice.
Who takes part on your side. Sanctions analysts and the Compliance Head, who decide what level of residual risk and alert workload is acceptable.
What you hold at the end. An accuracy analysis per vendor, built on your own data, that the board or a regulator can follow.
Week five: evaluate commercial and integration fit
What happens. Accuracy is necessary but not sufficient. We score each vendor on integration with your core and onboarding systems, handling of list updates, alert workflow, audit trail, and the commercial terms of a multi-year agreement. Accuracy results and these criteria are combined into a single scored evaluation against the requirements set in week one, so the recommendation can be explained and defended.
Who takes part on your side. Compliance, IT and architecture, procurement and, where relevant, legal and finance.
What you hold at the end. A vendor evaluation report and recommendation that shows how each vendor scored and why one is recommended.
What happens after selection
Selection is not the end of the work. In our AML Compliance Solution the vendor implements the software. We define the requirements, oversee the implementation and validate on your behalf, which gives you independent assurance. The later phases cover data integration, scenarios and risk models, followed by pre-go-live validation and a governance pack. If you are replacing a current system, migration of history, rules and open cases is part of the scope.
For a broader view of your obligations, a sanctions compliance program review covers UN, local, OFAC and EU regimes, including screening, trade and payment controls, escalation and reporting. Customer due diligence quality also affects screening results, because poor customer data produces poor matching.
Full list of deliverables
The AML Compliance Solution produces the following:
- Requirements specification
- Vendor evaluation report and recommendation
- Scenario and model design documents
- Validation report and governance pack
Next step
If you are about to sign a multi-year screening contract, the most useful step is to test the candidates on your own names first. Speak with CISO Consulting about our AML Compliance Solution, and we will discuss how a selection built on your actual portfolio would work for your institution.
