Real-Time Fraud Detection on a Legacy Core
Score 50,000 transactions a day from a legacy Oracle core without touching it - change data capture, sub-200ms scoring, and decisions written to a separate store.
Last updated:
A bank processes 50,000 transactions a day in a legacy Oracle core and wants AI fraud detection without touching Oracle. That constraint is the design: no triggers, no schema changes, nothing synchronous on the core's commit path. You read the change log, build features and score outside the core, and write decisions to a store you own. The volume is small - the engineering is about how you get the data out, how fast you decide, and how you prove the core was never modified.
“Without touching Oracle is not a preference, it is the acceptance criterion. Read the redo log, score outside, write decisions to your own store. The only change on their side is a read-only CDC user and supplemental logging - put that in writing, because it is what gets the project approved.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| Is 50,000 a day the average or the peak? | Under 1 TPS average, but end-of-day batches spike hard - size for the peak |
| Score inline before settlement, or after the fact? | Blocking a payment and flagging it for review are different systems |
| Latency budget for an inline decision? | Sub-200ms is a hard ceiling on model size and feature lookups |
| Can we add a CDC user and enable supplemental logging? | If not, the fallback is a polled read replica with a watermark |
| Cost of a false positive versus a missed fraud? | Sets the threshold - and the business owns that number, not the model |
| Is there an analyst review queue? | Precision targets differ enormously with a reviewer in the loop |
| Any regulatory explanation duty (adverse action, audit)? | Forces reason codes stored with every decision |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1CDC, not polling and not triggers - Debezium reads the redo log, so the core sees a log reader rather than load on the transaction path. A trigger puts your latency and your failures inside their commit; polling on an updated_at column misses deletes and out-of-order commits, and clock skew drops rows silently.
- 2Partition by account, not by transaction - Fraud features are per-account sequences - velocity, deviation from that account's own baseline. Partitioning by account_id keeps an account's events ordered on one consumer and keeps stateful windows local instead of shuffling.
- 3One feature definition, materialised twice - The classic failure is training on a 30-day average computed in SQL and serving one computed in Flink with a different window boundary. Define the feature once and materialise both the online and offline copies from the same job.
- 4Rules in front, model behind - Hard rules - sanctions hit, impossible geography, card present in two countries within five minutes - run first and are deterministic, auditable and changeable in minutes. The model covers what rules cannot express. Most production fraud stacks are still mostly rules by volume.
- 5Reason codes are a requirement - Every decision stores its top contributing features or the rule ids that fired. An analyst working the case, a customer disputing a block and a regulator asking why all need the same record, and it cannot be reconstructed later.
- 6Decisions live in your store, never in theirs - The core stays the system of record for money; you own the system of record for decisions. Reconcile daily by txn_id and alert on gaps, because a transaction that was never scored is a silent failure rather than a loud one.
What Separates a Strong Answer
- 1Use CDC over polling - A strong candidate uses CDC to avoid the pitfalls of polling, such as missing deletes or out-of-order commits, ensuring reliable and low-latency data ingestion.
- 2Partition by account - Partitioning by account_id maintains event order per account, which is crucial for accurate feature computation and prevents unnecessary data shuffling.
- 3Dual feature storage - Storing features both online and offline from the same definition prevents discrepancies between training and serving environments, ensuring consistent model performance.
- 4Hybrid decision path - A combination of rules and models allows the system to handle both deterministic and complex fraud patterns, optimizing for both speed and accuracy.
Why Not Simply Query Oracle?
| Approach | Problem |
|---|---|
| Trigger on the transaction table | Puts your latency and your outages inside their commit path - DBAs will refuse, correctly |
| Polling a read replica on updated_at | Misses deletes, misses out-of-order commits, and clock skew drops rows |
| Nightly batch export | The decision arrives a day late, after the money has moved |
| CDC from the redo log | Read-only, low overhead, ordered and replayable - the standard answer |
Latency Budget for a 200ms Inline Decision
| Stage | Budget |
|---|---|
| CDC to stream | 20ms |
| Feature lookup (Redis, batched) | 15ms |
| Rules engine | 5ms |
| Model inference | 30ms |
| Decision write (asynchronous, off the path) | 0ms |
| Headroom for p99, network and GC | 130ms |
3 Biggest Risks
- 1Label delay - Confirmed fraud arrives weeks later as chargebacks, so today's model is trained against a stale definition of fraud. Mitigated by using analyst dispositions as an early proxy label and retraining on a schedule once chargebacks land.
- 2Class imbalance and adversarial drift - Fraud is well under one percent of volume, so accuracy is meaningless, and fraudsters adapt to your rules within weeks. Mitigated by tracking precision and recall at a fixed alert budget, plus feature-drift monitors per segment.
- 3Replay correctness - A consumer restart must not double-score or skip transactions. Mitigated by idempotent writes keyed on txn_id and committed offsets, so a replay converges to the same decision set rather than a different one.
Google Stack: Datastream (CDC from Oracle) → Pub/Sub → Dataflow (windowed features) → Memorystore for online features + BigQuery for offline → Vertex AI endpoint (scorer) → BigQuery decision store → Looker for the analyst queue
Azure Stack: Debezium or Oracle GoldenGate on AKS → Event Hubs (Kafka protocol) → Azure Stream Analytics or Flink on HDInsight → Azure Cache for Redis (online features) → Azure ML managed endpoint → Azure SQL decision store → Azure Monitor
AWS Stack: AWS DMS or Debezium on ECS → Amazon MSK → Managed Service for Apache Flink → ElastiCache + SageMaker Feature Store → SageMaker real-time endpoint → DynamoDB decision store → CloudWatch + a case UI on ECS
Other Options: Debezium + self-managed Kafka + Flink + Feast + XGBoost served by BentoML for a fully portable stack | Buy instead of build - Stripe Radar for card payments, Feedzai or Amazon Fraud Detector - which is a reasonable answer at this volume and worth raising.
Frequently asked questions
50,000 a day is tiny. Why Kafka at all?
Throughput is not the reason, and say so first. The reasons are replay - reprocessing a month when a feature was wrong - decoupling, so adding a second consumer never touches the core, and per-account ordering. If none of those were required, a queue and a scheduled job would genuinely be enough.
What happens if the model is down?
Fail to the rules engine and mark the decision as degraded, so the analyst queue knows which decisions were made without a score. Never fail open silently, and never let the failure propagate into the core - the core has no dependency on you by design.
How do you backfill features for a new model?
Run the same feature code in batch over the warehouse copy of the CDC stream. That is precisely why raw events are retained rather than only the aggregates - aggregates encode today's feature definitions and cannot be rewound.
How do you prove you never touched Oracle?
The change list on their side is a read-only CDC user and supplemental logging, both reviewable. No triggers, no DDL, no synchronous calls. Document it, have the DBA team sign it, and monitor redo-log reader lag so you can show the overhead you actually impose.
Would you use online learning?
No. An adversary can then poison the model with cheap probing transactions. Retrain on a schedule with reviewed labels and ship through the same evaluation gate each time, with the model version recorded on every decision.
Where does an LLM actually fit here?
Not on the 200ms path. It belongs in the analyst's case view - summarising recent account behaviour, drafting the case narrative, clustering similar cases. Saying the scorer should not be an LLM is a point in your favour, not against you.