Blog

Something Went Bump in the Data

Megan Fantes

Published

Something Went Bump in the Data

Every machine learning model is a snapshot of yesterday. It learns the shape of the world from the data it was trained on, and then we ship it. From that moment, the world keeps changing. The model, and the data scientists, have to keep up.

We have a name for what happens next: drift. Internally, we like an even simpler definition — time passes, and things change. A partner adds a checkbox to a form. A new data source comes online, and the inputs get a little richer overnight. All of it leaves fingerprints on the model’s outputs.

This post is about how we listen for that drift on SentiLink’s fraud models, what the listening looks like in practice, and three real examples from the past year of what we’ve heard.

The most boring possible example of drift

Before any of the fraud stuff, here’s the simplest case we have. One of the features in our models is the age of a record in the Manifest, our internal database of identity history. The oldest records date from when we started collecting data. Every year, by definition, those records get one year older.

The whole population shifts one bucket to the right, exactly as the calendar would predict, right on time.

This is drift in its purest, most innocuous form, and it makes the first point cleanly: drift is not always bad. It’s just change. Our job is to figure out which changes are a problem, which are a signal, and which are a yawn.

Two flavors: feature drift and score drift

Changes show up in one of two places.

The inputs can change. Maybe a partner starts collecting an email field they didn’t previously collect, or our coverage on a particular data source suddenly improves. We call this feature drift.

The outputs can change. The distribution of scores we calculate moves — more mid-range scores, fewer high ones, a thinning of one tail. This is score drift, and it’s often a downstream consequence of feature drift, though not always.

We watch both every week. The summary statistics we use are called Stability Indices (Population Stability Index, or PSI, for score drift, and Characteristic Stability Index, or CSI, for feature drift). Simply put, these are single numbers that add up when two distributions diverge. Below 0.1 we call green. Between 0.1 and 0.2 is yellow — a polite cough from the data. Above 0.2 is red.

A weekly report rolls in. Most of the time, nothing’s there. Sometimes there’s a story.

Story 1: a required field changed everything

Last June, our drift report flagged a model we run for a partner. PSI had been quiet for months, and then it wasn’t.

Line chart of PSI over a rolling 4-week period from May to December 2025. PSI jumps from near 0 to about 0.037 at the beginning of June 2025 and stays between roughly 0.025 and 0.038 after

We had to worry if the model was getting worse, and if so, worse how?

We looked at the score histogram.

Histogram of score distributions, Q1 2025 baseline vs. June 2025. June has fewer scores in the 0-200 range and more mid-range scores, from about 200 to 700

The very-high and very-low scores held mostly steady. The middle of the distribution — roughly 200 through 700 — got noticeably fatter. The model was producing more “there’s some kind of signal here” answers than before. So we walked through the inputs, looking for a feature whose change matched the change in the outputs. We found it on the first try.

Histogram of years since an email was first seen, Q1 2025 baseline vs. June 2025. In the baseline over 50% of emails were missing; in June 2025 they are almost never missing and spread across all age ranges

The big blue bar on the right of the chart is what email history used to look like for this partner: more than half of applications came in without an email address. By June, that bar had nearly disappeared. The partner had quietly made a change to their application form.

A feature that used to be missing most of the time was now populated almost every time. From the model's perspective, that's a meaningful change in the world: an email address carries all kinds of signals that the model didn’t have access to before the model had learned how to confidently score applications without seeing an email address, and now it was more hesitant, because it was being handed information it had less practice digesting.

We talked to the partner. Other performance metrics — actual fraud caught, false positive rates — were still fine. The model wasn't broken, it was just being compared against a baseline that no longer existed. So we didn't replace or redesign anything, we reset the baseline. June 2025 is the new normal. From then on, when we asked "is anything drifting?", we asked it relative to the post-June world.

The takeaway: not every twitch on a drift chart is a model going bad. Sometimes it's the world becoming more informative than it was before.

Story 2: a bot attack, visible only in the histograms

Three months later, a different partner lit up the same monitor. This one looked different from the start.

Line chart of PSI from August 2025 to April 2026. PSI rises into the 0.1-0.2 alert range from September to December 2025, peaking around 0.15, then falls back to near 0 by April 2026

Yellow zone, persistent. We pulled up the score distribution.

Histogram of score distributions, Q2 2025 baseline vs. December 2025. A chunk of scores that used to sit above 700 moved down into the 500-650 range

Look at the blue then vs. the red now, where the blue used to be. There was a meaningful chunk of applications scoring above 700 — high risk, by our scoring convention. In October, that chunk evaporated and reappeared around 500–600. Our magic 8-ball had gone from “outlook suspicious” to “reply hazy, try again.”

Now we were paying attention. Into the features.

Two histograms, Q2 2025 baseline vs. December 2025. Left: years since email first seen, with more brand-new emails (0-1 years) in December. Right: years since phone first seen, with more aged phones (over 7 years)

Two charts, two stories. The first chart tracks how old an email address is when it shows up in an application. We noticed a spike in the leftmost bucket, the partner was suddenly seeing lots of brand-new emails. The second chart tracks the same thing for phone numbers — and there we see a fat new spike on the right, they were seeing way more old phone numbers than unusual.

Brand-new emails plus aged phones, arriving together at unusual rates, is a textbook signature of a bot attack. Fresh emails are easy to spin up at scale; aged phone numbers get bought off lists. The combination is a tell.

The attack was clever. People don’t change their phone numbers much, so an old phone number usually means a real person living a real life. One application with an aged phone and a fresh email just looks like someone who got a new Gmail. Thousands of them, arriving all at once, looks like something else entirely — and that’s exactly what the shape of the histograms told us.

Story 3: we got better, and the old model couldn't tell

Here’s a happier story. Over six months, one of our models started drifting for the best possible reason — we knew more than we used to.

Line chart of PSI from November 2025 to April 2026 with a dashed upward trend line. PSI stays small, between about 0.0015 and 0.0076, but trends slowly upward

The numbers here are small — a sliver of the PSI scores we saw with the bot attack — but they continued to slowly climb, month after month. Something was changing. But what?

Left: the same small, slowly rising PSI trend from November 2025 to April 2026. Right: score distribution for the baseline vs. the most recent 4 weeks, nearly identical, with almost 90% of scores in the 0-200 range

New data sources had come online. In sports, a tough call sometimes becomes clear from a new camera angle. The new data worked the same way — helping us disambiguate a segment of previously murky applications and move them solidly into the "not fraud" scores, below 200.

Why this matters

In a perfect world, we could learn what fraud looks like once, and then nothing changes and we can identify the same patterns forever and ever…but we don’t live in that world. We live in a world where fraud, people, companies, and data are always changing. As experts in fighting fraud, we have to adapt — and to adapt, we have to know how the world is shifting around our models. When we see the shift before our partners, we can reach out. We can ask questions and work with them to understand what has changed.

There's a version of fraud investigation that resembles detective work — a crime occurs, you examine the scene, and the clues you find in that one case teach you what to look for across millions of others. This is a quieter side of fraud investigation, using graphs as clues. One person with a brand-new email and an old phone number is just a person. Ten thousand applications arriving at the same partner in the same month makes a bar chart that sounds an alarm.

See how SentiLink can help.

Talk with a fraud expert about stopping identity fraud at the point of application, without adding friction for real customers.

Subscribe

Share

Learn how we can help.

Schedule a demo with a fraud expert and evaluate our solutions.

SentiLink

© 2026 SentiLink. All rights reserved.

Get updates

Learn how we can help.

Schedule a demo with a fraud expert and evaluate our solutions.

SentiLink

© 2026 SentiLink. All rights reserved.

Get updates