Synthetic data seldom fails loudly. The pipeline runs, the charts look plausible and the vendor’s report is green. The failure shows up weeks later, when a model trained on it underperforms in production or an analysis gives a number nobody can reproduce on real data. These four signs show up earlier, if you look for them during the trial.
Sign 1: the model scores well on synthetic data and worse on real data
This is the one that matters most, and the one most often skipped. If you train a model on synthetic data and also test it on synthetic data, you are measuring how well the model learned the generator, not the world.
How to check: keep back a slice of real data the generator never saw. Train two models, one on real training data and one on synthetic, and score both on that real slice. The industry name is TSTR: train on synthetic, test on real. A small gap is normal. A gap that grows on the segments you care about is the warning.
What to do: look at where the gap is. If it sits in a few segments, the generator may need more source data there, or those segments need real data. If it is everywhere, the generator is not capturing the relationships your model depends on.
Sign 2: rare cases have vanished, or multiplied
Generators learn the bulk of a distribution well and the edges badly. In a tabular trial, compare the frequency of every category and the top and bottom percentiles of every number column. In an image trial, count instances of each rare class and inspect a sample by eye.
Two failures look opposite but share a cause. Some generators smooth rare values away, so a fraud label at 0.3% becomes 0.05%. Others overcorrect when asked to oversample a rare class and produce near-duplicates of the few real examples, which a model then memorizes.
| Check | Healthy | Warning |
|---|---|---|
| Rare category share | Close to the real share, or to the target if you asked for oversampling | Much lower, or many rows that differ only slightly |
| Tails of number columns | Similar 1st and 99th percentiles | Tails clipped, or extreme values repeated |
| New combinations | Plausible combinations not seen in the source | None at all (copying) or many impossible ones |
Sign 3: synthetic rows sit too close to real ones
A generator that has memorized part of its training data will produce rows that are near-copies of real records. That is a privacy problem and a quality problem: it means the generator learned individuals rather than patterns.
How to check: for each synthetic row, find the closest real training row and record the distance. Do the same for a set of real rows the generator never saw. If synthetic rows sit much closer to training rows than unseen real rows do, the generator is copying. Count exact matches too; there should be essentially none outside trivially common rows.
What to do: ask the vendor about stronger privacy settings, including differential privacy, and rerun. Expect some loss of fidelity in exchange. The page on synthetic data privacy guarantees explains the trade.
Sign 4: the rules between columns and tables are broken
Real data obeys rules that no statistic captures on its own. A ship date comes after an order date. A child row in an orders table points to a customer that exists. A city belongs to its state. A total equals the sum of its line items.
Generators that learn each column well can still break these rules, because the rules are logical, not statistical. One impossible row in a thousand is enough to crash a test suite, or to skew an analysis that sums across tables.
How to check: write the rules down as assertions and run them against the synthetic data before anything else. For multi-table data, check that every foreign key resolves and that counts per parent (orders per customer, say) look like the real distribution.
What to do: most tabular platforms let you add business rules or constraints on top of the learned model; K2view, for example, describes combining AI generation with rules, cloning and masking. If the vendor’s tool cannot express your rules, that is a reason to look at another one.
Why these slip past vendor reports
A vendor’s quality report usually compares column distributions and pairwise correlations, which a generator can match closely while still failing all four checks above. That is not bad faith. The report is generic because the vendor does not know your model, your rare cases or your business rules. Only you do.
So the trial plan should name, before the first dataset arrives, the real test slice, the rare segments, the closeness test and the list of rules. Agree pass marks for each with the vendor up front. A vendor confident in its product will usually welcome that, and the synthetic data companies we track differ a good deal in how much of it their tools support.
A note on images
The same logic applies to simulated images, with different checks. The gap between synthetic and real test scores becomes the domain gap; rare classes become rare scenes; broken rules become physically impossible scenes or labels that do not match the pixels. Rendered.ai describes its imagery as physically accurate and fully labeled, which is the right claim to test: sample the labels and check them yourself.
None of these checks takes long once it is written down, and every one is cheaper than finding the problem in production.