Every vendor trial ends with someone asking “is it good?” The useful answer has three parts, and each needs its own test. Here is a plan you can hand to a vendor before the first dataset arrives, so the trial measures what you care about instead of what is easy to show.
Before you start: set aside real data and write pass marks
Two things come first. Split off a slice of real data that the generator will never see, typically 20% to 30% of the rows, chosen the same way you would split for model validation. Every utility and privacy test depends on it.
Then write down pass marks. “Close to real” means nothing in a contract. “A model trained on synthetic data scores within a set margin of the real-data model on our holdout, overall and in each of these four segments” is a test both sides can run.
Fidelity: does it look like the real data?
Fidelity checks compare the synthetic dataset with the real training data directly.
Single columns
For each number column, compare the distribution shape, the mean, the spread and the extreme percentiles. For each category column, compare how often each value appears. Watch the edges: a generator can match the middle of a distribution and still clip its tails.
Pairs and groups of columns
Compare correlations between number columns and cross-tabulations between categories. These catch generators that model each column well but lose how they move together, such as age and plan type, or region and spend.
Across tables
For relational data, check that every foreign key resolves, that the number of child rows per parent follows the real distribution, and that rules spanning tables still hold (line items sum to the order total). The page on tabular synthetic data lists what a relational trial should include.
Utility: does it give the same answers?
Utility is the test that predicts whether the data will do its job, and the one most worth running yourself.
| Use | Utility test |
|---|---|
| Training a model | Train on synthetic, test on the real holdout (TSTR). Compare with train on real, test on real. Look at overall and per-segment scores. |
| Analytics | Run your ten most important queries or dashboards on both. Compare the answers, especially ranks and trends. |
| Software test data | Run the test suite. Count failures caused by impossible values or broken keys rather than real bugs. |
| Sharing with a partner | Ask the partner which analysis they will run, and run it on both versions first. |
For simulated images the utility test is the same idea in different clothes. Train a perception model on real only, synthetic only and a mix, then score all three on a real validation set, per class and per scenario. Vendors such as Parallel Domain sell simulation for training and testing perception systems; the mix result on your own validation set is the number that matters.
Privacy: can anyone learn about a real record?
Privacy tests run on the synthetic data and, where possible, on the generator itself.
- Exact and near matches. Count synthetic rows identical to a training row. Then, for each synthetic row, measure the distance to its closest training row and compare with the same measure for the real holdout. Synthetic rows that sit closer to training data than unseen real rows do suggest memorization.
- Membership inference. Simulate an attacker who tries to tell whether a given real record was in the training set. If the attacker does much better than a coin flip, the data gives real records away.
- Attribute inference. Give the attacker some columns of a real record and see whether the synthetic data helps them guess a sensitive column.
- The stated guarantee. If the vendor uses differential privacy, record epsilon, delta and the unit protected. Our writeup on differential privacy in synthetic data explains what to ask.
Certifications are a separate matter. Aindo, for example, lists a Europrivacy certification (a GDPR certification scheme) on its security page. A certification speaks to how a company handles data; it does not replace testing the dataset you receive.
What vendor reports usually cover, and what they skip
Most platforms deliver a quality report with each run. Read it, then check what it leaves out. Reports tend to cover single-column fidelity and pairwise correlations well, because those are generic. They rarely cover utility on your model, your business rules or attacks tuned to your data, because the vendor cannot know those. Tonic.ai, Syntho, K2view and the other synthetic data companies on our list each describe their tooling differently, so ask for a sample report before the trial and map it against the three tests here.
A one-page trial plan
- Real holdout set aside, and how it was chosen.
- Fidelity checks, with the columns and pairs that matter most named.
- Utility test, with the model or queries and the margin that counts as a pass.
- Privacy tests, with thresholds for closeness and membership inference.
- Rules and keys that must hold in every row.
- Who runs each test: the vendor, you, or both.
Hand that page to every vendor on the shortlist, and the comparison at the end of the trial writes itself.