AI decisions Architecture and technology
Synthetic data vs real data: what should you train, tune and test your AI on?
Use real data as the reference for anything that decides whether a model works: final evaluation, acceptance and monitoring should reflect the population and conditions the system will actually meet. Use synthetic data to fill gaps that real data cannot cover safely or in time: rare cases, adversarial tests, development environments without personal data, or balancing under-represented groups. Synthetic data generated from personal data is not automatically outside GDPR, and a model that only passes synthetic tests has not been shown to work.
The options
Synthetic data
Data generated by rules, simulations or models to imitate the structure and statistics of real data, or to create cases that do not exist in it.
Real data
Data collected from actual operations, customers or users, used as is or after pseudonymisation, anonymisation and curation.
Side by side
| Criterion | Synthetic data | Real data |
|---|---|---|
| Fidelity to production | Only as faithful as the generator; it reproduces the patterns it was given and can miss the ones nobody modelled. | Reflects how the process really behaves, including noise, exceptions and quirks no one documented. |
| Rare and edge cases | Can create as many rare, extreme or adversarial cases as needed, on demand. | Rare cases are scarce by definition; some may never appear before the system goes live. |
| Privacy and GDPR | Reduces exposure, but generating it from personal data is itself processing, and the output stays personal data if people can be re-identified. | Usually contains personal data: needs a legal basis, purpose limitation, minimisation, security and a plan for data subject rights. |
| Bias | Can rebalance under-represented groups, but also copies or amplifies the bias of the data and assumptions behind the generator. | Carries the historical bias of past decisions and of who is, or is not, in the data. |
| Validity for evaluation | Good for unit tests, stress tests and coverage of known scenarios; weak evidence of real-world performance on its own. | The reference for acceptance and for metrics you report to the business, auditors or regulators. |
| Availability and speed | Available early, before a process exists or while access to production data is being approved. | Depends on access rights, data quality and labelling; obtaining it can take longer than building the model. |
| Labels | Labels come for free with generated data, but they are only as correct as the generator's logic. | Labels must be collected or reviewed by people, and their quality varies. |
| Typical failure mode | The model learns the generator's simplifications and fails on the messiness of real inputs; model-generated data can also feed back its own errors. | The data leaks into places it should not be, or is so incomplete or skewed that the model inherits its gaps. |
| Documentation and audit | Must document how it was generated, from what source, with which parameters and how its fidelity and privacy were checked. | Must document origin, legal basis, consent where relevant, transformations and representativeness of the population. |
Choose Synthetic data when…
- You need to develop and test before production data is available, or in environments where personal data must not go.
- Rare events, fraud patterns, failures or adversarial inputs must be tested and real examples are too few.
- Some groups or scenarios are under-represented and you need to check or improve behaviour on them.
- You are testing an agent's tools and workflows against mock systems rather than measuring business accuracy.
Choose Real data when…
- You are measuring whether the system is good enough to go live, or reporting its performance to anyone.
- The task depends on real language, documents or behaviour whose variety is hard to simulate.
- You have a lawful, documented way to use the data and the means to protect it.
- You need to monitor drift and quality once the system is in production.
- The use case may fall under high-risk rules, where datasets must be shown to be relevant and representative.
When to combine them
Most mature teams use both, with clear roles. Real data, properly governed, is the ground truth: a held-out evaluation set that synthetic data never touches and that reflects the real population. Synthetic data extends coverage around it: more rare cases, adversarial prompts, privacy-safe copies for development and targeted rebalancing. Generate synthetic data from a separate slice of real data, never from the evaluation set, and check that a model trained with it improves on real held-out data, not only on more synthetic data.
Common mistakes
- Calling synthetic data anonymous by default, without testing how easily records could be linked back to real people.
- Evaluating a model on synthetic data produced by the same generator, or the same LLM, that produced its training data.
- Generating synthetic examples from the evaluation set, so the test measures memory rather than ability.
- Using synthetic data to hide a representativeness problem instead of fixing how real data is collected.
- Copying full production databases into development or test environments because it is faster than building safe test data.
How Thinkia approaches it
We start by asking what each dataset is for: building, tuning, testing or proving. Proving needs real data, so with the business owners we agree an evaluation set from their own cases, kept apart from anything used to train or to generate synthetic examples. The evaluation harness stays on in production to catch regressions when models, prompts or data change, and those evaluation sets belong to the client.
Synthetic data comes in where it adds coverage or removes exposure: rare cases, adversarial tests, agent workflows run against mock systems and development environments without personal data. We document how it was generated and check it against real held-out data before trusting any improvement it seems to bring.
Privacy and bias travel with the use case. The Trust Fabric dimension of the Compass brings EU AI Act, DPIA/FRIA and GDPR into the work from the start, and the dataset is treated as an asset with an owner, legal basis, retention rules and documentation. Our AI governance guide is operational guidance, not legal advice.
Thinkia products involved
- EU AI Act governance guideRisk tiers, timeline, roles and a 20-point checklist. Not legal advice.
- SynapseGoverned agentic platform: agents, models, costs and data in one place.
Related AI solutions
- AI-ready data platformA platform AI use cases ship on
- Data quality, lineage & governanceGovernance AI actually needs
- AI governance, risk & controlAI your board, legal team and regulators can sign off on.
- MLOps & AI infrastructureTurn experiments into production
- AI quality assuranceCatch defects before they reach the customer.
Frequently asked questions
Is synthetic data outside the scope of GDPR?
Not automatically. Generating synthetic data from personal data is processing and needs a legal basis. The output is outside GDPR only if it is truly anonymous, meaning individuals cannot be identified by means reasonably likely to be used, which must be assessed and documented, not assumed.
Can we evaluate an AI model only with synthetic data?
You can test specific behaviours with it, such as format, edge cases or robustness to adversarial inputs. To claim the model works for the business, you need evidence on real data that reflects the population and conditions it will meet in production.
Does synthetic data remove bias?
It can help rebalance groups that are under-represented, but the generator learns from existing data and from the assumptions of whoever configures it, so it can also reproduce or amplify bias. Measure fairness on real data, broken down by the groups that matter for the use case.
What does the EU AI Act say about training data?
For high-risk systems, Article 10 requires training, validation and testing data to follow data governance practices and to be relevant and sufficiently representative for the intended purpose, among other conditions. It also mentions synthetic or anonymised data as alternatives to consider before processing special categories of personal data for bias detection. Check the Regulation text and get qualified legal advice for your case.
Is it safe to train models on data generated by other LLMs?
It is common for narrow tasks, such as creating instruction examples or test questions, but check the licence terms of the model that generated it, review samples by hand and keep real data in the evaluation. Training repeatedly on model-generated data without checks can narrow the variety of outputs and reinforce the generator's mistakes.
Can we use production data in test environments?
Only with the same legal basis and protections as in production, which test environments rarely have. The usual options are pseudonymised or masked extracts with controlled access, or synthetic data built to match the structure, with real data reserved for a controlled final evaluation.
Keep exploring
Related decisions
- RAG vs fine-tuning: which one does your enterprise use case need?
- LLM evals vs human review: how should you assure the quality of enterprise AI?
- Fix the data first or start with AI now: which comes first?
- EU AI Act provider vs deployer: which role are you, and what does each one owe?