AI glossary Data & Retrieval
What is big data?
← All glossary termsData sets so large, fast-moving or varied that traditional databases and tools cannot store or process them efficiently; a key input for AI.
Explained
Big data refers to data sets whose size, speed or variety exceeds what traditional databases and tools can handle efficiently. It includes transaction records, logs, sensor readings, clickstreams, documents, images and audio. Working with it needs distributed storage and processing, such as data lakes, lakehouses and cloud data platforms. For AI, big data is the raw material: it is used to train and evaluate models and to feed AI applications with current information. Volume alone does not create value; quality, governance and a clear use do.
What makes data big
Big data is usually described by a few characteristics, often called the Vs:
- Volume: the amount of data is too large for a single machine.
- Velocity: data arrives quickly and may need processing in real time.
- Variety: it mixes structured tables with text, images, audio and logs.
- Variability: its rate and shape change over time.
Other qualities often added are veracity (how trustworthy the data is) and value (whether it is worth the effort).
Big data and AI
Machine learning models learn from examples, so the amount and quality of data shape what they can do. Big data platforms provide the pipelines to collect, clean and label that data for training.
With generative AI, the role shifts slightly. Large models are pre-trained by their providers, and companies mostly use their own data to ground answers (for example with retrieval), to fine-tune models or to evaluate them. Unstructured data, such as documents, emails and call recordings, becomes far more useful than before.
What a company needs
- A data platform that can store and process data at the scale required.
- Data quality checks and clear ownership of each data set.
- Governance: catalogue, lineage, access control and privacy rules.
- Use cases with a measurable outcome, so that data work serves a decision.
- People who can turn data into models, dashboards and applications.
Frequently asked questions
What is big data in simple terms?
It is data in such large amounts, arriving so fast or in so many formats that ordinary databases and spreadsheets cannot cope. Handling it requires distributed systems that split storage and processing across many machines.
What are the Vs of big data?
The classic three are volume, velocity and variety. Many descriptions add variability, veracity and value. Together they explain why big data needs different tools from traditional data processing.
What is the difference between big data and AI?
Big data is the material; AI is one way of using it. Big data platforms collect, store and process large data sets, and AI models learn from that data to make predictions or generate content.
Do you need big data to use AI?
Not always. Many generative AI uses rely on models already trained by providers and need only a well-organised set of the company's own documents and data. Training a custom model from scratch, by contrast, usually needs large amounts of data.
Where it shows up at Thinkia
Further reading
Related terms
Sources
- NIST · SP 1500-1r1: NIST Big Data Interoperability Framework, Volume 1: Definitions
- USENIX OSDI '04 · Dean and Ghemawat, MapReduce: simplified data processing on large clusters
- USENIX NSDI '12 · Zaharia et al., Resilient Distributed Datasets (Apache Spark)
Updated: 2026-10-21