AI glossary Models & Architecture
What is BERT in NLP?
BERT (Bidirectional Encoder Representations from Transformers)
← All glossary termsA Transformer language model from Google, pre-trained to read context in both directions; widely used for search, classification and embeddings.
Explained
BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced by Google researchers in 2018. It is pre-trained on large amounts of unlabelled text by hiding some words and learning to predict them from the words on both sides, which gives it a strong grasp of context. A pre-trained BERT can then be fine-tuned with one extra output layer for tasks such as question answering, classification or entity recognition. It reads and represents text rather than generating it, and BERT-style encoders remain the basis of many semantic search and embedding systems.
How BERT works
BERT uses only the encoder part of the Transformer architecture. During pre-training, some of the words in each sentence are masked and the model learns to fill them in using the words before and after. The original paper describes this masked language modelling objective as inspired by the Cloze task, the classic fill-in-the-blank exercise.
The result is a model that produces a contextual representation for every word: the same word gets a different vector depending on the sentence around it. Those representations are the starting point for downstream tasks.
BERT vs GPT-style LLMs
- BERT is an encoder: it reads text and produces representations. GPT-style models are decoders: they generate text one token at a time.
- BERT looks at context in both directions at once; generative models predict the next token from what came before.
- BERT-style models are small and fast enough for high-volume classification and retrieval; generative LLMs are more flexible but heavier.
- In many RAG systems both appear: an encoder for embeddings and re-ranking, a generative model for the answer.
What it is still used for
Encoders descended from BERT are still common where speed, cost and predictability matter: semantic search and embeddings, re-ranking retrieved passages, classifying tickets or emails, detecting entities in documents and filtering content. Smaller distilled versions make them practical to run on modest hardware.
Frequently asked questions
What does BERT stand for?
BERT stands for Bidirectional Encoder Representations from Transformers. Bidirectional refers to using context on both sides of each word; encoder refers to the part of the Transformer architecture it uses.
Is BERT a large language model?
It is a pre-trained language model and an important ancestor of today's LLMs, but it is much smaller than current generative models and does not generate free text. It is usually described as an encoder model rather than an LLM in the modern sense.
Is BERT still used today?
Yes. BERT and its descendants are widely used for embeddings, semantic search, re-ranking and classification, because they are efficient and well understood. For open-ended generation, teams use generative LLMs instead.
What is the difference between BERT and ChatGPT?
BERT reads text and turns it into representations that other components use for search or classification. ChatGPT is built on generative models that write answers. They solve different parts of a problem and are often used together.
How is BERT used in search?
BERT-style models turn queries and documents into vectors so that results can be matched by meaning, not just keywords. They are also used to re-rank a short list of results by how well each one answers the query.
Where it shows up at Thinkia
Further reading
Related terms
Sources
- Devlin et al. · BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Google Research · Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing
- Hugging Face · BERT (Transformers documentation)
Updated: 2026-10-21