AI glossary Models & Architecture
What is test-time compute?
Test-Time Compute (Test-Time Compute)
← All glossary termsThe allocation of additional processing power during the inference phase, allowing an AI model to generate, evaluate and refine intermediate steps before delivering a final answer.
Explained
Historically, improving AI performance meant scaling up the computing power used during pre-training. Test-time compute shifts this focus to the inference phase, when the model is actually running. By dedicating more processing time and resources to a specific query, reasoning models can think through a problem, explore multiple paths, check their own logic and correct mistakes before responding. This approach, popularised by models such as the OpenAI o1 series, allows smaller models to tackle highly complex reasoning tasks that would otherwise require massive pre-training scale.
How test-time compute works
The model breaks down a complex prompt into a sequence of smaller, manageable logical steps. It generates multiple potential pathways to solve the problem, rather than just predicting the next most likely word. It then evaluates its own intermediate outputs, discarding flawed reasoning and backtracking when necessary. Once it verifies a valid path to the solution, it outputs the final answer, hiding or summarising the underlying thinking process.
Pre-training vs test-time scaling
For years, the AI industry relied on the scaling laws of pre-training: feeding larger models exponentially more data and compute to make them smarter. This requires vast server clusters running for extended periods. Test-time compute introduces a new scaling law at inference. Instead of building a fundamentally larger model, developers can allocate more processing time to an existing model at the exact moment a user asks a difficult question. This makes it possible to trade latency for accuracy, spending more compute only when the complexity of the task demands it.
When it makes sense
This approach makes sense when the task requires deep logic, mathematics, coding or multi-step problem-solving. Accuracy and rigorous reasoning must be more important than an instant response. It is ideal when the organisation wants to deploy a smaller, more efficient model that can punch above its weight by thinking longer on hard queries. Finally, workflows must tolerate variable latency, as the model takes longer to answer complex questions than simple ones.
Frequently asked questions
Why is test-time compute important?
It allows AI systems to solve complex logic, math and coding problems that standard models fail at. By thinking longer, the model dramatically increases its accuracy without needing a larger foundational architecture.
Which models use test-time compute?
Reasoning models are the primary users of this approach. Notable examples include OpenAI's o1 series, which is specifically trained to utilise extra compute during inference to refine its answers.
Does it make AI slower?
Yes, for the specific queries that require it. Because the model is generating and evaluating multiple hidden steps before responding, the user experiences a noticeable delay depending on the problem's difficulty.
How does this affect AI costs?
It shifts costs from training to inference. While you might save money by using a smaller model, deploying it for complex tasks will incur higher per-query costs, as the system consumes more compute tokens to generate the hidden reasoning steps.
Is test-time compute the same as chain-of-thought prompting?
They are related but distinct. Chain-of-thought is a prompting technique where the user asks the model to show its work. Test-time compute is deeply integrated into the model's architecture, allowing it to autonomously search, evaluate and backtrack without the user seeing the messy intermediate steps.
Where it shows up at Thinkia
Further reading
Related terms
Sources
- OpenAI · Learning to Reason with LLMs
- arXiv · Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
- DeepSeek-AI · DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning (arXiv)
Updated: 2026-10-21