How Much Context Does a Small GPT Model Really Need?
Three controlled experiments on model quality, training throughput, and when additional context stops paying off.
Updated
August 19, 2026
•
12
min read
•
View as Markdown
A
Aman Jain
Product manager learning to build AI from the inside out — from LLM architecture to product applications.
Part of series
Inside GPT: Experiments and Trade-offs
TL;DR
I ran three context-length experiments using the same compact GPT model (
Github
):
Experiment 1 — Fixed training-token budget:
When every model processed approximately 200,000 tokens, the 64-token context produced the lowest loss, although the 32-token result was close. Contexts of 128 and 256 tokens did not improve quality in this setup.
Experiment 2 — Fixed training-time budget:
When every model trained for five minutes, the 32-token model processed the most data and produced the best validation result. The 64-token model remained close, while longer contexts completed fewer updates and produced higher loss.
Experiment 3 — Training context vs evaluation context:
Giving a model more context during evaluation than it had seen during training did not help. The model trained with 128-token sequences performed slightly worse when evaluated with 256 tokens.
Across these experiments, the most promising quality-cost region for this model, dataset, and training budget appeared to be around
32–64 tokens
.
This is not a general conclusion about language models. It is simply what I observed in this setup.
Starting with a simple question
Larger context windows are usually presented as an advantage.
More context gives a model access to more preceding text. In principle, that should help it connect ideas across sentences, retain earlier information, and make predictions using a broader view of the passage.
My initial assumption was therefore simple:
If I increase context length, the model should have more useful information available and make better next-token predictions.
But longer sequences also create larger attention calculations, reduce training throughput, and may allow the model to process fewer examples within a fixed amount of time.
More importantly, the additional context may not always contain useful information.
So I reframed the question:
In my setup, how does context length affect model quality and computational cost, and where do the gains begin to flatten?
The intuition: useful relationships may often be local
Consider this sentence:
The cat sat on the wooden table because it was tired.
To predict the word after
was
, the model probably does not need information from several paragraphs earlier. Nearby words already provide most of the useful signal.
Many next-token predictions may work this way. Grammar, punctuation, common phrases, and sentence-level meaning can often be inferred from relatively nearby tokens.
This led to my working hypothesis:
In this dataset, many useful token relationships may fall within approximately 32 or 64 positions. Increasing context beyond that may add longer-range information, but not enough to justify the additional cost.
This was only an intuition to test. Longer-range relationships clearly exist. The question was whether they appeared frequently enough—and whether my small model had enough capacity and training exposure—to benefit from them.
The context lengths I compared
I trained the same model with four context windows:
Context length
Rough interpretation
32 tokens
Part of a sentence or a short passage
64 tokens
One or more sentences
128 tokens
A longer paragraph
256 tokens
Multiple paragraphs
The model’s maximum supported context remained fixed at 256 tokens. Only the sequence length supplied during training changed.
This kept the positional-embedding table and parameter count identical across runs.
MAX_MODEL_CONTEXT = 256
TRAIN_CONTEXTS = [32, 64, 128, 256]

MODEL_CONFIG = {
    "vocab_size": 50_257,
    "context_length": MAX_MODEL_CONTEXT,
    "emb_dim": 128,
    "n_heads": 4,
    "n_layers": 4,
    "drop_rate": 0.1,
    "qkv_bias": False,
}
Experiment 1: Same number of training tokens
For the first experiment, I wanted every model to receive approximately the same training exposure.
I adjusted batch size so each optimizer step processed 512 tokens:
BATCH_SIZE_BY_CONTEXT = {
    32: 16,
    64: 8,
    128: 4,
    256: 2,
}
Every run completed 391 optimizer steps and processed 200,192 tokens.
The question was:
If every model processes the same number of tokens, does giving each sequence more context improve prediction quality?
The core training loop remained the same across configurations:
def train_one_context(train_context, device=DEVICE):
    batch_size = BATCH_SIZE_BY_CONTEXT[train_context]

    # Build loaders and restore the shared model initialization.
    model = GPTModel(MODEL_CONFIG)
    model.load_state_dict(SHARED_INITIAL_STATE)
    model = model.to(device)

    for step in range(NUM_TRAINING_STEPS):
        input_batch, target_batch = next(train_iterator)

        optimizer.zero_grad(set_to_none=True)
        logits = model(input_batch)
        loss = F.cross_entropy(
            logits.flatten(0, 1),
            target_batch.flatten(),
        )
        loss.backward()
        optimizer.step()

        # Runtime, loss, throughput, and memory tracking omitted.

    return result, history, generation_rows
Evaluating the same prediction with different context lengths
Ordinary validation windows at different context lengths may contain different passages and prediction targets. That makes it difficult to isolate the effect of context alone.
To make the comparison more controlled, I kept the target token fixed and changed only the amount of preceding context.
Suppose
tired
is the target:
Context 4:   table because it was                   → tired
Context 8:   sat on the wooden table because it was → tired
Context 16:  ...additional preceding text...        → tired
Each longer context contains the shorter one.
This allowed me to ask:
Does adding earlier information improve the prediction of the exact same next token?
The dataset implementation was relatively small:
class NestedNextTokenDataset(Dataset):
    def __init__(self, token_ids, target_positions, eval_context_length):
        self.token_ids = torch.tensor(token_ids)
        self.target_positions = target_positions
        self.eval_context_length = eval_context_length

    def __getitem__(self, index):
        target_position = self.target_positions[index]
        start = target_position - self.eval_context_length

        input_ids = self.token_ids[start:target_position]
        target_id = self.token_ids[target_position]

        return input_ids, target_id
Fixed-token results
The matched nested evaluation produced:
Training context
Target loss
Top-1 accuracy
32
6.8550
14.84%
64
6.8353
13.28%
128
6.8783
13.87%
256
6.9479
14.65%
The 64-token model produced the lowest target loss, although the difference from 32 tokens was small.
I reran the experiment without changing the code. The exact values changed, but the overall pattern remained similar:
32 and 64 tokens stayed relatively close.
64 tokens produced the lowest loss.
128 and 256 tokens did not improve average prediction loss.
I would not claim that 64 tokens is universally better than 32. The more defensible observation is that increasing context to 128 or 256 tokens did not help in this setup.
Why loss and accuracy did not agree
Top-1 accuracy did not follow exactly the same ordering as loss because the metrics measure different things.
Suppose two models both fail to rank the correct token first:
Model
Probability assigned to the correct token
Top-1 result
Model A
5%
Incorrect
Model B
20%
Incorrect
Both receive zero top-1 accuracy, but Model B receives a much better cross-entropy loss.
For this reason, I treated cross-entropy loss as the primary quality metric and top-1 accuracy as a supporting signal.
Experiment 2: Same amount of training time
For the second experiment, every context configuration trained for five minutes.
I also used a source file containing approximately 500,000 tokens. After the 90/10 split, 450,000 tokens were available for training.
The batch-size mapping remained unchanged, so every optimizer step still processed 512 tokens. The difference was that the number of optimizer steps was no longer fixed.
Faster configurations could complete more updates and process more training data.
The question became:
Given the same amount of training time, does additional context compensate for lower throughput?
TRAINING_TIME_SECONDS = 300.0

while runtime_seconds < TRAINING_TIME_SECONDS:
    step_started = time.perf_counter()

    input_batch, target_batch = next(train_iterator)

    optimizer.zero_grad(set_to_none=True)
    logits = model(input_batch)
    loss = F.cross_entropy(
        logits.flatten(0, 1),
        target_batch.flatten(),
    )
    loss.backward()
    optimizer.step()

    runtime_seconds += time.perf_counter() - step_started
    optimizer_steps += 1
    tokens_seen += input_batch.numel()

    # Periodic evaluation and logging omitted.
I also added untimed warm-up steps and restored the original model weights afterward. The execution order was randomized to reduce first-run and last-run effects.
Fixed-time results
Here is what each configuration completed within approximately 300 seconds:
Context
Steps
Tokens processed
Training passes
Tokens/sec
Final loss
Accuracy
32
687
351,744
0.78×
1,171
6.8528
15.08%
64
677
346,624
0.77×
1,155
6.8756
14.60%
128
633
324,096
0.72×
1,080
7.0078
13.77%
256
591
302,592
0.67×
1,009
7.0112
12.55%
In this run, the 32-token model:
completed the most optimizer steps,
processed the most training tokens,
achieved the highest throughput,
and produced the lowest final validation loss.
The 64-token result remained close.
The difference was clearer at 128 and 256 tokens. Those models processed less training data during the same five-minute period, and the additional context did not compensate for that reduced exposure.
A different winner depending on the constraint
The two experiments produced slightly different winners.
Equal training tokens
The 64-token context produced the best observed loss, with 32 tokens close behind.
Equal training time
The 32-token context produced the best observed loss and processed slightly more training data.
One possible interpretation is:
At equal training exposure, the additional context between 32 and 64 tokens may provide a small benefit. Under a fixed time budget, the higher throughput of the 32-token model may offset that benefit.
I would still treat the difference between 32 and 64 cautiously. The stronger signal was that 128- and 256-token contexts did not outperform the shorter configurations in these experiments.
Experiment 3: Training context versus evaluation context
For the third experiment, I separated:
the context length used during training,
the context length supplied during evaluation.
Because every model technically supported 256 tokens, I could train a model using 128-token sequences and evaluate it using 256 tokens.
The relevant results were:
Training context
Evaluation context
Relationship
Target loss
Perplexity
Accuracy
128
128
Matched
6.8783
970.97
13.87%
128
256
Beyond training
6.9137
1005.92
12.70%
256
256
Matched
6.9479
1040.93
14.65%
When the 128-token model received 256 tokens during evaluation, its loss became slightly worse.
One possible reason is that positions beyond 128 had not been meaningfully trained. The architecture accepted those positions, but that did not mean the model had learned to use them.
The model trained directly at 256 tokens also did not produce a better loss. In this setup, the additional distant tokens may not have contained enough frequently useful information, or the model may not have had enough capacity or training exposure to use them effectively.
Experiment images
The following four outputs can be presented together as a two-by-two visual summary.
Figure: Four views of the context-length trade-off in this experiment—quality, training exposure, throughput, and evaluation-length interaction.
How I interpret the combined results
Across the three experiments, I observed:
Contexts of 32 and 64 tokens produced relatively similar quality.
A 64-token context performed slightly better when training exposure was fixed.
A 32-token context performed slightly better when training time was fixed.
Contexts of 128 and 256 tokens reduced throughput without improving average loss.
Providing more context during evaluation than the model had seen during training did not help.
For this particular setup, the most promising region appeared to be around 32–64 tokens.
There are several possible explanations:
Much of the predictive signal in the dataset may have been local.
Longer-context batches contained fewer independent sequences.
The model may have needed more training to learn longer-range relationships.
Average next-token loss may underrepresent the relatively small number of predictions that require distant information.
The experiment also helped me distinguish among three related concepts:
Supported context:
The maximum sequence length the architecture can technically accept.
Training context:
The sequence lengths the model actually sees while learning.
Effective context:
The portion of the available history the model has learned to use productively.
All four models technically supported 256 tokens. That did not mean they had all been trained to use 256 positions, or that all 256 tokens contributed useful information to every prediction.
Final takeaway
I started with the intuition that giving the model more preceding text should improve next-token prediction.
In this setup, the relationship was less straightforward.
With the same number of training tokens, 64 tokens produced the lowest observed loss.
With the same amount of training time, 32 tokens processed more data and produced the lowest observed loss.
The longer contexts of 128 and 256 tokens reduced throughput but did not improve quality in these runs.
The conclusion I would draw is deliberately narrow:
For this compact GPT model, WikiText dataset, and training budget, the useful quality-cost region appeared to be around 32–64 tokens. Increasing available context beyond that point did not produce enough measurable benefit to offset the additional cost.
A larger context window gave the model the opportunity to use more information.
Whether that information was useful—and whether the model could learn to use it—was a separate question.
References
A Comprehensive Survey on Long Context Language Modeling
NoLiMa: Long-Context Evaluation Beyond Literal Matching
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
LongRoPE2: Near-Lossless LLM Context Window Scaling
#
deep-learning
#
context-length
#
large-language-models
#
artificial-intelligence
10
views