The Impact of miniGPT Parameters on Training Results – Single-Sentence Text

Last time we trained our own model for the first time, trying texts of different lengths (single-sentence text and Stray Birds). We also stumbled on a parameter set that worked reasonably well among the random configs. In this post we’ll look at how to find the training parameters that suit us best.

First, here’s what each parameter actually does:

These parameters shape how fast the model learns, how well it performs, and how stable it is:

1. batch_size (batch size)
   • Meaning: how many samples are processed in parallel in one training pass.
   • Small batch: more frequent updates → faster learning, but noisier, so it may bounce around.
   • Large batch: more stable updates → smoother convergence, but it needs more VRAM.
   • Worth noting: with a small dataset the batch shouldn’t be too big, or it overfits easily.
2. block_size (context length)
   • Meaning: how much of the sequence the model can see at once.
   • Small block: the model’s “field of view” is short, so it only learns local relationships.
   • Large block: the model can learn long-range dependencies, but training time and VRAM usage climb noticeably.
   • What I noticed: lots of GPT papers simply keep raising block_size and watch the performance improve.
3. embed_size (embedding dimension / hidden size)
   • Meaning: the dimension of the character/word vectors.
   • Small embedding: not enough capacity for the model to express much.
   • Large embedding: much more expressive, but more parameters and longer training.
4. n_layers (number of layers)
   • Meaning: how many Transformer layers are stacked.
   • Few layers: the model is shallow and can’t pick up complex patterns.
   • More layers: stronger modeling power, but it overfits more easily and trains slower.
5. learning_rate (learning rate)
   • Meaning: the step size of each parameter update.
   • Too small: converges slowly.
   • Too large: the loss oscillates or even diverges.
   • Usually paired with a learning rate scheduler, like warmup or decay.
6. dropout (drop rate)
   • Meaning: a regularization method for preventing overfitting.
   • 0: nothing dropped, the strongest fitting power, but a high risk of overfitting.
   • 0.1~0.3: the usual range.

2. How to tell whether the results are good

1. Training/validation loss
   • The lower the loss, the better the model fits the training (or validation) set.
   • But watch out for overfitting (low training loss, high validation loss).
2. Generated text quality
   • More intuitive.
   • You can just read what it generated:
     • Coherent (does the sentence read naturally and smoothly).
     • Reasonable (does the logic hold together).
     • Diverse (is it more than mechanical repetition).
3. Convergence speed
   • With identical parameters, reaching a lower loss faster means the setup is more efficient.
   • That’s exactly why logging the training time of each run is worth doing.
4. Perplexity (PPL)
   • A common metric for language models.
   • PPL = exp(loss), lower is better.
   • For example:
     • PPL=50 → the model is pretty uncertain on average about which token comes next.
     • PPL=10 → the model is far more confident.

 3. Goal

To compare how different parameters affect training, watch:

1. The loss curve (does it fall faster and more steadily).
2. Training time (cost vs. payoff).
3. The sample text it generates (does it look like the original).

Do that enough times and you slowly build an intuitive map from “parameters → training behavior”.

On top of last time’s project we added sweep.py, which calls train.py to train the data with different parameter combinations so we can watch what each one does. Every run produces a folder for its parameters, the loss log and curve, the matching model, and finally a text sample generated from that model.

One thing worth mentioning: I set up the block_size=64 combination in the sweep, and the raw data may be too short, so the index overflows when load_dataset builds xb/yb. Without drop_last=True, the last batch can end up shorter than block_size, which makes the model reach for tokens that don’t exist during forward. That’s why this option has to be there.

This run dragged on far too long, so I stopped it once the single-sentence text finished training. Let’s look at how it did on the single-sentence text first.

The best result, according to GPT

Best Results

• Best experimental setup:
  • batch_size=8
  • embed_dim=64
  • num_layers=2
  • hidden_dim=128
  • lr=0.0001
  • dropout=0.3
• final_loss = 0.000022, which is excellent.

How the parameters affected the results

1. Dropout

• On average the best was 0.1 (0.5439), then 0.3.
• 0.0 (no regularization) and 0.2 were actually a bit worse.
  ➡️ So a little dropout does help.

2. Learning rate (lr)

• 0.0001 was the best (0.5465), and as lr grows the results get slightly worse:
  • 0.0001 < 0.0005 < 0.0010
    ➡️ The smaller learning rate is steadier.

3. Batch size

• 8 and 64 came out ahead (around 0.5459), while the middle values (16, 32) were slightly worse.
  ➡️ Either small or large batches suit this better; the middle seems to oscillate.

4. Embedding dimension

• Bigger is better: 256 was the winner (0.5457), then 128.
• Small embeddings (32, 64) did poorly.
  ➡️ Semantic capacity improves as the dimension goes up.

5. Hidden dimension

• 128 was best (0.5453); going bigger buys little, and 64 and 256 were a bit worse.

6. Number of layers (num_layers)

• 2 layers was clearly the best (0.00011); add more layers and the loss shoots up, with 12 layers almost unusable.
  ➡️ Too deep and the model overfits or training falls apart.

Takeaways

1. Best setup:
   batch_size=8, embed_dim=128~256, hidden_dim=128, num_layers=2, lr=0.0001, dropout=0.1~0.3
2. Trends:
   • A small learning rate converges steadily.
   • Moderate dropout helps.
   • Bigger embedding is better, but hidden_dim 128 is plenty.
   • Shallow models work best; deep ones make training collapse.

But for the training data as a whole, this run’s loss values are pretty good while the generated text isn’t that great. Maybe the data is too simple / overfitting: the model may have memorized the training set, which pushes training loss close to 0. Plenty of runs produced output identical to the training data, just followed by a lot of junk.

Why this happens

1. Way too little data
   • The training data is a single sentence, yet the model has a huge number of parameters (hundreds of thousands, even millions).
   • So the model easily memorizes that sentence by heart, and the loss sits near 0 during training.
2. Repetition and runaway output
   • At inference the model keeps emitting tokens according to probability.
   • It can reproduce the training sentence, then drifts into making things up (there’s no other training data to lean on).
   • So the opening matches exactly and the rest is a mix of junk.
3. What a language model really is
   • An LM learns the distribution of the next word, not how to recite one sentence.
   • Once the context moves past the sentence it was given, the model “hasn’t seen it” and can only sample wildly.

How to improve

If you want the model to only produce that sentence, or to reproduce content more reliably, a few options:

1. The fine-tuning angle
   • Training on one sentence ≠ fine-tuning. What you have now is “overfitting to a single sentence”.
   • Add more similar sentences (say tens to hundreds of short ones) so the model learns a sturdier distribution.
2. Tweaks to the training
   • Shrink the model: smaller embed_dim, hidden_dim, num_layers. A model that’s too big leads to memorizing by rote plus rambling output.
   • Raise dropout: keep the model from nailing one sentence down so fast.
   • Change the training objective: if all you need is reproduction, switch to seq2seq or even a plain classification task instead of language modeling.
3. Controls at inference time
   • Turn temperature down (0.1, say) and the model outputs what you trained it on with more confidence instead of wandering off.
   • You can even set greedy decoding (top_k=1), and it will almost only reproduce the training data.

In one line:
Your model has “perfectly memorized” that sentence (so the loss is close to 0), but with so little training data it never learned a natural language distribution, which is why the second half of what it generates “makes things up”.

This time was mainly about extending the original project. Next time we’ll train on Stray Birds; there’s still plenty to sort out, like the training taking far too long.

The version corresponding to this run is 1.6-1.8.

Published on August 24, 2025.