The Impact of miniGPT Parameters on Training Results – Stray Birds

In the last training experiment we ran a systematic sweep over different parameter combos: hidden layer size (H), dropout ratio (dp) and learning rate (lr). This time I swapped the dataset to Stray Birds, went through the results, and broke down how each parameter affects training, plus some recommended configs worth trying. One thing to know about this round: the timeout was set way too short, so most runs never got the multi-epoch training the earlier single-sentence run had — they bailed out after a few rounds. My hardware was a limit too (though the 4070 shouldn’t be the bottleneck, the dataset is probably just too small so the GPU sat there waiting), and so was time (it took about a week to finish), so I never got through every parameter combination I’d set up and the results aren’t great. If you’re curious, go tune the training script yourself and dig further.

1. Experimental design

• Dataset: Stray Birds (about 800 lines)
• Model architecture: a simplified Transformer / RNN, hidden size H from 64, 128, 256, 512
• Hardware: one RTX 4070, CPU i5-13600K, 32GB RAM
• Training parameter sweep:
  • Hidden layer H: 64 / 128 / 256 / 512
  • Dropout dp: 0.0 / 0.1 / 0.2 / 0.3
  • Learning rate lr: 0.0001 / 0.0005 / 0.001
• Training limits: some runs never fully converged because of the per-round timeout.

2. Comparison of results

3. Parameter analysis

3.1 Effect of learning rate

• lr=0.001: training tends to oscillate or overfit, and it shows most when the hidden layer is large.
• lr=0.0005: training is at its most stable here, loss usually sits in the 0.9–1.1 range, and what it generates generalizes pretty well.
• lr=0.0001: convergence is slow, it tends to stall at a high loss, which shows up as underfitting.

3.2 What dropout does

• dp=0.0: loss often drops below 0.1, the model basically memorizes the training set, and generalization is bad.
• dp=0.2–0.3: loss holds steady in 0.9–1.1, and the output reads more natural and generalizes better.

3.3 Model size H

• H=64: too small, capacity is limited, results are mediocre.
• H=128–256: middle-of-the-road capacity, it converges without tipping into overfitting.
• H=512: lots of capacity, but slow to train. Some runs timed out after a single epoch, so the loss never fully converged.

3.4 The timeout problem

• The per-round timeout is what killed some runs, especially big models paired with a small learning rate.
• Fixes:
  • Shorten the sequence to cut the compute
  • Bump the learning rate so it converges faster
  • Shrink the sweep parameter space, no big model + small lr combos

4. Recommended setup

• Best combo:
  • Hidden layer H: 128 or 256
  • Dropout dp: 0.2–0.3
  • Learning rate lr: 0.0005
• Training results: loss sits in 0.9–1.1, the model isn’t rote-memorizing, the generated text is more natural, and generalization is decent.
• Things to watch:
  • A very low loss is not a good result, it usually means overfitting
  • Too big a model or too little training time and the run won’t fully converge

Training results for some configurations

5. Conclusion

1. Moderate capacity plus a little dropout is the best combo on a small corpus, it keeps the style and still leaves room for novelty.
2. Too much capacity or too little dropout and it overfits, pumping out mechanical text.
3. Too few training epochs caps what a big model can do, even with lots of capacity it will still spit out the occasional incoherent line.
4. What I’d try next:
   • Train for more epochs so big models actually converge
   • Add data augmentation (recombining lines, synonym rewriting)
   • Try a mix of model sizes to balance speed and generation quality

The matching version this time was 1.9-2.3.

Published on September 14, 2025.