Build and Train a GLM-5.3-Flash Model From Scratch with Python
GLM-5.3-Flash combines linear and sparse attention, a mixture-of-experts design, and manifold-constrained hyperconnections. A scaled-down version with 25 million parameters can be built in Python and trained on a standard CPU—making it a practical entry point for anyone who wants to understand modern language model architecture end to end.
We just published a course on the freeCodeCamp.org YouTube channel that teaches you how to build and train a multimodal language model from scratch, covering both pre-training and post-training. The course is taught by @vukrosic, and all of the code, slides, and experiments are available in the GitHub repository linked in the video description.
The model uses a 260-token byte-level vocabulary, a 192-token context window, and a small vision encoder that turns a 32×32 image into 16 tokens.
What You Will Learn
Tokenization and Embeddings
Why a small model benefits from a small byte-level vocabulary, and how weight tying shares a single matrix between the embeddings and the output head.
Linear and Sparse Attention
How linear attention maintains a fixed-size memory of recent tokens, while every fourth layer uses sparse attention with an indexer to retrieve distant details.
Mixture of Experts
How tokens are routed to a few selected experts, plus a shared expert that learns information common to every token.
Adding Vision
How images are split into patches using 2D RoPE and merged into image tokens so the model can recognize digits.
Pre-Training Experiments
How data diversity, interleaving, and curricula affect learning—including how interleaving helps prevent catastrophic forgetting.
Reinforcement Learning
How to design reward functions, compute advantages as in GRPO, update only the final layers, and test group size, temperature, and seeds.
After reinforcement learning, the model improves from solving 0 of 24 Python tasks to 16 of 24.
Watch the Full Course
Watch the full course on the freeCodeCamp.org YouTube channel (44-minute watch).
Published November 14, 2026 · #llm
via FreeCodeCamp