How to create a language model: understand the full path and train a tiny one

What does “create my LLM” actually mean?

You want an AI that helps with your product, code or company knowledge. There are several very different ways to achieve that. Calling an existing model with instructions creates an application around a model. RAG supplies documents at question time. Fine-tuning changes some or all weights of an existing model. Training from scratch learns weights starting from an initial random state. These options solve different problems and have very different costs.

This guide explains that landscape, then lets you train a genuinely small neural language model on your computer. It predicts characters, saves learned weights and generates text. It is an educational MLP, not a large Transformer or a useful chat assistant. The purpose is to make training concrete: you will see data, predictions, error, an update and a measurable result, rather than mistaking an API wrapper for a model you trained.

1. Choose the right project for your need

If your goal is answering from changing manuals, start by evaluating RAG. If the goal is consistent formatting or a specialised behaviour across many good examples, prompting or fine-tuning may help. Neither guarantees factual accuracy. If the goal is learning how models work, this small from-scratch experiment is appropriate. A general-purpose production LLM is a much larger research and infrastructure project.

For example, an event support assistant needs current procedures and sources: RAG is a reasonable baseline to evaluate. A ticket classifier returning a fixed JSON schema can start with prompting and validation, then compare fine-tuning if needed. Reproducing broad multilingual reasoning from random weights is not what this laptop exercise delivers. Write a measurable task before choosing a model or GPU.

2. Learn the essential vocabulary

A tokenizer maps text to IDs; this exercise uses one ID per character. An embedding maps each ID to learned numbers. Parameters are the numbers training adjusts. The context is the input history visible for one prediction. The model produces logits, softmax converts them to probabilities, and the loss penalises giving low probability to the actual next character. Gradients describe how to change the parameters to reduce that error.

A batch contains several training examples. A training step applies an update; an epoch corresponds to passing through a dataset. This lab samples batches, so 400 steps does not mean 400 epochs. A checkpoint stores weights needed to resume or generate. Inference uses those weights without updating them. Temperature and sampling affect output variety; they do not add knowledge.

3. See the complete training workflow

For a real project: define the task and evaluation first; collect data with appropriate rights; remove sensitive material and duplicates; create train, validation and test splits that do not overlap. Choose a tokenizer and architecture, estimate memory and compute, train while monitoring loss, save versioned checkpoints, evaluate on held-out tasks, then consider deployment. Record dataset, code, model and configuration versions.

A useful assistant may also require instruction tuning, alignment and safety evaluation. Serving requires authentication, request limits, context handling, monitoring and update procedures. There is no universal training budget: architecture, tokens, optimiser states, precision, sequence length and hardware all matter. This guide does not promise a production LLM for a fixed number of euros or a short script.

4. What the downloadable model really contains

The original model has an eight-character context, 12-dimensional character embeddings, one 48-unit tanh hidden layer and an output distribution over the supplied vocabulary. With this corpus it has 6,303 learned parameters. It uses NumPy, runs on CPU and contains explicit gradient updates so you can follow the learning loop. It does not implement attention or a Transformer.

The corpus is a few fictional, original sentences about events and learning. It is deliberately too small for broad language ability. The tests check that training reduces its own training loss, probabilities stay finite and sum to one, unknown characters fail clearly, and a saved/reloaded checkpoint gives the same predictions. They do not test general knowledge.

5. Install, train and save

Extract into a new folder. Python 3.9+ and NumPy are sufficient; the installation uses the package registry, while training itself does not need a model download, account, API key or GPU. The included requirements allow NumPy 2.x; exact numeric results can vary by platform.

Run the tests before training. model.py prints the parameter count, before/after training loss and a sample, then writes tiny-model.npz in the working folder. Use the same supplied corpus for this first run. If you replace it, keep a space in the vocabulary for padding and check data rights. Do not use secrets or private conversations as casual training material.

# Python 3.9+; create an isolated environment in the extracted folder.
python3 -m venv .venv
# macOS/Linux (Windows: .venv\Scripts\activate)
. .venv/bin/activate
python -m pip install -r requirements.txt
python -m unittest discover -p 'test_*.py'
python model.py --steps 400
# Output checkpoint: tiny-model.npz

6. Understand the measured result

On our local run, the loss fell from approximately 3.3004 to 0.9185 after 400 steps. That means the model became better at predicting characters in its training corpus. The generated sample still contains nonsense. This is the expected scale of the exercise, not a defect to hide.

The training text is also used to report loss here: no held-out generalisation claim is being made. A model can memorise and still perform badly on unseen material. More steps can reduce training loss while worsening overfitting. For the next experiment, create a meaningful non-overlapping validation set, compare both losses and inspect outputs on a fixed set of prefixes. A lower number alone does not establish useful or truthful answers.

Measured local run, seed 7, supplied corpus, 400 training steps:
parameters: 6303
training_loss_before: 3.3004
training_loss_after: 0.9185
sample begins: seatimin whean sert tes s vitg ...

Interpretation: the model learned local character patterns.
It still produces nonsense and has not demonstrated general knowledge.
These are training-set measurements, not held-out quality scores.

7. Reload and generate: inference is different from training

The code below reloads learned weights and samples characters repeatedly. It does not update the model. The eight-character window slides as output grows. A character absent from the vocabulary produces a clear error rather than an invented ID. The checkpoint is a NumPy archive loaded without pickle; still treat model files as trusted artifacts and validate their provenance.

# The training objective: predict the NEXT character.
# Input:  "seating "
# Target: the following character in the training text.
# In the supplied model, each input contains exactly 8 characters.

# Generation uses the last 8 characters to predict a distribution,
# samples one character, appends it, then repeats.
from model import load
model = load("tiny-model.npz")
print(model.generate("seat", count=120, seed=42))

7b. Make the trained model available to a local application

You have trained weights, but how does another application use them? Start the supplied Python server after model.py has produced tiny-model.npz. It loads those weights and exposes one local POST /generate endpoint. Send prefix="seat" and count=40; it returns generated_text and educational=true. The model still produces learning text, not trustworthy advice.

Training remains in Python/NumPy. The tabs below show four equivalent ways to call that SAME model; they do not pretend that PHP or browser JavaScript trains this network. The tiny server is for localhost learning, not production deployment. It has input limits and no public binding. Keep it local; a real service needs a suitable framework, authentication and operational controls.

# First train the model with the instructions above.
# Keep your Python virtual environment active.
python server.py
# Keep this terminal open. The API listens only on 127.0.0.1:8765.
# Browser demo: open http://127.0.0.1:8765/

# In ANOTHER terminal, from this same project folder:
# Choose your client (Python needs the same environment).
python client.py
# OR: Node.js 22+
node client.mjs
# OR: PHP 8.1+ with HTTP URL wrappers enabled
php client.php

7c. Choose Python, Node.js, JavaScript or PHP

Choose a tab, copy its code and run the supplied matching file. Python, Node.js and PHP call the absolute localhost URL. Browser JavaScript runs on the demo page served at http://127.0.0.1:8765/; the relative /generate URL is intentionally same-origin. Do not paste it into a different website. Open the demo and press Generate 40 characters.

All clients send the same JSON and receive the same deterministic character sample. A response with educational=true confirms this local experiment, not language understanding. If the connection fails, check that server.py is running after training and port 8765 is available. Unknown characters or count outside 1..200 return a readable error.

import json
from urllib.request import Request,urlopen
request=Request('http://127.0.0.1:8765/generate',
 data=json.dumps({'prefix':'seat','count':40}).encode(),
 headers={'Content-Type':'application/json'},method='POST')
with urlopen(request,timeout=10) as response:
 result=json.load(response)
print(json.dumps(result,indent=2))

// Node.js 22+. First run the Python model and local server.
const response = await fetch('http://127.0.0.1:8765/generate', {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: JSON.stringify({prefix: 'seat', count: 40}),
  signal: AbortSignal.timeout(10_000),
});
const result = await response.json();
if (!response.ok) throw new Error(result.error || 'Generation failed');
console.log(JSON.stringify(result, null, 2));

// Run on http://127.0.0.1:8765/ served by this local project.
// That page already contains <button id="generate"> and <pre id="result">.
document.querySelector('#generate').addEventListener('click', async () => {
  const output = document.querySelector('#result');
  try {
    const response = await fetch('/generate', {
      method: 'POST',
      headers: {'Content-Type': 'application/json'},
      body: JSON.stringify({prefix: 'seat', count: 40}),
      signal: AbortSignal.timeout(10_000),
    });
    const result = await response.json();
    if (!response.ok) throw new Error(result.error || 'Generation failed');
    output.textContent = JSON.stringify(result, null, 2);
  } catch (error) {
    output.textContent = error.message;
  }
});

<?php
// PHP 8.1+ with HTTP URL wrappers enabled; run the Python server first.
$context = stream_context_create(['http' => [
  'method' => 'POST',
  'header' => "Content-Type: application/json\r\n",
  'content' => json_encode(['prefix' => 'seat', 'count' => 40]),
  'timeout' => 10,
  'ignore_errors' => true,
  'follow_location' => 0,
]]);
$raw = file_get_contents('http://127.0.0.1:8765/generate', false, $context);
if ($raw === false) throw new RuntimeException('Local server unavailable');
$result = json_decode($raw, true, 512, JSON_THROW_ON_ERROR);
if (isset($result['error'])) throw new RuntimeException($result['error']);
echo json_encode($result, JSON_PRETTY_PRINT).PHP_EOL;

{
  "model": "tiny-character-mlp",
  "educational": true,
  "generated_text": "seatimin whean sert tes s vitg bemovge peans"
}

8. How this relates to a real Transformer LLM

Modern autoregressive Transformer models also predict the next token, but use deeper architectures and much larger training data. Attention lets a token combine information from other allowed positions. A causal mask prevents training predictions from reading future tokens. Position information, feed-forward blocks, residual paths and normalisation are additional architectural components. Our fixed-window MLP is not a reduced implementation of all those parts.

To continue, implement a small causal Transformer as a separate exercise with tensor and causal-mask tests, or follow the Stanford course linked below. For useful product behaviour, compare a licensed pretrained model before attempting large-scale pretraining. LoRA adapts selected weights through small low-rank updates; it does not create a new foundation model from nothing. Check base-model licence, data rights and actual evaluation before publishing an adapter.

9. Use AI as a teacher and an experiment reviewer

The first English prompt asks the assistant to explain the actual code and execute the local experiment. Expect parameter counts, commands, measured loss and a sample, with limits clearly stated. An answer claiming “your own ChatGPT is ready” would be misleading here. The second prompt frames a real adaptation plan and prevents an expensive training run being started as an unstated next step.

Act as a patient machine-learning teacher. Inspect this tiny NumPy character
language model and explain its data flow before modifying anything.
Identify the vocabulary, eight-character context, embeddings, hidden layer,
softmax probabilities, next-character targets, loss and gradient update.
Run the local tests, train with 400 steps, save the checkpoint and reload it.
Report actual parameter count, training loss before/after and a generated sample.
Explain why reduced training loss is not evidence of held-out quality or truth.
Do not describe this 6,303-parameter educational MLP as a large Transformer,
a chat assistant or a production LLM. Do not download a model or private dataset.
Propose one next learning experiment with a measurable success criterion.

10. Your final result and a sensible next step

You have trained weights from scratch, measured learning, saved and reloaded a checkpoint, and generated text. You can distinguish a model, an application, RAG and fine-tuning. The practical next step is one controlled experiment: change context length or add authorised varied data, keep a held-out evaluation and compare with the original baseline. To build an actual business assistant, start from the user task and compare existing models and RAG before choosing training.

Before proposing a REAL model adaptation, ask for the task, licensed base model,
data rights, deployment language, evaluation set, hardware and budget.
Compare RAG, prompting, LoRA fine-tuning and training from scratch for that task.
If fine-tuning is justified, propose an isolated reproducible experiment:
base-model revision, tokenizer, non-overlapping train/validation/test splits,
training configuration, checkpoint policy, metrics and comparison with baseline.
Include privacy leakage, memorization and unsupported-answer evaluations.
State estimates and untested assumptions explicitly. Do not start an expensive
training job, publish weights or use private data without the agreed scope.

Official references and further reading

• Stanford CS336 · Language Modeling from Scratch
• Vaswani et al. · Attention Is All You Need
• Hugging Face PEFT · LoRA
• NumPy · Random generator
• Python · Local teaching HTTP server
• Node.js · fetch
• PHP · HTTP context options

Read this tutorial as reference material:
https://nethanelfitoussi.com/article/create-language-model-explained?lang=en

Goal: apply the documented workflow to my development project.
Read the existing project instructions, inspect the relevant files and explain the smallest useful setup.

Available starter files:
https://nethanelfitoussi.com/api/media/08d5b33c-5d92-4495-ae4a-729eb570d94d.zip
Download and inspect these files in a temporary folder before using them. Do not overwrite existing project instructions.

If I have not supplied a project folder, ask which development project to use. Do not invent a repository URL.

Additional project setup instructions:
Extract the supplied ZIP into a NEW folder. Read README.md before installing or running code.

Act as a patient machine-learning teacher. Inspect this tiny NumPy character
language model and explain its data flow before modifying anything.
Identify the vocabulary, eight-character context, embeddings, hidden layer,
softmax probabilities, next-character targets, loss and gradient update.
Run the local tests, train with 400 steps, save the checkpoint and reload it.
Report actual parameter count, training loss before/after and a generated sample.
Explain why reduced training loss is not evidence of held-out quality or truth.
Do not describe this 6,303-parameter educational MLP as a large Transformer,
a chat assistant or a production LLM. Do not download a model or private dataset.
Propose one next learning experiment with a measurable success criterion.

Before proposing a REAL model adaptation, ask for the task, licensed base model,
data rights, deployment language, evaluation set, hardware and budget.
Compare RAG, prompting, LoRA fine-tuning and training from scratch for that task.
If fine-tuning is justified, propose an isolated reproducible experiment:
base-model revision, tokenizer, non-overlapping train/validation/test splits,
training configuration, checkpoint policy, metrics and comparison with baseline.
Include privacy leakage, memorization and unsupported-answer evaluations.
State estimates and untested assumptions explicitly. Do not start an expensive
training job, publish weights or use private data without the agreed scope.

Adapt example paths, commands and acceptance criteria to this repository rather than assuming they match.

Give me a short plan, then make the agreed local changes and run the relevant checks. Report actual results and anything not verified. Stop before credentials, paid services, production changes, push, merge or deployment.