איך ליצור מודל שפה: להבין את הדרך ולאמן מודל קטן

# Python 3.9+; create an isolated environment in the extracted folder.
python3 -m venv .venv
# macOS/Linux (Windows: .venv\Scripts\activate)
. .venv/bin/activate
python -m pip install -r requirements.txt
python -m unittest discover -p 'test_*.py'
python model.py --steps 400
# Output checkpoint: tiny-model.npz

Measured local run, seed 7, supplied corpus, 400 training steps:
parameters: 6303
training_loss_before: 3.3004
training_loss_after: 0.9185
sample begins: seatimin whean sert tes s vitg ...

Interpretation: the model learned local character patterns.
It still produces nonsense and has not demonstrated general knowledge.
These are training-set measurements, not held-out quality scores.

# The training objective: predict the NEXT character.
# Input:  "seating "
# Target: the following character in the training text.
# In the supplied model, each input contains exactly 8 characters.

# Generation uses the last 8 characters to predict a distribution,
# samples one character, appends it, then repeats.
from model import load
model = load("tiny-model.npz")
print(model.generate("seat", count=120, seed=42))

# First train the model with the instructions above.
# Keep your Python virtual environment active.
python server.py
# Keep this terminal open. The API listens only on 127.0.0.1:8765.
# Browser demo: open http://127.0.0.1:8765/

# In ANOTHER terminal, from this same project folder:
# Choose your client (Python needs the same environment).
python client.py
# OR: Node.js 22+
node client.mjs
# OR: PHP 8.1+ with HTTP URL wrappers enabled
php client.php

import json
from urllib.request import Request,urlopen
request=Request('http://127.0.0.1:8765/generate',
 data=json.dumps({'prefix':'seat','count':40}).encode(),
 headers={'Content-Type':'application/json'},method='POST')
with urlopen(request,timeout=10) as response:
 result=json.load(response)
print(json.dumps(result,indent=2))

// Node.js 22+. First run the Python model and local server.
const response = await fetch('http://127.0.0.1:8765/generate', {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: JSON.stringify({prefix: 'seat', count: 40}),
  signal: AbortSignal.timeout(10_000),
});
const result = await response.json();
if (!response.ok) throw new Error(result.error || 'Generation failed');
console.log(JSON.stringify(result, null, 2));

// Run on http://127.0.0.1:8765/ served by this local project.
// That page already contains <button id="generate"> and <pre id="result">.
document.querySelector('#generate').addEventListener('click', async () => {
  const output = document.querySelector('#result');
  try {
    const response = await fetch('/generate', {
      method: 'POST',
      headers: {'Content-Type': 'application/json'},
      body: JSON.stringify({prefix: 'seat', count: 40}),
      signal: AbortSignal.timeout(10_000),
    });
    const result = await response.json();
    if (!response.ok) throw new Error(result.error || 'Generation failed');
    output.textContent = JSON.stringify(result, null, 2);
  } catch (error) {
    output.textContent = error.message;
  }
});

<?php
// PHP 8.1+ with HTTP URL wrappers enabled; run the Python server first.
$context = stream_context_create(['http' => [
  'method' => 'POST',
  'header' => "Content-Type: application/json\r\n",
  'content' => json_encode(['prefix' => 'seat', 'count' => 40]),
  'timeout' => 10,
  'ignore_errors' => true,
  'follow_location' => 0,
]]);
$raw = file_get_contents('http://127.0.0.1:8765/generate', false, $context);
if ($raw === false) throw new RuntimeException('Local server unavailable');
$result = json_decode($raw, true, 512, JSON_THROW_ON_ERROR);
if (isset($result['error'])) throw new RuntimeException($result['error']);
echo json_encode($result, JSON_PRETTY_PRINT).PHP_EOL;

{
  "model": "tiny-character-mlp",
  "educational": true,
  "generated_text": "seatimin whean sert tes s vitg bemovge peans"
}

Act as a patient machine-learning teacher. Inspect this tiny NumPy character
language model and explain its data flow before modifying anything.
Identify the vocabulary, eight-character context, embeddings, hidden layer,
softmax probabilities, next-character targets, loss and gradient update.
Run the local tests, train with 400 steps, save the checkpoint and reload it.
Report actual parameter count, training loss before/after and a generated sample.
Explain why reduced training loss is not evidence of held-out quality or truth.
Do not describe this 6,303-parameter educational MLP as a large Transformer,
a chat assistant or a production LLM. Do not download a model or private dataset.
Propose one next learning experiment with a measurable success criterion.

Before proposing a REAL model adaptation, ask for the task, licensed base model,
data rights, deployment language, evaluation set, hardware and budget.
Compare RAG, prompting, LoRA fine-tuning and training from scratch for that task.
If fine-tuning is justified, propose an isolated reproducible experiment:
base-model revision, tokenizer, non-overlapping train/validation/test splits,
training configuration, checkpoint policy, metrics and comparison with baseline.
Include privacy leakage, memorization and unsupported-answer evaluations.
State estimates and untested assumptions explicitly. Do not start an expensive
training job, publish weights or use private data without the agreed scope.