Model Distillation in Python: Fit a Tiny Student to Soft Teacher Labels

Distillation trains a student to match signals from a teacher. A small numerical experiment makes that objective visible: fit a logistic student to soft probabilities, then evaluate on inputs excluded from training.

Crazyrouter publishes this local distillation demonstration with nine training inputs, four held-out points and four executed checks; it does not claim to have trained an LLM.

By Boluo Z · Tested November 3, 2026

How does a student model learn from soft teacher labels?#

The teacher is sigmoid(2x − 0.5), a deterministic function with no API call and no proprietary training data. The student starts with two zero-valued parameters and learns a logistic function from nine training inputs. Each target is a probability rather than a hard zero-or-one class.

The objective is binary cross-entropy against those soft targets. Gradient descent performs 800 fixed updates at learning rate 0.1. The script uses only Python’s standard library; saving the code and executing it reproduces the fitted parameters and every observation.

November 3 experiment and observed results#

This local Python experiment used no API and needs no credential. Each result describes this run only. A test program exiting successfully means its observations were recorded, not that every model response passed.

Training cross-entropy decreased from 0.693147 to 0.348672 after 800 updates. This is a synthetic two-parameter experiment.

Keep the held-out inputs outside training#

Four distinct x values are used only for evaluation. The checks require disjoint train/test inputs, decreasing training loss, bounded probabilities and a maximum held-out probability error below 0.01. All four checks passed.

This toy teacher belongs to the same two-parameter family as the student, so the exercise is intentionally easy. It demonstrates matching probability targets, not compressing a large neural network or proving out-of-distribution generalization. Cross-entropy need not reach zero when the teacher probabilities themselves have uncertainty.

Translate the lesson carefully to language models#

Real language-model distillation introduces a much larger vocabulary, sequence data, a model architecture, optimizer choices and evaluation challenges. Soft-label distillation can use temperature-scaled distributions; this toy uses a binary distribution at temperature one. Response-based datasets are another approach and do not expose the same teacher information.

Before building such a dataset, verify permissions for the source model and data. This example avoids external teacher outputs entirely and makes no legal claim about a specific provider. Keep evaluation examples independent, report the teacher’s weaknesses and measure the actual student workload rather than assuming matching a few labels transfers all capabilities.

Reproduce the experiment#

Save the script and run it with Python 3.10 or later; only the standard library is used.

import json
import math

def sigmoid(x):return 1/(1+math.exp(-x))
def teacher(x):return sigmoid(2*x-0.5)
def loss(w,b,points):
    return sum(-teacher(x)*math.log(sigmoid(w*x+b))-(1-teacher(x))*math.log(1-sigmoid(w*x+b)) for x in points)/len(points)

train=[-2,-1.5,-1,-0.5,0,0.5,1,1.5,2]
holdout=[-1.75,-0.75,0.25,1.25]
w=b=0.;initial=loss(w,b,train)
for step in range(800):
    errors=[sigmoid(w*x+b)-teacher(x) for x in train]
    w-=0.1*sum(e*x for e,x in zip(errors,train))/len(train)
    b-=0.1*sum(errors)/len(train)
rows=[{"x":x,"teacher_probability":teacher(x),"student_probability":sigmoid(w*x+b),"absolute_error":abs(teacher(x)-sigmoid(w*x+b))} for x in holdout]
tests=[{"case":"disjoint held-out inputs","passed":not set(train)&set(holdout)},
{"case":"cross entropy decreased","passed":loss(w,b,train)<initial},
{"case":"probability error below 0.01","passed":max(r["absolute_error"] for r in rows)<.01},
{"case":"probabilities bounded","passed":all(0<r["student_probability"]<1 for r in rows)}]
assert all(t["passed"] for t in tests)
print(json.dumps({"local_tests":tests,"requests":[],"steps":800,"training_points":len(train),"heldout":rows,"initial_loss":initial,"final_loss":loss(w,b,train),"student_parameters":{"w":w,"b":b},"scope":"Two-parameter logistic student fitted to a local synthetic teacher; no LLM trained or API called"}))

Sources and related guides#

Sources checked November 3, 2026: Distilling the Knowledge in a Neural Network. Official descriptions and gateway observations are separate evidence.

• Open-Weight vs Commercial Model APIs: Test the Output Contract Before Switching
• Self-Hosted Model Readiness: Test an Evidence Gate Before Deployment
• Self-Hosted vs Hosted AI Costs: Test the Break-Even Formula
• AI Structured Output in Python: JSON Mode, Schema, and Live Validation
• AI SaaS Unit Economics: Calculate Cost per Usable Output

FAQ#

Was an LLM trained?#

No. A two-parameter logistic student was fitted to a local synthetic teacher.

How many examples were used?#

Nine training inputs and four disjoint held-out inputs.

Why use probabilities?#

They provide the student with the teacher’s confidence, not just a hard class label.

Why is the final loss nonzero?#

Soft target distributions have uncertainty; matching them does not generally make cross-entropy zero.

Does this demonstrate a smaller production model?#

No. The teacher and student share a deliberately simple function family.

Do I need an API key?#

No. This particular experiment runs entirely locally with Python’s standard library.

Last actually tested: November 3, 2026. This local conceptual tutorial is due for review February 1, 2027.