Experiment notes · September 2026

A computed hello

A little model, thinking out loud in the shape of an introduction.

The homepage opens with “Hi I’m Adam.” Its words separate into tokens, become matrix rows and enter an illustrative attention block. The camera then pulls back into the greeting. The opening choreography uses a small deterministic attention sketch; the zoomed-out letterhead and interactive inspection display the pretrained model’s actual calculations.

One model, one playground

The letterhead and its controls use MicroT-test1-1M-TinyStories, an Apache-2.0 model by llaa33219. Its 996,736 parameters use five transformer layers, eight attention heads, RoPE, RMSNorm, GELU, and a 256-byte vocabulary. It was trained on TinyStories, not to follow instructions.

The introduction beside the animation is editable and is the model’s context. Its exact text is prefixed with “Adam says:” and followed by “He builds projects that”. Reset text restores the public bio. Predicted text appears immediately after that editable introduction. Edits are local to the page and are not saved. It is ordinary prefix text, not a privileged system message: this model has no chat-role training. Its continuation is a prediction and may be fanciful or wrong.

The zoomed-out letterhead initially uses a reproducible saved trace from the quantized model and default context. Loading the model and running a prediction replaces the letterhead’s calculations with live attention from the selected layer and head. Replay restores the greeting’s tokenization sequence and zooms back into the latest model trace. Timing of the presentation is illustrative; inference speed is measured separately.

What each matrix shows

S = QKᵀ / √16
P = softmax(S + causal mask)
O = PV

Each readable tile shows four actual terms from a scaled Q·K product or probability-weighted value sum. The partial contribution is labeled separately from the full score or context feature. Attention probabilities always use the entire causal context; slices are never renormalized to make them look stronger. Values are rounded only for display. Token position, layer, head, dimension range or key range identify the calculation.

Hover expands the field around the pointer. On phones, enable “Touch explore” and drag; “Done exploring” restores page scrolling over the canvas. “Look closer” supports pinch and manual zoom. With reduced motion enabled, the page opens directly on the greeting.

A model in 1 MB

Our symmetric per-row INT8 export contains 1,024,704 bytes of weights, plus a manifest. The weight download is opt-in, verifies SHA-256, and unpacks into 3.99 MB of Float32 arrays. Inference runs in a CPU Web Worker with Float32 state and JavaScript Number accumulation. Prompts and predictions stay in your browser. The source revision is e2418d4e103db0fd4222e2a950caec39cee389c5; independent PyTorch fixtures verify the quantized forward pass.

Reading the utilization meter

Calibrated MFU divides the model’s counted FLOP/s by a measured ceiling from the same browser’s warmed matrix-vector kernel. It is an empirical backend utilization measure, not a vendor’s theoretical hardware MFU. Browsers do not expose a reliable hardware peak; you can supply a matching peak in GFLOP/s separately.

Each multiply-add counts as two FLOPs. Counts include projections, feed-forward layers, attention scores and value accumulation, and the output head; normalization, RoPE, activations, softmax, sampling and trace extraction are excluded from the numerator. Decode wall time includes their overhead and worker scheduling, but excludes download, calibration and prefill. Readings over 100% are not capped. Latency percentiles measure individual forward steps. Tokens are UTF-8 bytes, so throughput is not directly comparable with word-based tokenizers. This is inference, with no backward pass or optimizer.

The older design studies retain their original deterministic, untrained attention illustration. For the attention formulation, see Attention Is All You Need.