Adventures in AI: neural networks, machine learning, and language models, explored through small, hands-on projects in C# and .NET.
Run these .NET commands from the repository root:
dotnet build src/Jc.AI.slnx
dotnet test src/Jc.AI.slnx --no-buildThe NUnit tests cover the hand-worked examples, the full dataset, empty text, whitespace, casing, punctuation, accents, combining marks, emoji, and invalid input. Pull requests also run a Release build and the tests through GitHub Actions.
Bigram tests cover pair counts, smoothing, normalized probabilities, loss, repeatable sampling, invalid input, and the training/validation workflow.
This repository follows my journey of building a tiny GPT.
The character tokenizer builds a vocabulary of distinct, sorted UTF-16 code units from supplied text, encodes text into token IDs, and decodes those IDs back into the original text. Unknown code units and invalid token IDs produce explicit errors.
The bundled dataset in data/facts.txt produces a vocabulary of 29 code units. With that vocabulary, Mira encodes to [4, 16, 23, 9] and decodes back to Mira.
The counted bigram model learns how often each token follows another (at this point, a token is a single character), then uses those counts with add-one smoothing to predict and generate the next character. It considers only the previous token, so generated text captures local character patterns but has little word or sentence structure.
The vocabulary and counts are trained on the data/facts.txt from part 1. The additional text in data/validation.txt measures how well the model predicts unseen text without training on it. Loss is the average negative log-probability of each next token - lower is better. A uniform baseline provides a comparison with guessing every token equally.
Part 3 introduces a linear neuron that predicts weight * input + bias. Starting from zero, it learns the rule y = 2x + 1 from five numerical examples by adjusting its weight and bias with gradient descent to reduce mean squared error.
The demo checks the calculated gradients against numerical estimates, shows progress over 100 updates, and predicts 7 for the unseen input 3. Tests cover the gradients, parameter updates, convergence, and invalid input.
Learning XOR with a Hidden Layer
XOR outputs 1 when its two inputs differ and 0 when they match. A single linear neuron cannot fit all four cases. Part 4 adds a hidden layer with four tanh units, giving the network the nonlinearity it needs to learn XOR.
The network uses backpropagation to calculate gradients and gradient descent to reduce mean squared error.
The demo first trains a linear baseline, followed by training our small neural network with four tanh hidden units. It shows the network's mean squared error at five checkpoints over 10,000 training updates.