rmsnorm

Root Mean Square Layer Normalization

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Optimized CUDA kernels for NVIDIA GB10 Blackwell (sm_121, DGX Spark). RMSNorm + GELU. First sm_121 kernel on HuggingFace Kernel Hub.

面向 Transformer 推理的可验证 Triton 融合算子与 torch.library 集成

Reproducible CUDA inference kernel experiments with tested RMSNorm baselines and optimizations

A non-official implementation of Qwen 3.5, as there doesn’t seem to be a paper or any code available that I can find, so I decided to implement it just for fun.

Modular, step-by-step implementation of GPT & Modern LLMs from scratch in PyTorch. Featuring RoPE, RMSNorm, SwiGLU, GQA, KV-Cache, Unit Tests, and an Interactive Web Playground.

A character-level GPT model built from scratch in PyTorch, exclusively trained to generate text in the style of Shakespeare. Features a modular RAG pipeline.

You imports the model's weights and start fine-tuning, when you are missing out on more than 60% of gpu utilization. Axiom gets all the juice out of the box.

Full-stack LLM inference engine with custom CUDA, Triton and C++ kernels, MGX Prophet cache, MegaMesh, and MicroGEMM CPU runtime.

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

Simple character level Transformer

A transformer language model built from scratch, from byte-level BPE tokenization through pretraining and QA fine-tuning.

An automated GPU kernel optimization and profiling workstation compiling fused OpenAI Triton kernels (RMSNorm, SwiGLU, Tiled Attention) to maximize SRAM reuse. Exposes custom backward autograd functions directly in GPU memory, using roofline models to verify Peak HBM bandwidth and hardware TFLOPs efficiency.

Decoder only Transformer trained from scratch in PyTorch, featuring GQA attention with QK Norm, RoPE and KV caching, plus a Mixture of Experts FFN with top k routing and a shared expert.

First-principles reproduction of Multi-Head Latent Attention (MLA) , Mixture-of-Experts (MoE) , and Multi-Token Prediction (MTP) , the three mechanisms that make DeepSeek-V3 unique. Built from scratch, no pre-built libraries. 109M parameters trained on 2.5B tokens for $9.53 on a single A100.

Generative models nano version for fun. No STOA here, nano first.