swiglu

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

使用Decoder-only的Transformer进行时序预测，包含SwiGLU和RoPE(Rotary Positional Embedding)，Time series prediction using Decoder-only Transformer, Including SwiGLU and RoPE(Rotary Positional Embedding)

Fused SwiGLU Triton kernels

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Official code of <Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM> (ICML2026)

Hackable PyTorch template for decoder-only transformer architecture experiments. Llama baseline with RoPE, SwiGLU, RMSNorm. Swap components, train, compare

Modular, step-by-step implementation of GPT & Modern LLMs from scratch in PyTorch. Featuring RoPE, RMSNorm, SwiGLU, GQA, KV-Cache, Unit Tests, and an Interactive Web Playground.

A 197M-param decoder-only LM trained from scratch on FineWeb-Edu — PyTorch with RoPE, GQA, SwiGLU, 32k BPE, built for 6GB VRAM

Paired JAX/T4 comparison of SwiGLU and Kimi K3's SiTU-GLU

A 36M-parameter goldfish language model with a 10-second memory + pixel-art PWA desk pet. Runs in your browser, fully offline. Adopt it at den-sec.github.io/glublm/desk-pet/

A transformer language model built from scratch, from byte-level BPE tokenization through pretraining and QA fine-tuning.

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

Transformer Models for Humorous Text Generation. Fine-tuned on Russian jokes dataset with ALiBi, RoPE, GQA, and SwiGLU.Plus a custom Byte-level BPE tokenizer.

An automated GPU kernel optimization and profiling workstation compiling fused OpenAI Triton kernels (RMSNorm, SwiGLU, Tiled Attention) to maximize SRAM reuse. Exposes custom backward autograd functions directly in GPU memory, using roofline models to verify Peak HBM bandwidth and hardware TFLOPs efficiency.

"A from-scratch implementation of a modern Large Language Model (LLaMA architecture). Features full backpropagation calculus in pure NumPy, followed by a GPU-accelerated PyTorch training loop."

World Structured SwiGLU FFN

LLM 推理性能决策基线：TTFT/TPOT、KV Cache、吞吐与解码策略对照

An industry-grade, production-ready Multi-Modal AGI Vision-Language LLM & Agentic Reasoning Framework built from scratch in PyTorch. Features ViT patch encoding, RoPE, SDPA Flash-Attention, KV-Cache, autonomous Plan-Act-Reflect agentic loop with tool dispatch & Google Colab T4 GPU support.

A compact, research-oriented reimplementation of a GPT-OSS-style transformer combining Mixture-of-Experts (MoE), sliding-window attention, and rotary embeddings for efficient LLM training. Compares against GPT-2 baseline under identical conditions to isolate architectural improvements.