layernorm

Root Mean Square Layer Normalization

Code for the paper "On the Expressivity Role of LayerNorm in Transformers' Attention" (Findings of ACL'2023)

Implement layer normalization GRU in pytorch

Some common CUDA kernel implementations (Not the fastest).

Unveiling the Layers: Neural Networks from first principles

MNIST Digit Prediction using Batch Normalization, Group Normalization, Layer Normalization and L1-L2 Regularizations

WGAN with feedback from discriminator& LayerNorm instead of BatchNorm

Pure C implementation of Layer Normalization with learnable γ and β parameters—built from scratch without ML frameworks or autograd.

Implemented a custom LayerNorm forward and backward pass extension in C++ using LibTorch and validated numerical equivalence with PyTorch.

Fused normalisation CUDA kernels for PyTorch — residual-add+LayerNorm/RMSNorm, fp8 outputs, fast deterministic backward, torch.compile-native, with reproducible A100/H100 benchmarks

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

Topology and geometry of Transformer hidden states via persistent homology. The manifold is a 1D arc (PC1=norm, r=0.999), not a torus or sphere. PC2=position, PC3=surprisal+POS. Validated up to 40B.

Fundamentals of Artificial Intelligence and Deep Learning Frameworks

A complete, operational Transformer Decoder Layer built from scratch — combining self-attention, Pre-LN normalization, residual connections, and a SwiGLU feed-forward network, with an empirical demonstration of why residual connections prevent signal collapse in deep stacks.

DEPRECATED. Canonical: https://github.com/szl-holdings/szl-lambda-gate. Archived frozen for provenance; HF model SZLHOLDINGS/szl-governed-norm remains available. Apache-2.0.

Geometric phase structure in Transformer hidden states. LayerNorm placement predicts manifold geometry — 6x difference in PCA concentration. 9 models, 13 experiments.

Fused LayerNorm (forward + backward) in Triton, benchmarked against PyTorch