Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 17 additions & 15 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,23 @@ cargo add libsdca # Rust crate (Rust>=1.74 and a C++11 compiler)
- Prebuilt wheels: Linux `x86_64` (`manylinux_2_28`) and macOS `arm64` (11+).
- PyTorch and JAX compile small C++ extensions on first use when a compiler and Python
headers are present; otherwise they fall back with a one-time warning
([TORCH.md](python/TORCH.md#build-tools)).
([TORCH.md](docs/TORCH.md#build-tools)).
- Check the selected path: `libsdca.torch.explain_backend(cross_entropy, logits, target)`
returns `(backend, reason)` for the call `cross_entropy(logits, target)`.

## Which API to use

| User goal | API | Guide |
| --- | --- | --- |
| Replace `torch.nn.CrossEntropyLoss` (labels or probabilities, weights, label smoothing) | `libsdca.torch.CrossEntropyLoss`, `cross_entropy` | [TORCH.md](python/TORCH.md#replace-pytorch-cross-entropy) |
| Large-vocabulary head + cross-entropy without `(N, V)` logits | `LinearCrossEntropyLoss`, `linear_cross_entropy` | [TORCH.md](python/TORCH.md#linear-layer-and-cross-entropy-without-the-logits) |
| Top-*k*, hinge and multilabel PyTorch losses | `TopKCrossEntropyLoss`, `TopKHingeLoss`, `MulticlassHingeLoss`, `MultilabelHingeLoss`, `MultilabelCrossEntropyLoss` | [TORCH.md](python/TORCH.md) |
| Logit distillation in PyTorch | `DistillationLoss`, `distillation_loss` | [TORCH.md](python/TORCH.md#teacher-logits-and-distillation) |
| Losses in JAX (`jit`, `grad`, `vmap`) | `libsdca.jax` | [JAX.md](python/JAX.md) |
| Linear or kernel SVM / softmax classifier (scikit-learn) | `libsdca.sklearn.SDCAClassifier` | [python/README.md](python/README.md#classifier) |
| Same, with duality-gap history and warm starts | `libsdca.fit(X, y, loss)` with loss objects such as `libsdca.TopKHingeBeta(k=2)` | [python/README.md](python/README.md#training) |
| Losses and gradients on NumPy arrays | `libsdca.loss_and_grad` | [python/README.md](python/README.md#losses-and-gradients) |
| Rust, C++, C, MATLAB | crate `libsdca`, headers, `libsdca_c`, `matsdca_*` | [Rust](rust/README.md), [C++](docs/CPP.md), [C](docs/C.md), [MATLAB](docs/MATLAB.md) |
| Replace `torch.nn.CrossEntropyLoss` (labels or probabilities, weights, label smoothing) | `libsdca.torch.CrossEntropyLoss`, `cross_entropy` | [TORCH.md](docs/TORCH.md#replace-pytorch-cross-entropy) |
| Large-vocabulary head + cross-entropy without `(N, V)` logits | `LinearCrossEntropyLoss`, `linear_cross_entropy` | [TORCH.md](docs/TORCH.md#linear-layer-and-cross-entropy-without-the-logits) |
| Top-*k*, hinge and multilabel PyTorch losses | `TopKCrossEntropyLoss`, `TopKHingeLoss`, `MulticlassHingeLoss`, `MultilabelHingeLoss`, `MultilabelCrossEntropyLoss` | [TORCH.md](docs/TORCH.md) |
| Logit distillation in PyTorch | `DistillationLoss`, `distillation_loss` | [TORCH.md](docs/TORCH.md#teacher-logits-and-distillation) |
| Losses in JAX (`jit`, `grad`, `vmap`) | `libsdca.jax` | [JAX.md](docs/JAX.md) |
| Linear or kernel SVM / softmax classifier (scikit-learn) | `libsdca.sklearn.SDCAClassifier` | [docs/PYTHON.md](docs/PYTHON.md#classifier) |
| Same, with duality-gap history and warm starts | `libsdca.fit(X, y, loss)` with loss objects such as `libsdca.TopKHingeBeta(k=2)` | [docs/PYTHON.md](docs/PYTHON.md#training) |
| Losses and gradients on NumPy arrays | `libsdca.loss_and_grad` | [docs/PYTHON.md](docs/PYTHON.md#losses-and-gradients) |
| Rust, C++, C, MATLAB | crate `libsdca`, headers, `libsdca_c`, `matsdca_*` | [Rust](docs/RUST.md), [C++](docs/CPP.md), [C](docs/C.md), [MATLAB](docs/MATLAB.md) |

Loss names, options, defaults and errors are shared across languages:
[docs/API.md](docs/API.md).
Expand All @@ -42,15 +42,17 @@ Loss names, options, defaults and errors are shared across languages:

- **Class axis:** `CrossEntropyLoss` takes classes on `dim=1` like PyTorch; every other
loss defaults to `dim=-1`. Pass `dim=1` for `(N, C, H, W)` inputs to the others
([TORCH.md](python/TORCH.md#argument-parity)).
- **Hyperparameters are keyword-only** and belong to their loss: `TopKHingeLoss(k=2)`,
([TORCH.md](docs/TORCH.md#argument-parity)).
- **Loss parameters** belong to their loss: `TopKHingeLoss(k=2)`,
`libsdca.TopKHingeBeta(k=2)`; passing a parameter another loss uses is an error.
PyTorch/JAX hyperparameters are keyword-only; NumPy loss objects also accept positional
arguments.
- **Distillation** takes raw logits for both inputs and applies no $T^2$ factor
([TORCH.md](python/TORCH.md#teacher-logits-and-distillation)).
([TORCH.md](docs/TORCH.md#teacher-logits-and-distillation)).
- **Memory:** CUDA losses can reuse the logits buffer for the gradient when nothing else
references it ([TORCH.md](python/TORCH.md#memory)).
references it ([TORCH.md](docs/TORCH.md#memory)).
- **Solver objective:** $\frac12\Vert W \Vert_2^2 + C\sum_i \ell_i$ with no intercept; standardize
features; for $d \gg n$ pass a Gram matrix ([python/README.md](python/README.md#training),
features; for $d \gg n$ pass a Gram matrix ([docs/PYTHON.md](docs/PYTHON.md#training),
[API.md](docs/API.md#options)).
- **Convergence:** only `status == "solved"` guarantees the duality gap
([API.md](docs/API.md#results)).
Expand Down
20 changes: 14 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
exponentials.
- **Faster CUDA paths:** Adaptive distillation kernels and packed short hinge rows.
- **JAX CPU fusion:** `SDCA_JAX_CPU_FUSION=auto` times fused and fenced array programs once
per machine and caches the choice ([JAX.md](python/JAX.md#cpu-pass-fusion)).
per machine and caches the choice ([JAX.md](docs/JAX.md#cpu-pass-fusion)).
- **Packaging:** JAX-only installs no longer need PyTorch; the `torch` extra works with
PyTorch 2.1; transient build failures no longer suppress retries for 24 hours. The
version is read from one `VERSION` file.
Expand Down Expand Up @@ -72,12 +72,20 @@
`sdca`).
- **MATLAB:** shared names only, 1-based labels, `precision` follows `X`, `num_classes`,
`num_threads` and `seed`; `matsdca_loss` evaluates the distillation losses.
- **One array layout:** NumPy, Rust, C and C++ share row-major features `(n, d)`,
weights `(m, d)` (one weight vector per class) and duals `(n, m)`, passed to the native
library in place ([API.md](docs/API.md#array-layouts)). NumPy `Model.weights` and
`center` were Fortran-ordered `(d, m)`, and Rust copied its weights into `(d, m)`.
C++ `make_input_feature(n, d, X)` and `make_input_kernel(n, n_train, K)` take the
number of examples first (they took it second). `sdca_model_scores` (C),
`sdca::batch_scores` (C++) and Rust's `Model::decision_function` compute a batch of
scores with one BLAS `gemm`.
- **Stale builds:** Loading a native library built in the source tree before its sources
changed warns, and the test suite refuses to run against it.
- **CI:** `pytest --conformance ci` runs a profile of the conformance matrix (4,362 of
46,400 cells); the default still runs it all.
- **Docs and citation:** The README starts with the installation and shows the same two losses
in every interface; [TORCH.md](python/TORCH.md#benchmarks) lists CUDA benchmarks;
in every interface; [TORCH.md](docs/TORCH.md#benchmarks) lists CUDA benchmarks;
`CITATION.cff` prefers the TPAMI 2018 paper.
- **Known limits:** [API.md](docs/API.md#known-limits) lists the recorded `float64`
cancellation errors (below `5e-17` absolute) and the behavior of finite rows at
Expand Down Expand Up @@ -111,13 +119,13 @@
`ProximalDistillationLoss`, `SDCALoss`, and `LinearCrossEntropyLoss` (chunked linear head
+ cross-entropy without materializing `(N, V)` logits) across `float16`, `bfloat16`,
`float32` and `float64` on CPU and CUDA, with `torch.compile(fullgraph=True)` and
`DTensor` support ([TORCH.md](python/TORCH.md)).
`DTensor` support ([TORCH.md](docs/TORCH.md)).
- **JAX (`libsdca.jax`):** Classification and distillation losses for `jit`, `grad` and
`vmap`, with zero-copy XLA FFI custom calls on CPU ([JAX.md](python/JAX.md)).
`vmap`, with zero-copy XLA FFI custom calls on CPU ([JAX.md](docs/JAX.md)).
- **Python & scikit-learn:** `fit`, `loss_and_grad`, `distillation_loss_and_grad`,
`proximal_distillation_loss_and_grad`, `coordinate_update`, and `SDCAClassifier`
([python/README.md](python/README.md)).
- **Rust, C, C++ and MATLAB:** Rust crate `libsdca` ([rust/README.md](rust/README.md)),
([docs/PYTHON.md](docs/PYTHON.md)).
- **Rust, C, C++ and MATLAB:** Rust crate `libsdca` ([docs/RUST.md](docs/RUST.md)),
versioned C ABI `libsdca::sdca_c` ([docs/C.md](docs/C.md)), CMake target
`libsdca::libsdca` ([docs/CPP.md](docs/CPP.md)), and MEX gateways `matsdca_fit`,
`matsdca_loss` and `matsdca_prox` ([docs/MATLAB.md](docs/MATLAB.md)).
Expand Down
35 changes: 29 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,14 @@ OCTAVE_HOME=<prefix> tests/matlab/run_octave_tests.sh

- **Markers:** `slow` and `long` tests are scheduled first under `--dist worksteal` (CI
deselects `long`); `serial` tests run without `-n` on an unloaded machine.
- **Conformance matrix:** `--conformance ci` runs about a tenth of the conformance cells
(every loss on the default backends, cross-entropy and the multiclass hinge on every
backend, dtype and precision), as CI does; the default runs the full matrix. A manual CI
run (*Run workflow*, `conformance: full`) runs the full matrix on the hosted runners and,
when cells fail, uploads their records (`SDCA_CONFORMANCE_RECORD`) as an artifact; a
reviewed failure set of other hardware is added to `known_failures/` as a variant.
- **Conformance matrix:** The default runs the full matrix.
- `--conformance ci` runs about a tenth of the conformance cells (every loss on the
default backends, cross-entropy and the multiclass hinge on every backend, dtype and
precision), as CI does.
- A manual CI run (*Run workflow*, `conformance: full`) runs the full matrix on the hosted
runners and, when cells fail, uploads their records (`SDCA_CONFORMANCE_RECORD`) as an
artifact.
- A reviewed failure set of other hardware is added to `known_failures/` as a variant.
- **CUDA:** On a machine with an NVIDIA GPU, install a CUDA build of PyTorch (and
of JAX); the same pytest command then runs the CUDA tests, which skip otherwise.
`python tests/python/release_numerics.py --device cuda --output <dir>` also checks the
Expand Down Expand Up @@ -92,6 +94,27 @@ OCTAVE_HOME=<prefix> tests/matlab/run_octave_tests.sh
- **Numerical changes:** Every fix comes with a regression test against an independent
analytic, `mpmath` or KKT reference. Do not loosen a tolerance to make a test pass.

## Documentation

Keep API guides in `docs/`.

- **Structure:** Use the common structure: installation, a runnable quick start, task
examples and reference tables, precision, errors, benchmarks (when measured), and tests.
Keep language-specific sections such as C layouts and PyTorch transforms.
- **Shared contract:** Link to `docs/API.md` for shared definitions and defaults instead of
repeating the contract.
- **Style:** Use short paragraphs and tables for options and results; put longer derivations
after the usage examples.
- **Benchmarks:** Benchmark tables must identify hardware, versions, timing boundaries and
their evidence, including slower cases. Do not attribute one binding's timings to another.
- **Checks:** `python tools/check_docs.py` checks local links and heading anchors and
rejects unsupported math commands; `tools/quality.py` runs it in CI. Use `\mathrm{...}`
for named math functions.
- **Absolute links:** The Python and Rust guides also render on PyPI and crates.io, so use
absolute links there.
- **Rust guide:** Cargo packages `docs/RUST.md` as `RUST.md`; the build script feeds the
same guide to doctests in both a checkout and an extracted crate.

## Releases

```bash
Expand Down
2 changes: 1 addition & 1 deletion DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,4 @@ CBLAS; the headers need no separate source file.

The crate has no Rust dependencies and needs Rust 1.74+. `build.rs` compiles `src/c_api.cpp`
with `CXX` (default `c++`) and the built-in BLAS, or links an external CBLAS given by
`SDCA_BLAS` ([rust/README.md](rust/README.md#building)).
`SDCA_BLAS` ([docs/RUST.md](docs/RUST.md#building)).
3 changes: 1 addition & 2 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
recursive-include include *.h
include src/c_api.cpp src/c_api_internal.h src/tensor_losses.h
include sdca_build.py VERSION
include python/README.md
include python/TORCH.md python/JAX.md
recursive-include docs *.md
include python/libsdca/_cpu_*.h
include python/libsdca/_torch_entropy.cpp python/libsdca/_torch_*.h python/libsdca/_jax_ffi.cpp python/libsdca/_jax_cuda.cpp
include CHANGELOG.md DEPENDENCIES.md AGENTS.md CITATION.cff
Expand Down
46 changes: 24 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ classifiers with them.
- **Fast:** The losses are fused CPU and CUDA kernels, not chains of tensor operations. On an
A100 with 32k–128k classes, cross-entropy matches `torch.compile` and is 2.4–3× faster than
eager PyTorch, and the multiclass hinge is 4–8× faster than compiled and 8–14× faster than
eager PyTorch implementations ([benchmarks](python/TORCH.md#benchmarks)).
eager PyTorch implementations ([benchmarks](docs/TORCH.md#benchmarks)).
- **Accurate:** Losses and gradients of confident predictions are kept down to the subnormal
range of the output dtype: a naive `float32` cross-entropy rounds to zero from a margin of
about 17 (softmax collapse [7]), and `libsdca` stays accurate up to about 104. Distillation
gradients are exactly zero at equal student and teacher logits [6]. The entropy and
distillation losses are tested on CPU and CUDA against 100-digit `mpmath` references
([known limits](docs/API.md#known-limits)).
range of the output dtype ([known limits](docs/API.md#known-limits)).
- A naive `float32` cross-entropy rounds to zero from a margin of about 17 (softmax
collapse [7]), and `libsdca` stays accurate up to about 104.
- Distillation gradients are exactly zero at equal student and teacher logits [6].
- The entropy and distillation losses are tested on CPU and CUDA against 100-digit
`mpmath` references.
- **Many losses:** Cross-entropy, the multiclass hinge (Crammer–Singer SVM [1]), the top-*k*
hinge and entropy losses [4, 5], the multilabel hinge [2] and cross-entropy, logit
distillation and proximal variants, with nonsmooth or smoothed hinges, and
Expand All @@ -32,11 +33,12 @@ classifiers with them.
C++, C and MATLAB/Octave, with the same loss and option names in each
([API reference](docs/API.md)).
- **SDCA solver with a duality-gap guarantee:** SDCA trains linear and kernel classifiers
(multiclass, top-*k* and multilabel SVMs, softmax regression). The duality gap bounds the
distance to the optimum, so `status == "solved"` guarantees that the objective of the
returned model is within the relative tolerance `tol` of the optimal value
([results](docs/API.md#results)). Its steps use efficient projections onto the simplex and
the top-*k* cone [3, 4], which C++ and MATLAB also provide directly.
(multiclass, top-*k* and multilabel SVMs, softmax regression).
- The duality gap bounds the distance to the optimum, so `status == "solved"` guarantees
that the objective of the returned model is within the relative tolerance `tol` of the
optimal value ([results](docs/API.md#results)).
- Its steps use efficient projections onto the simplex and the top-*k* cone [3, 4], which
C++ and MATLAB also provide directly.

## Installation

Expand Down Expand Up @@ -73,12 +75,12 @@ options:

| Interface | Install | Guide |
| --- | --- | --- |
| Python (NumPy) | `pip install libsdca` | [python/README.md](python/README.md) |
| scikit-learn estimator | `pip install 'libsdca[sklearn]'` | [python/README.md](python/README.md#classifier) |
| PyTorch losses | `pip install 'libsdca[torch]'` | [python/TORCH.md](python/TORCH.md) |
| JAX losses (CPU) | `pip install 'libsdca[jax]'` | [python/JAX.md](python/JAX.md) |
| JAX losses (Linux NVIDIA GPU) | `pip install 'libsdca[jax-cuda]'` | [python/JAX.md](python/JAX.md) |
| Rust | `cargo add libsdca` | [rust/README.md](rust/README.md) |
| Python (NumPy) | `pip install libsdca` | [docs/PYTHON.md](docs/PYTHON.md) |
| scikit-learn estimator | `pip install 'libsdca[sklearn]'` | [docs/PYTHON.md](docs/PYTHON.md#classifier) |
| PyTorch losses | `pip install 'libsdca[torch]'` | [docs/TORCH.md](docs/TORCH.md) |
| JAX losses (CPU) | `pip install 'libsdca[jax]'` | [docs/JAX.md](docs/JAX.md) |
| JAX losses (Linux NVIDIA GPU) | `pip install 'libsdca[jax-cuda]'` | [docs/JAX.md](docs/JAX.md) |
| Rust | `cargo add libsdca` | [docs/RUST.md](docs/RUST.md) |
| C++ (header-only) | CMake target `libsdca::libsdca` | [docs/CPP.md](docs/CPP.md) |
| C | CMake target `libsdca::sdca_c` (`-DSDCA_BUILD_C_API=ON`) | [docs/C.md](docs/C.md) |
| MATLAB / Octave | `run matlab/make.m` / `tests/matlab/run_octave_tests.sh` | [docs/MATLAB.md](docs/MATLAB.md) |
Expand Down Expand Up @@ -119,7 +121,7 @@ CMake and BLAS options are in [DEPENDENCIES.md](DEPENDENCIES.md).
- **Compilers:** PyTorch and JAX compile small C++ extensions on first use when a C++17
compiler and Python headers are present (`build-essential python3-dev` on Debian/Ubuntu,
`xcode-select --install` on macOS); otherwise they fall back with a one-time warning
([build tools](python/TORCH.md#build-tools)). uv-managed and conda Pythons include headers.
([build tools](docs/TORCH.md#build-tools)). uv-managed and conda Pythons include headers.

</details>

Expand Down Expand Up @@ -226,10 +228,10 @@ struct train {
template <typename Objective>
void operator()(Objective objective) const {
double features[] = {1, 0, 0, 1, -1, -1}; // 3 examples, 2 features
double weights[6] = {}, dual[9] = {}; // W (2, 3), A (3, 3)
double weights[6] = {}, dual[9] = {}; // weights (3, 2), dual (3, 3)
std::vector<int> labels = {0, 1, 2};
auto context = sdca::make_context(
sdca::make_input_feature(2, 3, features),
sdca::make_input_feature(3, 2, features),
sdca::make_output_multiclass(labels.begin(), labels.end(), 3),
std::move(objective), weights, dual);
sdca::make_solver(context).solve();
Expand Down Expand Up @@ -298,8 +300,8 @@ An index of the guides is in [docs/](docs/README.md).

- [API reference](docs/API.md): loss formulas, names across languages, options, precision,
results, threads, environment variables and known limits
- Interface guides: [Python](python/README.md), [PyTorch](python/TORCH.md),
[JAX](python/JAX.md), [Rust](rust/README.md), [C++](docs/CPP.md), [C](docs/C.md),
- Interface guides: [Python](docs/PYTHON.md), [PyTorch](docs/TORCH.md),
[JAX](docs/JAX.md), [Rust](docs/RUST.md), [C++](docs/CPP.md), [C](docs/C.md),
[MATLAB](docs/MATLAB.md)
- [Dependencies](DEPENDENCIES.md), [contributing](CONTRIBUTING.md),
[guide for AI coding assistants](AGENTS.md) and [changelog](CHANGELOG.md)
Expand Down
Loading
Loading