Skip to content
DynamicsAndNeuralSystemsPublic

About

Comparative analysis of pairwise interactions in multivariate time series.

Topics

Resources

Code of conduct

Security policy

Stars

261 stars

Watchers

8 watching

Forks

Latest commit

Β 

History

621 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

pyspi logo

pyspi: Python Toolkit of Statistics for Pairwise Interactions


Python 3.11 | 3.12 | 3.13
pyspi downloads

pyspi is a comprehensive python library for computing statistics of pairwise interactions (SPIs) from multivariate time-series (MTS) data. The toolbox provides easy access to hundreds of methods for evaluating the relationship between pairs of time series, from simple statistics (like correlation) to advanced multi-step algorithms (like Granger causality). The code is licensed under the GNU GPL v3 license (or later).

Feel free to reach out for help with real-world applications. Feedback is much appreciated through issues, or pull requests.

Section Description
Installation Installing pyspi and its dependencies
Getting Started A quick introduction on how to get started with pyspi
Choosing an SPI set The bundled configs, and how to pick one
Running at scale Parallelism, checkpointing, and HPC clusters
SPI Descriptions A link to the full table of SPIs and detailed descriptions
Documentation A link to our API reference and full documentation on GitBooks
Contributing to pyspi A guide for community members willing to contribute to pyspi
Acknowledgement A citation for pyspi for scholarly articles
Our Contributors A summary of our primary contributors

Installation πŸ“₯

pyspi requires Python 3.11 or newer.

Upgrading from 2.x? Version 3.0 removes the Java/JIDT dependency and changes the Calculator API. See CHANGELOG.md for the migration table.

pip install pyspi

pyspi depends on a large scientific stack, so installing into a dedicated environment is strongly recommended:

python -m venv .venv && source .venv/bin/activate
pip install pyspi

Developing on pyspi itself, with uv:

git clone https://github.com/DynamicsAndNeuralSystems/pyspi.git
cd pyspi
uv sync                       # runtime + dev dependencies from uv.lock
uv run pytest -q              # fast test suite

Optional extras: .[tsfile] to load sktime/aeon .ts files, .[bench] for the compute-cost benchmark suite in bench/.

For a more detailed guide, see the full documentation, the troubleshooting guide, and alternative installation options.

Getting Started πŸš€

import numpy as np
import pyspi
from pyspi.calculator import Calculator

dataset = np.random.randn(5, 500)      # 5 processes, 500 observations
calc = Calculator(dataset=dataset)     # z-scores each process by default
calc.compute()                         # compute every SPI

calc.table                             # rows = processes, columns = (SPI, process)
calc.table["cov_EmpiricalCovariance"]  # one SPI's 5x5 matrix

calc.save("results.npz")               # canonical on-disk format
pyspi.load_table("results.npz")        # round-trips exactly

Results are stored as an (n_spis, M, M) array plus the SPI and process names, which is the data's natural shape. .csv is also accepted for eyeballing small results, but it is one-way and impractical for the full SPI set.

Or from the command line, writing a results table next to your data:

python -m pyspi compute --data ts.npy --config fast --n-jobs 4 --output results.npz

Try it on a bundled example dataset:

from pyspi.data import load_dataset, available_datasets

available_datasets()                   # forex, cml, standard_normal
calc = Calculator(dataset=load_dataset("forex"), config="fabfour")
calc.compute()

forex is quantised: every one of its seven processes contains repeated values (24–212 distinct values across 250 observations). Consequently, all KSG/kraskov variants refuse it under pyspi's continuous, tie-free input contract. pyspi does not provide a general discrete MI, TLMI or DI plug-in estimator; its symbolic support is TE-specific. Use an external discrete estimator when the missing quantities are needed.

Walkthrough tutorials in the full documentation: simple demonstration Β· finance Β· neuroimaging

Choosing an SPI set

Computing all 306 SPIs is expensive, and cost grows steeply in both the number of processes M and the series length T. config= takes either a bundled name or a path to your own YAML:

config= SPIs Use when
"full" (default) 306 You want everything and can afford it.
"fast" 197 General use; drops the slowest SPIs.
"benchmarked_p99" 291 Near-complete, with only the worst cost outliers removed.
"benchmarked_p95" 278 Good coverage/cost trade-off.
"benchmarked_p90" 264 Recommended default for large batches.
"benchmarked_p80" 237 Cost-constrained sweeps.
"sonnet" 14 One representative SPI per module (M01-M14).
"fabfour" 4 Smoke tests and quick sanity checks.

The benchmarked_p<N> sets keep the fastest N% of SPIs by measured amortized compute cost, so the cut point is the Nth percentile of the cost distribution. They were derived from a benchmark grid spanning M ∈ {4..64} and T ∈ {200..3200}; see bench/README.md for the methodology and bench/results/analysis/report.md for the measurements.

To build your own subset by keyword:

from pyspi.utils import filter_spis
filter_spis(["nonlinear", "directed"], output_name="my_spis")
calc = Calculator(config="my_spis.yaml")

Running at scale

Prefer one dataset per process. Calculator.compute(n_jobs=...) parallelises within a single dataset, but SPIs sharing a cache are grouped into one task that runs serially in one worker, so the makespan is floored by the longest single task. Measured on the benchmark grid at M=64, T=1600, that ceiling is 2.3-4.5x regardless of n_jobs, on every config. If you have more datasets than cores β€” the usual case on a cluster β€” run each dataset in its own single-core process instead. That scales close to linearly, isolates failures to one dataset, and schedules far faster as an array job.

Use n_jobs > 1 when you have fewer datasets than cores, when one dataset must finish under a walltime limit, or when memory forces you to share a single copy of a large dataset across workers.

# PBS array job: one dataset per task, one core each
#PBS -J 0-499
#PBS -l ncpus=1,mem=8GB
python -m pyspi compute --data "datasets/${PBS_ARRAY_INDEX}.npy" \
    --config benchmarked_p90 --checkpoint-dir "results/${PBS_ARRAY_INDEX}/"

--checkpoint-dir writes each SPI to <dir>/<identifier>.npy as it finishes and resumes from those on a re-run, so a job killed at the walltime limit picks up where it stopped. PYSPI_N_JOBS sets n_jobs from the environment.

When n_jobs > 1, workers pin their nested BLAS/OpenMP thread pools to one thread each to avoid oversubscription. macOS is an exception: its Accelerate BLAS cannot be pinned this way, so prefer n_jobs=1 there for a single dataset.

SPI Descriptions πŸ“‹

To access a table with a high-level overview of the pyspi library of SPIs, including their associated identifiers, see the table of SPIs in the full documentation. For detailed descriptions of each SPI, as well as its associated estimators, we provide a full breakdown in the SPI descriptions page of our documentation.

Documentation

The full documentation is hosted on GitBooks. Use the following links to quickly access some of the key sections:

Contributing to pyspi πŸ‘¨β€πŸ‘¨β€πŸ‘¦β€πŸ‘¦

Contributions play a vital role in the continual development and enhancement of pyspi, a project built and enriched through community collaboration. If you would like to contribute to pyspi, or explore the many ways in which you can participate in the project, please have a look at our detailed contribution guidelines about how to proceed. In contributing to pyspi, all participants are expected to adhere to our code of conduct.

SPI Wishlist

We strive to provide the most comprehensive toolkit of SPIs. If you have ideas for new SPIs or suggestions for improvements to existing ones, we are eager to hear from and collaborate with you! Any pairwise dependence measure, provided it is accompanied by a published research paper, typically falls within the scope for consideration in the pyspi library. You can access our SPI wishlist via the projects tab in this repo to open a request.

Acknowledgement πŸ‘

If you use this software, please read and cite this article:

Note that preprint and free-to-read versions of this article are available.

Click here for a BibTex reference:
@article{Cliff2023:UnifyingPairwiseInteractions,
	title = {Unifying pairwise interactions in complex dynamics},
	volume = {3},
	issn = {2662-8457},
	url = {https://www.nature.com/articles/s43588-023-00519-x},
	doi = {10.1038/s43588-023-00519-x},
	number = {10},
	journal = {Nature Computational Science},
	author = {Cliff, Oliver M. and Bryant, Annie G. and Lizier, Joseph T. and Tsuchiya, Naotsugu and Fulcher, Ben D.},
	month = oct,
	year = {2023},
	pages = {883--893},
}

Other highly comparative toolboxes 🧰

If you are interested in trying other highly comparative toolboxes like pyspi, see the below list:

  • hctsa, the highly comparative time-series analysis toolkit, computes over 7000 time-series features from univariate time series.
  • hcga, a highly comparative graph analysis toolkit, computes several thousands of graph features directly from any given network.
  • pyhctsa, a python implementation of the highly comparative time-series analysis toolkit.

Our Contributors 🌟

We are thankful for the contributions of each and everyone who has helped make this project better. Whether you've added a line of code, improved our documentation, or reported an issue, your contributions are greatly appreciated! Below are some of the leading contributors to pyspi:

License 🧾

pyspi is released under the GNU General Public License.

About

Comparative analysis of pairwise interactions in multivariate time series.

Topics

Resources

Code of conduct

Security policy

Stars

261 stars

Watchers

8 watching

Forks

Releases

Packages

Used by

Contributors

Languages