emdatabase

This is a simple project for aggregating different Electron Microscopy files which are hosted over different sources. It uses pooch to download datasets and should be used as a way to host simple example datasets for method validation.

Downloads go to ~/.cache/emdatabase by default; shared read-only locations can be added with emdatabase.add_location.

List of datasets https://electronmicroscopy.github.io/emdatabase/datasets.html

Installation

pip install emdatabase

Usage

Every dataset is a class under emdatabase.data. Calling download() fetches the file to the data directory, verifies its checksum, and returns a path handle. Files that are already present are not downloaded again.

import emdatabase.data as data
import hyperspy.api as hs

path = data.LayeredCuNb4DSTEM().download()
s = hs.load(path, lazy=True)

By default the download runs on a background thread so a notebook cell returns immediately. The handle it returns is the file path — a pathlib.Path subclass pointing at the file's final location — so you can hand it straight to a loader as above; it only blocks at the moment the file is actually opened. Read path.done to check progress without blocking, or call path.result() to wait explicitly. download(background=False) blocks instead, and returns the same type.

Any path pointing at the same file waits, however it was built, so a derived path (handle.parent / handle.name, handle.with_suffix(...)) behaves too. The exceptions are str(handle) and Path(handle): both hand back an ordinary value with no download attached, so hs.load(str(handle)) will not wait. Keeping str() non-blocking is deliberate — repr() needs it — so pass the handle itself.

Finding a dataset

search() is the browser widget's search box, callable from Python; filter() matches named fields. Both return dataset objects, so a result can be downloaded directly.

import emdatabase

emdatabase.list_datasets()                                     # everything
emdatabase.search("amorphous")                            # any field
emdatabase.search("jeol eels")                            # all terms, any field
emdatabase.filter(technique="4D-STEM", tags="Strain")     # exact, case-insensitive
emdatabase.filter(microscope_vendor=["JEOL", "Hitachi"])  # a list means any of
emdatabase.filter(downloaded=True)                        # what is already here

technique, tags, authors and version are several values per dataset, so they test membership: a dataset that is both in-situ and 4D-STEM matches either.

An unknown field raises rather than being ignored, so a typo cannot quietly return the whole index.

Configuration

Data lives in named locations. personal is the one writable location, where downloads go; every other one is read-only and searched first, so a copy already on a group drive is used instead of refetched.

from emdatabase import config

config.add_location("/group/example_data")                    # read-only
config.add_location("/big/disk/emdatabase", name="personal")  # where downloads go
config.locations()

[Location(name='example_data', path=PosixPath('/group/example_data'), kind='shared'),
 Location(name='personal', path=PosixPath('/big/disk/emdatabase'), kind='personal')]

locations() is the search order: the shared locations in the order they were added, then personal last. A location is named after the last component of its path unless you pass name=, and that name is the provenance — it is what catalogue.entry()["location"], emdatabase.filter(location="example_data") and the browser widget report for a copy found there. Nothing is written to a shared location unless you name it as a download's destination, which is how one is seeded.

Removing one takes either the name or the path; "personal" is not deleted but reset, putting downloads back in the default cache directory:

config.remove_location("example_data")
config.remove_location("/group/example_data")   # the same thing, by path
config.remove_location("personal")

Both functions persist to ~/.config/emdatabase/config.yaml, which is read on every import. Pass persist=False to change this process only, or use config.set as a context manager for a change that lasts for a block:

config.add_location("/scratch/em", name="personal", persist=False)  # this process
with config.set({"locations.personal": "/scratch/em"}):             # this block
    ...

The path does not have to exist when you add it — a share may be mounted later — but you get a warning saying so.

Seeding a shared location

destination= takes a location's name, which is how the copy gets onto the share in the first place — run it once, from an account with write access:

from emdatabase import data

data.CuZnHAADF().download(destination="example_data")

The file is written with your umask, so chmod it group-readable afterwards if your umask is not; emdatabase does not set permissions for you.

The key underneath

Configuration is dask-style: shipped defaults, then every *.yaml in ~/.config/emdatabase/ (or wherever EMDATABASE_CONFIG points), then environment variables, then config.set — each layer overriding the one before. There are two keys, and add_location is a wrapper over writing the first one yourself:

# ~/.config/emdatabase/config.yaml
locations:
  example_data: /group/example_data
  cluster: /cluster/em_data
  personal: /big/disk/emdatabase
check_updates: true

personal: null means pooch's cache directory (~/.cache/emdatabase on Linux), and config.data_dir() reports whichever it resolves to.

check_updates is whether downloading a model's latest weights asks the index on the project's main branch — kept current by a weekly job — whether newer weights have been published, and warns if they have; download(refresh=True) fetches them. Set it to false to skip the request.

On HPC, where a config file is often the wrong place to put a machine-specific path, set the same key from the environment instead — prefix EMDATABASE_, double underscore to nest — which needs no file and no write access:

export EMDATABASE_LOCATIONS__PERSONAL=/scratch/emdatabase
export EMDATABASE_LOCATIONS__GROUP=/group/example_data

Adding a dataset

Datasets are described by a YAML file in emdatabase/index/, one entry per file, validated against emdatabase/index/json-schema.json. The class name is generated from the top-level key:

MyDataset:
  description: What the data is, how it was acquired and how it is calibrated.
  source: https://zenodo.org/records/<record>/files
  file: MyDataset.zspy
  checksum: md5:<hash>
  size_bytes: 1200000000
  technique:
    - 4D-STEM
  license: CC-BY-4.0

A dataset may list more than one technique - in-situ 4D-STEM, cryo EELS - and is then listed under each of them. emdatabase/index/techniques.yaml is the vocabulary they come from: acquisition (how the data was taken) and ml_task (what a model does). A dataset declares acquisition techniques only; a kind: weights entry declares one of those plus the ML tasks it performs.

size_bytes is the file's Content-Length in bytes; the test suite checks it against the server on every run. emdatabase/index/vendors.yaml lists the microscope vendors and detector manufacturers already in use - a new one is fine, but a name close to one already on the list fails CI as a misspelling. A technique close to one in techniques.yaml fails the same way.

Open an issue with the new dataset template, fill in the Add Dataset form, or run python -m emdatabase.new_dataset <url>, which fetches the checksum and size, prompts for the rest and writes the file for you to open a pull request with. See CONTRIBUTING.md.

All three take one download link and split it into source, file and, when the file is not served at source/file, url. A Google Drive share link - what the share button copies - is rewritten to the uc?export=download&id=<id> link that serves the file. The form also has a local file picker: point it at the copy on your machine and it fills in the file name, size and md5, hashing the file in the browser without uploading it.

Neither web route needs the checksum or the size. A pull request carrying an entry that is missing either one has the file downloaded on GitHub and the fields filled in and pushed back to the branch; a pull request from a fork, whose branch cannot be pushed to, fails with the values to paste in instead.