CDIF converters

Scripts that convert metadata to or from CDIF JSON-LD. Each format lives in its own sub-directory with its converter(s), a README, and (where useful) mapping docs and examples. All of these are ordinary command-line tools — most also expose their conversion as an importable function.

Path	Direction	Format
soso2cdif.py	SOSO (file or URL) → CDIF	Science-on-Schema.org
soso/ConvertToSOSO.py	CDIF core+discovery → SOSO	Science-on-Schema.org
soso/ConvertFromSOSO.py	SOSO → CDIF core+discovery	Science-on-Schema.org
croissant/ConvertToCroissant.py	CDIF → Croissant 1.1	MLCommons Croissant
croissant/ConvertFromCroissant.py	Croissant → CDIF	MLCommons Croissant
DCAT/dcat_to_cdif.py	DCAT → CDIF	W3C DCAT
UMM/umm_to_cdif.py	NASA CMR UMM-C (file or URL) → CDIF core+discovery	NASA UMM-C JSON
FAIR2/fair2_to_cdif.py	FAIR² fair2.json (file or URL) → CDIF (reuses the Croissant converter for the Croissant core)	FAIR²
DDI/ddi_to_cdif.py	DDI Codebook 2.5 (Harvard Dataverse) → CDIF	DDI Codebook XML
DDI/ddi122_to_cdif.py	DDI 1.2.2 (ICPSR, source-agnostic) → CDIF	DDI XML
DDICodebook/ddi25_to_cdif.py	DDI Codebook 2.5 (source-agnostic) → CDIF	DDI Codebook XML
DDI-CDI/ddicdi_to_cdif.py	DDI-CDI 1.0 → CDIF (phased; all six phases)	DDI-CDI XML
DDI-CDI/ddicdi_jsonld_to_cdif.py	DDI-CDI JSON-LD → CDIF	DDI-CDI 1.1beta compact JSON-LD (CODATA export)
OHDSI/harvest_ohdsi_to_cdif.py	OHDSI gaiaCatalog schema.org records → CDIF core/discovery (+ provenance)	OHDSI gaiaCatalog
ROCrate/ConvertToROCrate.py	CDIF → RO-Crate 1.2	RO-Crate
ROCrate/ROCrateToCDIF.py	RO-Crate 1.2 → CDIF	RO-Crate
ROCrate/ValidateROCrate.py	RO-Crate structural + SHACL validator	RO-Crate

Harvesters (harvesters/)

Harvesters fetch records from a catalogue and convert them to CDIF. They differ from the converters above in where the input comes from, not in what they produce. For converting a SOSO file or URL you already have, use soso2cdif.py.

Path	Source	Notes
harvesters/geocodes_harvester.py	EarthCube GeoCodes SPARQL catalog → source landing-page JSON-LD → CDIF core/discovery	Moved from the validation repo

geocodes_harvester.py and soso/ConvertFromSOSO.py share one schema.org naming step, schemaorg_names.py: an unprefixed name resolves through the source record's @context (a term it defines, or schema.org when the context makes that the default vocabulary) before falling back to a fixed list of schema.org names, and only then to unk:.

Setup

git clone https://github.com/Cross-Domain-Interoperability-Framework/converters.git
cd converters
git submodule update --init      # pulls the `validation` repo (detect_conformance + schemas)
pip install -r requirements.txt

The validation repo is bundled as a git submodule at validation/. The format → CDIF converters import detect_conformance from it to set conformsTo from content, and the converters' --validate options frame and validate with its FrameAndValidate.py / ConformanceValidate.py against the schemas and frame it carries. The submodule is optional at runtime: every converter guards the import, so conversion still works without it — the conformsTo declaration just falls back to the built-in default, and --validate is skipped. Initialize the submodule to get content-derived conformance detection and validation.

The corpus regression builds (DCAT/build_corpus.py, DDI/build_ddi_corpus.py) validate against the building blocks themselves, so their schema and SHACL checks also need a metadataBuildingBlocks checkout beside this repo (../metadataBuildingBlocks); without it they report those checks as skipped.

The "detect conformance" convention (for format → CDIF converters)

A CDIF record declares which profiles it conforms to in a schema:subjectOf catalog record via dcterms:conformsTo. Rather than hard-coding a profile list, a converter should set that from the record's actual content: the detect_conformance.py module (in the bundled validation/ submodule) tests, per CDIF class, a presence SPARQL ASK (the elements the class introduces beyond its base) gated by a content-SHACL validity check, and apply_conformance() writes the detected cdif: URIs into subjectOf/dcterms:conformsTo (preserving any non-cdif: domain claims). detect_conformance has a remote-SHACL fallback, so it works without a local building-blocks checkout.

The format → CDIF converters — ConvertFromSOSO.py (and soso2cdif.py), ConvertFromCroissant.py, dcat_to_cdif.py, umm_to_cdif.py, fair2_to_cdif.py, and the DDI converters (ddi2cdif.py and the ddi_sssom_to_cdif.py engine behind it, ddi_to_cdif.py, ddi122_to_cdif.py, DDICodebook/ddi25_to_cdif.py, DDI-CDI/ddicdi_to_cdif.py, DDI-CDI/ddicdi_jsonld_to_cdif.py) — run detect_conformance by default and write the detected conformsTo into the catalog record. Each takes a --static-conformance flag that skips detection and keeps the converter's built-in default instead (and detection degrades to the built-in default automatically if detect_conformance or its deps are unavailable).

Detection decides, including when it finds nothing. A record whose content meets no CDIF profile declares no conformsTo rather than falling back to a built-in claim it has not earned — the fallback list exists only for --static-conformance and for when detect_conformance cannot be imported. Over-claiming is not a cosmetic error: the declared URI is what selects the schema and shapes to validate against, so a false claim also makes the record look checkable. ConformanceValidate and FrameAndValidate -v fail a record that over-claims; under-declaring is advisory. For example, the DDI converter's output carries cdi:InstanceVariable variables, so detection adds data_description to the declared profiles.

Why framing / structure matters

CDIF is JSON-LD built on schema.org (plus DDI-CDI, PROV, DQV, …). The JSON Schemas validate a framed tree rooted at schema:Dataset with schema:- prefixed property names and a required subjectOf catalog record. The other formats differ in shape — SOSO uses bare schema.org terms via @vocab and has no catalog record; Croissant nests RecordSet/Field; DCAT and DDI use their own vocabularies. Each converter reconciles those structural differences (see each format's README / mapping doc for the property-by-property detail).

---

soso2cdif.py — SOSO file/URL → CDIF

The front-end for the SOSO→CDIF engine. Reads a SOSO schema:Dataset from a local file path or an http(s) URL (extracting embedded application/ld+json from an HTML landing page when the response isn't JSON), converts it via soso/ConvertFromSOSO.py, derives conformsTo from content, and writes <input-stem>-cdif.json (or the -o path — a file or a directory).

python soso2cdif.py path/to/soso.json                  # -> soso-cdif.json
python soso2cdif.py https://example.org/dataset -o out.json
python soso2cdif.py soso.json --cdif core --static-conformance

soso/ — CDIF ↔ Science-on-Schema.org

Both are schema.org profiles, so conversion is structural alignment, not vocabulary translation. ConvertToSOSO.py reshapes the @context to SOSO style, strips schema: prefixes, drops the CDIF catalog record (SOSO has no equivalent), passes CDIF-only properties through open-world, and warns on SOSO-required gaps (--https emits https://schema.org/). ConvertFromSOSO.py prefixes names, rewrites the @context, ensures CDIF-required fields where derivable, wraps creators in a JSON-LD @list, and adds the required catalog record — with conformsTo from detect_conformance. Mapping detail: soso/README.md and the property-by-property comparison in ../doc-corediscovery/documents/CDIF-Discovery-vs-SOSO-comparison.md (in a sibling doc-corediscovery checkout) (ESIP issue #283).

croissant/ — CDIF ↔ MLCommons Croissant

ConvertToCroissant.py converts CDIF to Croissant 1.1 for ML dataset discovery (DataDownload → cr:FileObject; variableMeasured + physical mapping → cr:RecordSet/cr:Field; CDIF-only properties passed through). The lossy inverse ConvertFromCroissant.py converts Croissant (1.0 or 1.1) back to CDIF DataDescription/Discovery and sets conformsTo via detect_conformance. Mapping docs: croissant/CDIFtoCroissant.md (forward) and croissant/CroissantToCDIF.md (inverse).

DCAT/ — DCAT → CDIF

dcat_to_cdif.py converts a DCAT JSON-LD catalog or dataset to CDIF schema.org form, mapping DCAT / Dublin Core properties to their schema.org equivalents per the CDIF DCAT implementation guide. It can list the datasets in a catalog, convert a selection, and optionally (--validate) frame and validate each record against the schema of every profile it declares. See DCAT/README.md.

UMM/ — NASA CMR UMM-C → CDIF

umm_to_cdif.py converts NASA Common Metadata Repository collection metadata (UMM-C JSON, from a file or a CMR search URL) to CDIF core + discovery, reading mappings/ummc-to-cdif.sssom.tsv at runtime. It can pull the collection's associated UMM-Var records as variables. See UMM/README.md.

FAIR2/ — FAIR² → CDIF

fair2_to_cdif.py converts a FAIR² data package (fair2.json, file or URL): the Croissant core goes through the Croissant converter, and what FAIR² adds through mappings/fair2-to-cdif.sssom.tsv (with an aliases table for older exports). See FAIR2/README.md.

DDI/, DDICodebook/, DDI-CDI/ — DDI → CDIF

DDI/ddi2cdif.py is the single entry point: it sniffs the DDI flavor and version and dispatches. DDI Codebook 1.2.2 and 2.5 go through one data-driven engine, DDI/ddi_sssom_to_cdif.py, which applies the DDI worksheets (compiled to mappings/ddi_mappings.json): study-level metadata → the dataset; <var> → schema:variableMeasured (cdi:InstanceVariable); <fileDscr> → schema:DataDownload (cdi:TabularTextDataSet) with physical mappings. DDICodebook/ddi25_to_cdif.py is a thin 2.5 shim over it, and DDI/ddi_to_cdif.py a Harvard-Dataverse layer that adds file size, checksum and column headers from the Dataverse API. DDI-CDI 1.0 XML goes to DDI-CDI/ddicdi_to_cdif.py, and DDI-CDI compact JSON-LD to DDI-CDI/ddicdi_jsonld_to_cdif.py. See DDI/README.md.

OHDSI/ — OHDSI gaiaCatalog → CDIF

harvest_ohdsi_to_cdif.py converts the schema.org records of the OHDSI gaiaCatalog (16 in OHDSIMetadata/) to CDIF core + discovery (+ provenance from their ETL action). It is hand-coded; mappings/ohdsi-to-cdif.sssom.tsv documents it and check_ohdsi_mappings.py keeps the two in step. See OHDSI/README.md.

---

Validating converter output

# CDIF output -> a CDIF profile schema (frame first; schemas live in the submodule)
python validation/tools/FrameAndValidate.py out.json -v \
    --schema validation/CDIFDiscoverySchema.json --frame validation/CDIF-frame-2026.jsonld

# SOSO output -> SOSO v1.3 SHACL (use ConvertToSOSO --https so the shapes target it)
#   soso_common_v1.3.0.ttl from the ESIP science-on-schema.org repo

# Croissant output -> mlcroissant
mlcroissant validate --jsonld out-croissant.json

Repository layout

converters/                  (repository root)
├── soso2cdif.py             SOSO file/URL -> CDIF (front-end for soso/)
├── sssom_engine.py          shared table-driven mapping engine
├── soso/  croissant/  DCAT/  UMM/  FAIR2/  OHDSI/        the converters
├── DDI/  DDICodebook/  DDI-CDI/  ROCrate/                 the converters (cont.)
├── harvesters/              catalogue harvesters (GeoCodes)
├── mappings/                SSSOM crosswalk tables for every converter
└── validation/              git submodule: the CDIF `validation` repo

Root files

File	Description
soso2cdif.py	Front-end for the SOSO→CDIF engine: reads a SOSO record from a path or http(s) URL and writes CDIF (see below)
schemaorg_names.py	Resolves the unprefixed names in a schema.org record to CURIEs through the record's own @context; shared by soso/ConvertFromSOSO.py and harvesters/geocodes_harvester.py
sssom_engine.py	The shared table-driven mapping engine (MappingSet) that the Croissant and FAIR² converters read their SSSOM tables through (DCAT, UMM and DDI have their own readers)
requirements.txt	Python dependencies (see Setup)
LICENSE, LICENSE-CC-BY-4.0	Apache-2.0 (software) and CC BY 4.0 (mappings, docs, example metadata); see License
README.md, CLAUDE.md	This file, and the project guide for Claude Code
.gitignore, .gitmodules	Git configuration; .gitmodules declares the validation submodule

Converter subdirectories

Each holds its converter(s), a README.md, and (where useful) mapping docs and example corpora.

Directory	Contents
soso/	ConvertToSOSO.py / ConvertFromSOSO.py (CDIF ↔ ESIP Science-on-Schema.org), the check_soso_mappings.py drift-checker, examples/, and README.md
croissant/	ConvertToCroissant.py / ConvertFromCroissant.py (CDIF ↔ MLCommons Croissant 1.1), check_croissant_mappings.py, the mapping docs CDIFtoCroissant.md / CroissantToCDIF.md (+ AGENTS.md and background .docx notes), and the croissantExamples/ and MLCroissantExamples/ corpora
DCAT/	dcat_to_cdif.py (table-driven DCAT → CDIF), build_corpus.py (regression harness), make_coverage_xlsx.py + dcat_profile_coverage.xlsx, the dcat-ap-vs-dcat-us.md comparison, README.md, and the corpora: dcat-examples/ (783 upstream files), dcatExamplesOK/ (the 239 that describe a dcat:Dataset, including the PSDI catalogue), and cdifOK/ (289 converted CDIF records; the 50 in cdifOK/psdi/ were made with dcat_to_cdif.py, not build_corpus.py)
DDI/	The DDI entry point ddi2cdif.py (flavor sniff + dispatch), the data-driven engine ddi_sssom_to_cdif.py, ddi122_to_cdif.py (1.2.2), the Harvard-Dataverse-specific ddi_to_cdif.py, build_ddi_corpus.py, the DDI Codebook 1.2.2 XSD, Examples/, and README.md
DDICodebook/	ddi25_to_cdif.py (a thin DDI Codebook 2.5 shim over the engine), the 2.5 XSD, the ddi25-additions-cdif-mapping.md notes, Examples/, and README.md
DDI-CDI/	ddicdi_to_cdif.py (DDI-CDI 1.0 XML → CDIF), ddicdi_jsonld_to_cdif.py (DDI-CDI JSON-LD → CDIF), Examples/, and README.md
UMM/	umm_to_cdif.py (NASA CMR UMM-C → CDIF), the UMM-C and UMM-Var JSON Schemas under schemas/, example CMR records and their CDIF conversions under examples/, and README.md
FAIR2/	fair2_to_cdif.py (FAIR² → CDIF, a FAIR² extension pass over the Croissant converter), the FAIR² specification example and its conversion, and README.md
OHDSI/	harvest_ohdsi_to_cdif.py (OHDSI gaiaCatalog → CDIF), its corpus (OHDSIMetadata/, 16 source records, → cdifMetadata/), the check_ohdsi_mappings.py drift-checker, README.md and AGENTS.md. Formerly the separate OHDSI repo
harvesters/	geocodes_harvester.py: EarthCube GeoCodes SPARQL catalogue → CDIF
ROCrate/	ConvertToROCrate.py (CDIF → RO-Crate 1.2), ROCrateToCDIF.py, ValidateROCrate.py (structural + optional SHACL), example RO-Crate/CDIF records, requirements.txt, and README.md
mappings/	The SSSOM crosswalk tables (*.sssom.tsv + .yml sidecars) for every converter path, the alias tables, the compiled ddi_mappings.json, the sync scripts (sync_sssom.py, sync_ddi_mappings.py), ddiwalk_lib.py, and README.md. See the next section
validation/	git submodule — the CDIF validation repo, providing detect_conformance.py, the CDIF schemas, the frame, and tools/. Run git submodule update --init to populate it (see Setup)
.github/	CI: workflows/check-mappings.yml runs the SOSO, Croissant and OHDSI mapping drift-checkers

---

REST API & Docker Service

The repository includes a FastAPI service (cdif_converter/) implementing the REST API contract patterned after DDICDI_generator_gen2. It provides both instant one-shot file conversion and session-based workflow management with auto-detection of source format (Croissant, SOSO, DCAT, DDI, RO-Crate, NASA UMM-C, FAIR²).

API Endpoints

Method & Route	Description
POST /api/convert	One-shot conversion: Uploads a file, auto-detects source format, converts to target_format (default: cdif), and streams back the converted document.
POST /api/files	Session upload: Uploads a metadata file, auto-detects format, extracts variables/preview, and returns session file_id.
GET /api/files/{id}	Retrieve file metadata, detected format, and variables list.
GET /api/files/{id}/preview?rows=N	View first N rows / variable summaries.
POST /api/files/{id}/output	Generate live preview of converted JSON-LD (returns JSON-LD).
POST /api/files/{id}/generate	Generate and download converted file (<stem>_<FORMAT>.jsonld).
PUT /api/files/{id}/roles	Update variable role assignments (measure, identifier, attribute).
POST /api/files/{id}/variables	Add a variable / column definition.
PATCH /api/files/{id}/variables/{name}	Edit variable labels, format, data type, or concepts.
DELETE /api/files/{id}/variables/{name}	Delete a variable from the session.
DELETE /api/files/{id}	Delete session working directory and files.
GET /api/health	Service health status and version.
GET /api/info	Supported source and target formats, extensions, and endpoints.

Running with Docker

Build and run the standalone container:

# 1. Build the Docker image
docker build -t cdif-converters .

# 2. Run the container on port 8082
docker run --rm -p 8082:8000 --name cdif-converters cdif-converters

# 3. Access Swagger API documentation
open http://localhost:8082/docs

Running with Docker Compose

A docker-compose.yml is provided at the repository root:

# Start in background
docker compose up -d --build

# Inspect container logs
docker compose logs -f

# Check health
curl http://localhost:8082/api/health

# Stop service
docker compose down

Optional environment variables (can be placed in a .env file):

• PORT: internal port (default: 8000)
• CONVERTERS_PORT: exposed host port (default: 8082)
• CONVERTER_API_KEY: optional secret key; when set, requests require header X-API-Key: <key>
• CONVERTER_UPLOAD_TTL_SECONDS: session TTL (default: 21600 / 6 hours)

---

Testing Examples

1. Health & Service Info

# Check health
curl -s http://localhost:8082/api/health | jq .

# List supported formats & endpoints
curl -s http://localhost:8082/api/info | jq .

2. One-Shot Conversion (POST /api/convert)

Croissant to CDIF

curl -X POST "http://localhost:8082/api/convert" \
  -F "file=@croissant/MLCroissantExamples/openml-iris.json" \
  -F "target_format=cdif" \
  -o iris_CDIF.jsonld

CDIF to Croissant 1.1

curl -X POST "http://localhost:8082/api/convert" \
  -F "file=@iris_CDIF.jsonld" \
  -F "target_format=croissant" \
  -o iris_Croissant.json

SOSO (Science-on-Schema.org) to CDIF

curl -X POST "http://localhost:8082/api/convert" \
  -F "file=@soso/examples/soso-dataset-example.json" \
  -F "target_format=cdif" \
  -o soso_CDIF.jsonld

DCAT to CDIF

curl -X POST "http://localhost:8082/api/convert" \
  -F "file=@DCAT/dcatExamplesOK/01-dcat-ap/3.0.1-core/dataset-complete.jsonld" \
  -F "target_format=cdif" \
  -o dcat_CDIF.jsonld

DDI Codebook XML to CDIF

curl -X POST "http://localhost:8082/api/convert" \
  -F "file=@DDI/Examples/XML/MWI-PHIM-CAC-2026-v01.xml" \
  -F "target_format=cdif" \
  -o ddi_CDIF.jsonld

3. Session-based Workflow

Step 1: Upload and get file_id

UPLOAD_RES=$(curl -s -X POST "http://localhost:8082/api/files" \
  -F "file=@croissant/MLCroissantExamples/openml-iris.json" \
  -F "target_format=cdif")

FILE_ID=$(echo $UPLOAD_RES | jq -r .file_id)
echo "Uploaded file_id: $FILE_ID"

Step 2: Inspect metadata and variables

curl -s "http://localhost:8082/api/files/$FILE_ID" | jq .

Step 3: Preview the converted JSON-LD output

curl -s -X POST "http://localhost:8082/api/files/$FILE_ID/output" \
  -H "Content-Type: application/json" \
  -d '{"target_format": "cdif"}' | jq .

Step 4: Download converted document

curl -X POST "http://localhost:8082/api/files/$FILE_ID/generate" \
  -H "Content-Type: application/json" \
  -d '{"target_format": "cdif"}' \
  -O -J

Step 5: Clean up upload session

curl -X DELETE "http://localhost:8082/api/files/$FILE_ID"

---

Mappings (SSSOM)

mappings/ holds an SSSOM mapping set for each converter path — cdif→soso, soso→cdif, cdif→croissant, croissant→cdif, dcat→cdif, ummc→cdif, fair2→cdif, ohdsi→cdif, and the DDI worksheets (ddi-common, ddi25, ddi122) — plus alias tables for DCAT, Croissant and FAIR² that map variant source IRIs onto the ones a table names. DCAT, UMM, Croissant, FAIR² and DDI read their tables at runtime; the SOSO and OHDSI tables document hand-coded converters and are checked against them in CI. See mappings/README.md.

License

This repository is dual-licensed by content type:

Content	License
Software: the Python converters, checkers and scripts (*.py) and the CI workflow (.github/)	Apache License 2.0 — see LICENSE
Mapping tables (mappings/*.sssom.tsv and their .yml sidecars, mappings/ddi_mappings.json), documentation (*.md), and the example metadata produced by these converters (e.g. DCAT/cdifOK/, the cdif/ folders under each format's examples, the CDIF→Croissant output examples)	Creative Commons Attribution 4.0 International (CC BY 4.0) — see LICENSE-CC-BY-4.0

Third-party material bundled here for testing and reference keeps its own license: the upstream source examples (e.g. DCAT/dcat-examples/, DCAT/dcatExamplesOK/, croissant/MLCroissantExamples/, the harvested Croissant exports in croissant/croissantExamples/, the XML/ source folders, the UMM-C source records, the RO-Crate inputs), the DDI Codebook XSDs, the NASA UMM schemas under UMM/schemas/, the FAIR² specification example in FAIR2/examples/, and the background .docx papers in croissant/. The validation/ submodule is a separate repository with its own license.