Installation Instructions

1. Download and unzip the file from the Releases section.
2. Select the datasets this node will process, as described in Node configuration.
3. Run the startup script:
   • Linux: startup.sh
   • Windows: startup.ps1

Note:
Running the script for the first time will trigger the download of approximately 6 GB of data required to build the necessary Docker images. Make sure you have Docker installed and sufficient disk space available.

Installation requirements

• Docker
• Docker compose

Node configuration

A clean deployment serves no dataset: it downloads no mapping until it is told which datasets this node processes. Uncomment and edit the datasetsList line in:

• Linux and macOS: local_env.sh
• Windows: local_env.ps1

datasetsList="40dbe9fb-c607-445d-a582-dea531b676b1"

Each code is the Dataset ID from the EUCAIM catalogue, the same one used as prefix of the input files. Several datasets are separated by commas, with no spaces.

On every start the mappings of the selected datasets are downloaded from EUCAIM/etl-mappings into the flows directory. If any selected dataset has no mapping there, the startup stops and reports which ones are missing.

The same file holds two optional settings:

• downloadFlows="false" uses the mappings already in flows instead of downloading them, which is what you want while adjusting a mapping locally (see FAQ 3)
• logsRetentionDays="30" is the number of days the exported log CSV files are kept in output_data; older ones are deleted on each launch. The rows stay in the internal database, so nothing is lost. Set it to 0 to delete nothing

Basic usage

The ETL service runs continuously in the background. In the current version, the execution of the pipeline is triggered by placing the dataset files into the input_data directory.

The results are generated as a set of CSV files in the corresponding output_data subdirectories.

Dataset requirements

• Please include as prefix for ALL the files, the Dataset ID. Example 4fcdd34b95f8eed2a3d07291e4c2173e_bcancerA_sample.csv. This Dataset ID can be found in the EUCAIM catalogue
• CSV input files must be comma (,) separated
• CSV input files must use dot (.) as a decimal separator

Note: The dataset must have exactly the same header as the template sample file(s) provided to EUCAIM team for building and adjusting the mapping.

Dataset input folder

The dataset file with clinical data must be placed in:

• input_data\clinical_data

The csv file with the extraction of DICOM tags must be placed in:

• input_data\image_metadata

The csv file declaring the imaging timepoints must be placed in:

• input_data\image_timepoints

Please push first at least once, the files for clinical data and DICOM metadata, before submitting the file with the imaging timepoints.

Dataset output folders

The EUCAIM CDM deliverable, one CSV per CDM table and dataset, is written here:

• output_data\cdm

Files are named <dataset_id>__<table>.csv and their header is the column list of that table in the CDM, so they can be loaded as they are. A table with no rows for that dataset gets no file: what describes the whole model is the accompanying <dataset_id>__manifest.csv, which lists every CDM table with its row count, including the ones at zero that have no file. Read it before the rest, and take the file list from it rather than from the directory.

Each bundle also carries a <dataset_id>__report.md: how the dataset ingestion went, in one page meant to be read and shared.

The per-run export of the ingestion tables, holding the clinical data and the DICOM metadata as the mapping wrote them, is written here:

• output_data\ingestion

These files are a trace of what each pipeline run loaded, useful for reviewing a mapping without querying the database. They follow the ingestion tables, not the EUCAIM CDM: the identifiers that link the entities are not exported, dates keep the format of the source, and several CDM entities have no file at all. They are kept for the same number of days as the log folders below. Do not use them as the CDM deliverable.

Additional output to support the review of the mapping process is written here:

• output_data\mapping_logs

Additional output with info and error logs for the pipelines steps being processed is written here:

• output_data\etl_process_logs

Each process_logs_*_records.csv holds one row per pipeline step, with the dataset, the stage, the result and a message. Files the ETL discarded, and the reason, are recorded there too.

That same folder holds etl-errors.log, which collects every warning and error raised while the pipelines run, with the full detail of the underlying failure. It is the first file to look at when a dataset is not processed, and the one to attach when reporting a problem.

License

Except as otherwise noted this software is licensed under the Apache License, Version 2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

---

FAQ (frequently Asked Questions)

1. After the input file is copied into input_data\clinical_data, nothing happens and the file remains there, seemingly not being processed

First check that the dataset is selected: the file is only processed if its Dataset ID prefix is listed in datasetsList (see Node configuration). The startup script prints No dataset selected or the list of datasets it is deploying.

If the dataset is selected, most likely the setup script init.sh is not being properly executed.
In some instances, due to permission issues, this file cannot run inside the NiFi Docker container.

To check if this is the case, try:

docker logs nifi | grep init
sh: 1: /opt/nifi/init.sh: Permission denied

If you see the "Permission denied" message, the solution is to ensure that the file has valid read and execution permissions for the user running the startup script of the ETL.

2. Ingesting the imaging timepoints does not seem to work, and/or is generating errors

Please check the csv file containing the imaging timepoints includes the expected header, with at least the columns Timepoint (also "ImagingTimepoint" is valid as column name), StudyInstanceUID and PatientID. The column names are matched exactly, with this capitalisation: another spelling (for instance "timepoint") is not recognised and leaves the studies with timepoint 0. The values of the timepoint column must be whole numbers (1, 2, 3, not 1.0), and the StudyInstanceUID and PatientID must be the ones of the DICOM metadata of the dataset: a study the file does not reach keeps a null timepoint, with no error, and is left out of the CDM together with its series.

To see how far the studies of a dataset got, run scripts/check_data.sh <dataset_id> (or scripts/check_data.ps1): its last block, check_imaging_timepoints.sql, counts the studies with and without timepoint and link, and tells why the others are not linked. It prints no identifiers, so its output can be shared as it is.

3. A mapping placed by hand in flows is replaced every time the ETL is started

On every start the setup script downloads the mappings of the selected datasets from EUCAIM/etl-mappings into the flows directory, overwriting whatever is there. This is what keeps a node aligned with the published mappings.

While a mapping is being adjusted, or on a machine with no access to GitHub, the download can be turned off. In local_env.sh (Linux and macOS) or local_env.ps1 (Windows), set:

downloadFlows="false"

The mappings already present in flows are then used as they are. Every dataset listed in datasetsList must have its mapping file in that directory, otherwise the startup stops and reports which ones are missing.

Remember to set it back to true, or remove the line, to resume receiving mapping updates.

Note: The GitHub API allows 60 calls per hour and IP address. If the download fails because that limit is spent, set GITHUB_TOKEN in the environment with a personal access token, or set downloadFlows="false" and place the mappings in flows by hand.

4. The initilization of the nifi-postgres container (the internal ETL database) fails, when it was previously working

Unfortunatly, updating from any release prior to 0.3.X to these or later releases, renders the database storage not compatible. If it s absolutely necessary to preserve previously ingested data, change the version in the docker-compose or contact with us for help. If you can afford to re-ingest the dataset files, please run only the first time after upgrading the ETL:

docker compose down -v

Afterwards execute the launch script as normal, and ingest the files following the same instructions.

5. I updated the ETL (git pull) and a new lookup value, ontology concept or procedure is not taking effect

PostgreSQL runs the files in Database/ only the first time its volume is created, so an update of those files never reaches a database that already exists. You do not need to recreate it (and lose the ingested data): reload just the reference data and code with

scripts/reload_database.sh            # lookup tables + procedures (the usual case)
scripts/reload_database.sh all        # also the ontology concepts

On Windows use .\scripts\reload_database.ps1 (same groups: aux, procedures, ontology, all).

The ingested and exported data are not touched. Everything selected is applied in one transaction, so an error leaves the database as it was. Rows added by hand to a Lookup* table are replaced by the content of the file: add --backup (-Backup in PowerShell) to get a dump first. NiFi does not need to be restarted for these changes. A change in a Groovy script or in a flow does: the flows are loaded again when the ETL starts.