Validate de-identified DICOM with independent validators

Validate de-identified DICOM with independent validators#

De-identification must not break the files it releases: MIDI-BP-03 in the requirements register asks that each output still conform to its original IOD. The engine checks its own output against its IOD before release, but that check shares the engine’s reading of the standard. This page describes a second check that shares nothing with the engine: three validators written by others check each released instance and its input, and the comparison reports only what the de-identification changed.

Continuous integration runs it on every pull request, over the synthetic corpus, under each preset. This page also shows how to run it yourself, on the corpus or on a release of your own, such as the MIDI-B benchmark’s.

The code is in pymedphys._dicom.deidentify: dicom_validators.py runs the validators, dicom_validation.py compares inputs with outputs, and dicom_validation_known.toml lists the explained differences. The command is the development module pymedphys._dicom.deidentify.dicom_validation_command. It is not part of the pymedphys command line, because nothing de-identification related joins the public command line before the first supported release.

The validators#

Validator

From

Licence

Version in CI

What it checks

dciodvfy

dicom3tools, by David Clunie

BSD-3-Clause

Ubuntu 24.04’s package, snapshot 20240118131615

Each instance against the modules and attributes of its IOD, each value against its VR and VM, defined terms and enumerated values, and the file’s encoding.

dcentvfy

dicom3tools

BSD-3-Clause

as dciodvfy

That the instances of one patient give the same values for the attributes of each information entity: patient, study, series, frame of reference, and equipment.

dicom-validator

the pydicom project

MIT

0.9.0, pinned in the tests extra

Each instance against its IOD, as read from the DocBook source of PS3.3, PS3.4, and PS3.6 of the edition that the engine’s tables are generated from, which it downloads from NEMA.

None of them is installed for users of PyMedPhys: dicom-validator is in the tests extra, and dicom3tools is a separate program.

The dicom3tools snapshot that Ubuntu packages is from January 2024, so its dictionary lacks attributes that the standard has added since, such as Coding Scheme Resources Sequence (0008,001D). The CI job lists the newest snapshot on the dicom3tools site in its summary, so the gap stays visible. dicom-validator reads the current edition, so it covers what that snapshot cannot.

How the comparison works#

Each validator checks each released instance’s input and its output; dcentvfy checks each patient’s inputs together, and then their outputs. A finding is introduced where an output gives it more often than its input. A finding that the input already had is not charged to the de-identification, and one that the output no longer gives is counted as resolved.

Each introduced finding is then:

  • explained, by an entry in dicom_validation_known.toml, whose category says why: profile, where the action that PS3.15 Table E.1-1 gives the attribute causes it; validator, where it is a limitation or heuristic of the validator and the output conforms; or engine, a defect of the engine’s output, recorded until it is fixed. An entry whose explanation rests on a property of the output names it, and explains the finding only in an output that has it;

  • not comparable, where it lies below an attribute whose contents the validator could not read in the input, because its dictionary lacks the attribute and the input is in Implicit VR or gives it the VR UN, while it could read them in the output; or

  • unexplained.

The comparison passes when nothing introduced is unexplained, and a validator that checked an input but could not check its output gives an unexplained finding too.

The results hold no value and no path. Every validator quotes values, and dicom3tools also prints file names, so each message is reduced to the validator, the severity, the attribute as tags without item numbers or private creators, the message’s text before the first value it quotes, in angle brackets or quotation marks or after an equals sign, and the module or information entity it names. That text, and any reason that follows the value, is kept only where it is made of strings compiled into the validator’s own executable, with VRs in brackets, a few short words between them, and counts, which read <n>. Any other message reads message not shown. A value can span lines, and its later lines can look like messages, so the module, the information entity, an attribute keyword that pydicom’s dictionary lacks, and the IOD’s name are kept only where the validator’s executable holds them too. A line in any other form is counted without its text.

Known differences#

At present the synthetic corpus needs five known differences, each listed with its reason in every report:

  • profile: Study Date, Study Time, and Study ID, which the Basic Profile empties (action Z), are needed to build a DICOMDIR, so dciodvfy warns.

  • validator: dciodvfy warns about every Person Name without a component delimiter, such as the dummy value DEIDENTIFIED, as a retired form; its dictionary predates RT Assertions Sequence (0044,0110); and its condition for the Common Instance Reference Module’s Referenced Series Sequence (0008,1115) and Studies Containing Other Referenced Instances Sequence (0008,1200) counts references from only a few sequences, so it reports both as present without need once the Basic Profile removes or empties Referenced Image Sequence and Source Image Sequence in the functional groups of enhanced images. PS3.3 Section C.12.2 counts references from any module, and the corpus’s outputs still reference the instances they list elsewhere, for example from Referenced Image Evidence Sequence (0008,9092), so their condition is met. These two entries apply only to an output that still references, outside the module, an instance that the sequence lists; where the last such reference is gone, the finding is unexplained. Where no reference to an instance survives outside the module, the engine removes both sequences.

Run it on the synthetic corpus#

The corpus tests build the synthetic corpus, run the engine over it once for each policy, each defined by a module of lib/pymedphys/tests/dicom/corpus_runs, and compare each run. They hold basic, basic-clean-descriptors, and basic with the corpus’s images compressed, and each option joins them as a module of its own once the engine applies the option. They need Linux or macOS with dicom3tools on the PATH.

  1. Install Git and uv, as in the MIDI-B benchmark guide.

  2. Install dicom3tools. On Debian or Ubuntu, sudo apt-get install dicom3tools. Elsewhere, build it from a source snapshot on the dicom3tools site, following its instructions, and put dciodvfy and dcentvfy on the PATH. A newer snapshot than CI’s can give fewer findings, since its dictionary knows more attributes.

  3. Get PyMedPhys with the tests extra, and run the tests:

    git clone https://github.com/pymedphys/pymedphys.git
cd pymedphys
uv sync --no-dev --extra tests
export PYMEDPHYS_DICOM_VALIDATORS=required
export PYMEDPHYS_DICOM_VALIDATION_REPORT="$PWD/dicom-validation"
uv run --no-dev --extra tests pymedphys dev tests --dicom-validators \
  lib/pymedphys/tests/dicom/test_deidentify_dicom_validation_corpus.py

The tests are marked dicom_validators, so the ordinary test run leaves them out. Without PYMEDPHYS_DICOM_VALIDATORS=required they skip when a validator is missing, rather than fail. The first run downloads the DocBook source into ~/.pymedphys/dicom-validator, or into the directory that PYMEDPHYS_DICOM_VALIDATOR_STANDARD names. Each run’s report is written into a directory of dicom-validation named after the run.

In CI the job is DICOM validation of de-identified output, in the Unit Tests workflow. Its summary shows each run’s report, and its dicom-validation artefact holds the reports as Markdown and JSON.

Run it on a release of your own#

The command compares a release with its inputs. It pairs them by SOP Instance UID, through a CSV file with the columns id_old and id_new, and finds each output by its file name, the replacement SOP Instance UID with the suffix .dcm, as a run names it.

After a MIDI-B benchmark run, whose work directory holds the release and the UID mapping:

uv sync --no-dev --extra tests
uv run --no-dev --extra tests python -m pymedphys._dicom.deidentify.dicom_validation_command midi \
  --source /data/midi-b/validation/synthetic \
  --work /data/midi-b/runs/validation-basic-1 \
  --out /data/midi-b/runs/validation-basic-1-dicom-validation

For another release, give the release and the mapping instead:

uv run --no-dev --extra tests python -m pymedphys._dicom.deidentify.dicom_validation_command compare \
  --source /data/inputs \
  --release /data/runs/release-1/release \
  --uid-mapping /data/runs/uid-mapping.csv \
  --out /data/runs/release-1-dicom-validation

  • --out must not exist; its parent must. The command writes dicom-validation.md and dicom-validation.json there, and prints the Markdown.

  • It exits with status 0 when the comparison passed, 1 when it found unexplained findings or compared no released instance, and 2 when it could not compare, for example because a validator is missing or the release cannot be paired unambiguously: the mapping lacks a column or maps an input to two outputs, two outputs share a name, or an output has no input.

  • --workers sets how many processes run the per-file validators; it defaults to the number of CPUs.

  • Both result files hold only versions, counts, SOP Class names, attribute tags and keywords, and text that the validators’ own executables or tables hold, so they hold no value or path. The inputs, the release, and the mapping file must not be shared.

Read a report#

dicom-validation.md gives, in order:

  1. The verdict, passed or failed.

  2. The versions: the dicom3tools snapshot, dicom-validator’s version, the edition it read, and the SHA-256 of each DocBook file, so that another run can be shown to have used the same standard.

  3. The released instances compared, by SOP Class, and the inputs without an output of their own, such as withheld instances.

  4. For each validator, how many pairs it checked, and failed on, in inputs and outputs; the findings by severity in each; and how many input findings were resolved. A validator that cannot read a file is counted as failed, not as finding nothing: dciodvfy aborts on the corpus’s 32-bit RT Dose in Implicit VR, for example, so that pair is also left out of dcentvfy’s comparison, which stops at a file it cannot read, and counts only for dicom-validator.

  5. The introduced findings: unexplained first, then explained, each with the number of the known difference that explains it, then not comparable.

  6. The known differences, the engine ones first, with how many kinds of finding each explained and in how many files.

Read the unexplained findings first: each is a change that the de-identification made and that nothing accounts for. An engine entry is a known defect to fix, not an accepted behaviour.