Superdataset and subdatasets¶

Status: draft

Sources & anchors

Stack component: DataLad
Canonical artifact: gecog/.gitmodules (survey component 2)
Workshop session: Day-1 AM session 2 (DataLad)
Outline: _outline.md §B
Port source: teaching/NeuroPySeminar/docs/slides/bootcamp-ii/tutorial-plan.md (step 1) (NeuroPySeminar — needs refresh for current stack)

Frame¶

DataLad builds on git by adding content-addressed storage for large files and a nesting model for repositories. Understanding this two-layer design — git for metadata, git-annex for content, datasets for composition — is the prerequisite for everything else in the DataLad chapters.

git + git-annex: the two-layer design¶

DataLad layers one idea on top of git: content-addressable storage for files that should not live inside git's object database. Every binary, every multi-gigabyte EEG recording, every intermediate file that would bloat a repository into unusability stays in git-annex — a companion store managed alongside the git object store. git itself tracks the metadata: filenames, directory layout, checksums, and commit history. git-annex tracks the content, either locally or in a configured "special remote." The two stores move together; datalad save updates both atomically.

From the user's perspective the split is invisible most of the time. A datalad save -m "preprocess: add notch filter" commits scripts and config to git and adds large output files to the annex in one command. What makes the split visible is a freshly cloned repository: the directory tree is there, git history is there, but the annexed files are hollow symlinks pointing at a content hash that has not been fetched yet. Running datalad get raw/sub-01/ses-pre/ieeg/ pulls those files on demand from whichever sibling has them. This thin clone behavior is why DataLad is practical for hundreds-of-gigabytes datasets — you fetch exactly what the next step needs, nothing more.

Datasets, superdatasets, and subdatasets¶

The second idea DataLad adds is nested repositories. A dataset is a directory that is simultaneously a git repository and a DataLad-managed annex. A superdataset mounts other datasets at specific sub-paths, recorded in .gitmodules. The mounted datasets are subdatasets — they have independent git histories, independent annex stores, and their own siblings, but from the superdataset's perspective they look like a normal directory.

The superdataset records not just which subdatasets are mounted, but which commit of each subdataset is current. That commit SHA is the lock: changing it is an explicit action (datalad update code/lib/cogpy + datalad save) that produces a new superdataset commit. You do not get inadvertent upstream changes from a library someone else updated.

Reading `gecog/.gitmodules`¶

gecog's .gitmodules is the canonical teaching artifact (survey component 2 — nine entries, single RIA store, no rename aliasing). It shows the three-tier structure in its simplest form:

[submodule "raw"]
    path = raw
    url = /storage2/ria-store/alias/ds-gecog
    datalad-url = "ria+file:///storage2/ria-store/#~ds-gecog"

[submodule "code/lib/cogpy"]
    path = code/lib/cogpy
    url = /storage/share/git/ria-store/alias/cogpy
    datalad-url = "ria+file:///storage/share/git/ria-store#~cogpy"

[submodule "code/lib/labpy"]
    path = code/lib/labpy
    url = /storage/share/git/ria-store/alias/labpy
    datalad-url = "ria+file:///storage/share/git/ria-store#~labpy"

[submodule "derivatives/fullband_imported"]
    path = derivatives/fullband_imported
    url = /storage2/ria-store/alias/gecog-fullband_imported
    datalad-url = "ria+file:///storage2/ria-store#~gecog-fullband_imported"

[submodule "derivatives/spectrogram"]
    path = derivatives/spectrogram
    url = /storage2/ria-store/alias/gecog-spectrogram
    datalad-url = "ria+file:///storage2/ria-store#~gecog-spectrogram"

[submodule "derivatives/factor_analysis"]
    path = derivatives/factor_analysis
    url = /storage2/ria-store/alias/gecog-factor_analysis
    datalad-url = "ria+file:///storage2/ria-store#~gecog-factor_analysis"

[submodule "derivatives/preprocess_ieeg"]
    path = derivatives/preprocess_ieeg
    url = /storage2/ria-store/alias/gecog-preprocess_ieeg
    datalad-url = "ria+file:///storage2/ria-store#~gecog-preprocess_ieeg"

[submodule "derivatives/brainstate"]
    path = derivatives/brainstate
    url = /storage2/ria-store/alias/gecog-brainstate
    datalad-url = "ria+file:///storage2/ria-store#~gecog-brainstate"

Three tiers appear immediately: raw data (raw/), shared code libraries (code/lib/cogpy, code/lib/labpy), and per-flow derivatives (derivatives/*). Every URL is a RIA alias — no GitHub, no HTTP, just a content-addressed local path. The clean uniformity (study-specific subdatasets under ria-store/alias/gecog-*, shared libraries under storage/share/git/ria-store) is a deliberate design decision, not a default.

The datalad-url field uses the ria+file:// protocol with a #~alias fragment. This is how DataLad addresses a named alias inside a RIA store — #~cogpy resolves to the cogpy dataset regardless of where the store moves on disk, as long as the alias entry exists inside the store.

The subdataset-per-derivative convention¶

The convention across this stack (gecog and pixecog follow it fully; msol is mid-adoption) is: a mature Snakemake flow's derivative output can live in its own subdataset. This is an option, not a requirement. When pipeio_flow_new scaffolds derivatives/<flow>/, it stays a plain directory by default; registering it as a DataLad subdataset is worthwhile once the outputs stabilize and are worth versioning or sharing independently. Work-in-progress derivatives that get regenerated on every iteration are usually more hassle than benefit as subdatasets, so promote them later rather than up front. Where the convention is applied, the rationale is: reproducibility requires stating "this brainstate result was produced by this version of the preprocess_ieeg pipeline at this commit of cogpy." A flat git repository cannot make that statement in an auditable form — only the subdataset graph can.

The honest gap: this convention is socially enforced, not automatic. msol has three Snakemake flows and zero derivative subdatasets registered in .gitmodules. The workshop and handbook should teach the convention as a deliberate choice the user makes at flow-creation time, not as something projio enforces. See honest gaps for the fuller treatment.

When to create a new subdataset¶

Two clear triggers:

Large binary content — files that would bloat the superdataset's git history if committed directly. Raw EEG recordings, model checkpoints, large intermediate outputs. If it's bigger than a few megabytes and produced by a pipeline, it belongs in the annex, and probably in its own subdataset.
Independent version history — the directory's history diverges from the superdataset's own timeline. Code libraries evolve independently; derivative flows produce results on their own cadence. Both warrant a subdataset so their history can be pinned, updated, and rolled back independently.

What does not need a subdataset: a scripts/ directory under a flow (small files, tightly coupled to the flow), a configs/ directory (YAML, no large content), or a docs/ directory. These can live directly in the superdataset.

One trigger that looks obvious but is often wrong: "this is a separate concern." Code that is reused across projects belongs in a shared library subdataset mounted via a RIA alias — not as a copy in each project. The code/lib/cogpy and code/lib/labpy patterns in gecog illustrate this: one store, many consuming superdatasets, each at a pinned commit.