Skip to content

pipeio docs_collect does not collect overview.md or top-level architecture docs

Problem

pipeio docs_collect has two bugs:

  1. overview.md silently dropped. When a flow has code/pipelines/{flow}/docs/overview.md but no index.md, docs_collect generates a minimal stub index instead of using overview.md. When both exist, index.md wins and overview.md is never collected.

  2. Source-tree mutation. docs_collect writes a generated stub index.md back into code/pipelines/{flow}/docs/ when one doesn't exist. It should never write to the source tree — only to the output docs/pipelines/ directory.

Observed in pixecog

  • preprocess_ieeg: has only overview.md in source docs → collected as index.md. Works correctly.
  • preprocess_ecephys: had both index.md (stub) and overview.md (detailed) → stub won, detailed overview lost.
  • After deleting the ecephys index.md, docs_collect regenerated a new stub rather than falling back to overview.md.

Expected behavior

  1. If overview.md exists and index.md doesn't, use overview.md as the flow index content.
  2. If both exist, collect overview.md as a separate page alongside the index.
  3. Never write generated files to code/pipelines/{flow}/docs/ — only to docs/pipelines/.

Workaround

Don't create a separate index.md — put overview content in overview.md and let it serve as the index (works for preprocess_ieeg but fragile).

Source context: pixecog

PixEcog (pixecog): Neuropixels and ECoG dataset and analysis

Recent commits: 

8dc0d9d Pipeline docs: gitignore docs/pipelines/, relocate hand-authored files
96cd1ec Refactor sharpwaveripple/contracts: extract generic helpers to utils/io, remove pipelines __init__.py
36f9326 Add result note directory and sample note

README:

type: readme

Quick Start for Collaborators

Follow this checklist to get started with Pixecog documentation and workflows.

🐀 Pixecog Project — Compact Overview

Core principles

  • One immutable BIDS raw dataset (raw/) as the canonical baseline
  • Each analysis pipeline ha