Skip to content

Revised: dispatch-validate-record as a single loop, not two ideas

Supersedes idea-arash-20260408-035007 (auto-dispatch) and idea-arash-20260408-035035 (auto-QC). Those were correctly critiqued as premature architecture — splitting a single loop into two halves, over-formalizing the easy parts, and hand-waving the hard parts.

Core insight

Dispatch, validation, and recording are one loop. Design them together:

questio_gap → identify milestone → run pipeline → validate → present to human → record

The unit of design is the full cycle, not the individual steps.

What to do first: one manual walkthrough

Pick preprocess_ieeg (most scaffolded flow). Manually execute the entire loop with existing tools:

  1. questio_gap("H1") → identifies ttl-removal-validated as root blocker
  2. pipeio_flow_status("preprocess_ieeg") → check readiness
  3. pipeio_run(targets=...) with dry-run first, human confirms before real run
  4. Write validate_ttl_removal.ipynb — the first concrete validation notebook
  5. pipeio_nb_exec(notebook="validate_ttl_removal") → produces figures + metrics
  6. Human reviews QC output, agent proposes result note
  7. questio-record creates result note, proposes milestone update, human confirms

Document what works, what breaks, where improvisation was needed. Then generalize.

Design decisions from the critique

1. Milestone → flow link must be structural, not NLU. milestones.yml already has a pipelines field with flow names. Use it directly — no parsing of free-text recommendations. questio_gap should return structured flow names alongside recommendations.

2. No binary pass/fail for QC. Use status: pass | warn | fail | investigate. Human review is the default before any milestone update. The agent proposes, the human confirms. Full autonomy is a dial to turn later, not a launch feature.

3. Thresholds live in qc_config.yml per flow. Separate from notebook code. Reviewable, adjustable, versionable. Example: 

# code/pipelines/preprocess_ieeg/qc_config.yml
ttl_removal:
  spectral_suppression_db: {threshold: 20, level: warn}
  harmonic_residual_ratio: {threshold: 0.05, level: fail}

4. QC collection is questio's domain, not pipeio's. pipeio runs notebooks. questio interprets QC output and feeds it into evidence. Tool: questio_qc_collect(flow) — runs validation notebook via pipeio, parses output, proposes result note. Not pipeio_validate.

5. Failure modes are the design, not edge cases. The loop needs explicit handling for: - retry — transient failure (disk full, OOM). Re-run. - investigate — unexpected output (empty derivatives, NaN metrics). Open notebook interactively. - escalate — needs human judgment (borderline QC, ambiguous results). Present summary + figures, ask human. - skip — blocked by external dependency (data not available, upstream incomplete). Note the block, move on.

6. Don't build the generic layer yet. Write validate_ttl_removal.ipynb first. Then validate_swr_detection.ipynb. After 2-3 concrete implementations, extract the common patterns into a generic QC schema. The schema should emerge from real notebooks, not precede them.

Standalone dispatch skill: not worth it

The dispatch logic (questio_gap → resolve flow → pipeio_run) is five tool calls in a straight line. No branching, no non-obvious decisions. An agent with good questio-session instructions does this naturally. Don't over-formalize it. Instead, strengthen the execute step within questio-session to include dry-run preview and confirmation.

The honest framing

These tools help an agent assist a researcher, not replace one. The validation step especially requires human judgment. Design for propose → review → confirm. The agent runs the pipeline, runs QC, presents a summary with figures, and proposes a milestone update. The human confirms.

Source context: pixecog

PixEcog (pixecog): Neuropixels and ECoG dataset and analysis

Recent commits: 

36f9326 Add result note directory and sample note
62841d9 Add questio YAML data model (questions.yml + milestones.yml)
9b2f6fa Scaffold ecephys TTL removal mod, flow overview + mod docs, demo notebook

README:

type: readme

Quick Start for Collaborators

Follow this checklist to get started with Pixecog documentation and workflows.

🐀 Pixecog Project — Compact Overview

Core principles

  • One immutable BIDS raw dataset (raw/) as the canonical baseline
  • Each analysis pipeline ha