pipeio_run_status / log_parse: conda run buffers stdout, logs empty during BIDS indexing
Summary
pipeio_run_status and pipeio_log_parse return empty logs for minutes during BIDS indexing because conda run buffers stdout.
Context
- snakemake's BIDS index step (via snakebids
generate_inputs) can take several minutes on large datasets
- During this time,
conda run holds all stdout in a buffer — nothing reaches the log file
- Agent has no visibility into whether the run is progressing or stuck
Possible fixes
- [ ] Use
stdbuf -oL to force line-buffered stdout before the snakemake command
- [ ] Use
--show-failed-logs snakemake flag for better error visibility
- [ ] Tail the snakemake
.snakemake/log/ directory directly instead of relying on captured stdout
- [ ] Consider
conda run --no-capture-output if available