Skip to content

pipeio_run_status / log_parse: conda run buffers stdout, logs empty during BIDS indexing

Summary

  • pipeio_run_status and pipeio_log_parse return empty logs for minutes during BIDS indexing because conda run buffers stdout.

Context

  • snakemake's BIDS index step (via snakebids generate_inputs) can take several minutes on large datasets
  • During this time, conda run holds all stdout in a buffer — nothing reaches the log file
  • Agent has no visibility into whether the run is progressing or stuck

Possible fixes

  • [ ] Use stdbuf -oL to force line-buffered stdout before the snakemake command
  • [ ] Use --show-failed-logs snakemake flag for better error visibility
  • [ ] Tail the snakemake .snakemake/log/ directory directly instead of relying on captured stdout
  • [ ] Consider conda run --no-capture-output if available