Skip to content

Collect projio MCP tool usage stats per project

Motivation

As projio's MCP surface grows (~220 tools across subsystems), we have no visibility into which tools agents actually reach for in real use. This makes it hard to:

  • Prioritize which tools deserve polishing, skill docs, or renames
  • Detect dead tools that never get called (candidates for deprecation)
  • Spot naming confusion (tool called and then immediately followed by a different one — suggests the agent guessed wrong first time)
  • Drive which skills to write next (if pipeio_nb_snapshot is hot but has no skill, that's a signal)
  • Benchmark dispatch patterns across projects (pixecog vs sirocampus vs projio itself) to see whether agents use the same subset everywhere

Proposal

Instrument the projio MCP server to count every tool call, keyed per-project.

Where the counter lives: - Per-project: .projio/telemetry/mcp-usage.jsonl (append-only, one line per call) — stays with the project's git history via DataLad - Each line: {ts, tool, duration_ms, status, error_kind?} - No arguments, no results — just the name, status, and timing, so it's cheap and leaks no content

Where the hook lives: - src/projio/mcp/server.py — wrap the @server.tool decorator (or add a thin middleware) so every registered tool gets counted without touching individual tool functions - Gated by a config flag (telemetry.enabled in .projio/config.yml, default off); when off, zero overhead

Aggregation: - New MCP tool projio_telemetry_summary(window?) returns top-N tools, calls-per-day, error-rate, median latency - Or a CLI subcommand projio telemetry that prints the same

Concrete use cases

  • "Which pipeio tools did agents actually use in pixecog this week?" — directly answers the naming-transparency question I raised today
  • "Are pipeio_report and pipeio_nb_report confused at runtime?" — look for sessions where one is called, errors, then the other is called
  • "Is biblio_pdf_fetch_oa_status ever called without biblio_pdf_fetch_oa preceding it?" — spots orphan tools
  • Aggregate across lab projects via worklog to get a cross-project usage heatmap

Scope decisions (open)

  • Content-free: do NOT log arguments or results. Just name + meta.
  • Per-project, not central: keeps telemetry local, inspectable, and under DataLad version control. Cross-project aggregation is a separate tool that walks known projects.
  • Opt-in: default off. Enable per project via config.
  • Rotation: weekly or size-based rotation of mcp-usage.jsonl to avoid unbounded growth.

Relation to this conversation

Arose from the pipeio tool-naming audit: we identified ~10 tools with transparency problems (pipeio_report vs pipeio_nb_report, pipeio_completion, pipeio_cross_flow, etc.) but had no data to confirm which ones actually bite in practice. Usage stats would turn that audit from "my assessment" into evidence.