Skip to content

idea

Overview

FigureSpec: A Build Orchestrator for Scientific Figures

One-sentence summary

FigureSpec is a declarative orchestration layer that coordinates existing plotting libraries, layout engines, and build systems to produce reproducible, agent-operable scientific figures — without reimplementing any of them.

Problem

Scientific figures are reproducible at the panel level (Matplotlib, ggplot2, etc.) but manual at the composition level. The "last mile" — arranging panels, aligning labels, adding annotations, exporting to journal specs — is done in GUI editors or ad-hoc scripts. This makes figures non-reproducible, non-rebuildable, and opaque to automated agents.

The missing piece is not a better plotting library or a better layout solver. It is a coordination manifest that wires existing tools into a single, rebuildable pipeline.

Scope and non-goals

FigureSpec is an orchestration system, not a rendering engine.

In scope: - Declarative spec format (YAML) describing panels, layout intent, annotations, style, and data dependencies - Adapter contracts for panel backends (Matplotlib first) - SVG assembly from panel outputs with validated positioning - Build integration (Snakemake-compatible, simple fallback) - MCP server exposing figure operations to AI agents - Journal target profiles (column widths, font minimums, DPI)

Out of scope: - Building a new constraint solver (delegate to backend layout engines) - Reaching inside panels (axis configuration, tick formatting, colormaps) - Replacing existing plotting grammars - Image generation or learned rendering

Architecture

                    ┌──────────────┐
                    │  FigureSpec  │  ← YAML manifest (source of truth)
                    │   (YAML)     │
                    └──────┬───────┘
                           │
                    ┌──────▼───────┐
                    │  Orchestrator │  ← Python CLI + library
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │ Panel    │ │ Panel    │ │ Panel    │  ← Backend adapters
        │ Adapter  │ │ Adapter  │ │ Adapter  │     (Matplotlib, etc.)
        │ (mpl)    │ │ (vega)   │ │ (mermaid)│
        └────┬─────┘ └────┬─────┘ └────┬─────┘
             │             │             │
             ▼             ▼             ▼
          panel_a.svg   panel_b.svg   panel_c.svg
              │             │             │
              └─────────────┼─────────────┘
                            ▼
                    ┌───────────────┐
                    │ SVG Composer  │  ← Assembly, annotation, validation
                    └───────┬───────┘
                            ▼
                    ┌───────────────┐
                    │    Export     │  ← SVG → PDF/PNG, journal checks
                    └───────────────┘

1. FigureSpec manifest

The spec is an orchestration manifest, not a layout language. It declares:

  • Panels: each panel references a generator script, its backend, and its data inputs. Panels are opaque — FigureSpec doesn't look inside them.
  • Layout intent: grid structure plus a small set of named constraints (equal_height, align_ylabels, shared_colorbar). These are translated into backend-native calls, not solved by FigureSpec.
  • Annotations: typed (panel_label, callout, arrow, bracket), anchored to panel IDs.
  • Style tokens: font family, panel gap, line weights, color roles — applied at composition time.
  • Targets: journal profiles that set figure width, font minimums, DPI, colorblind checks.
figure:
  id: fig_sleep_psd
  target: nature_single_column  # sets width=89mm, font>=5pt, etc.

  panels:
    psd:
      generator: matplotlib
      source: panels/psd.py
      inputs: [data/session01_psd.parquet]
    topo:
      generator: matplotlib
      source: panels/topomap.py
      inputs: [data/session01_bandpower.nc]

  layout:
    type: grid
    rows: 1
    cols: 2
    constraints:
      - equal_height: [psd, topo]

  annotations:
    - type: panel_label
      target: psd
      text: A
    - type: panel_label
      target: topo
      text: B

  style:
    font_family: Helvetica
    panel_gap_mm: 3

2. Panel adapters

Each adapter's contract:

  • Input: panel script path + data inputs + style tokens
  • Output: SVG fragment + metadata envelope (bounding box, declared anchor points, semantic IDs for any elements the panel wants to expose)
  • Guarantee: adapter reports which constraints it can honor; orchestrator validates post-render

Phase 1: Matplotlib only. The adapter calls the script, captures SVG via savefig, extracts bounding box and axis positions from the figure object, and normalizes the SVG (deduplicate IDs, strip Matplotlib boilerplate, normalize viewBox).

Later adapters follow the same contract. Adding a backend means writing one adapter module, not changing the orchestrator.

3. SVG composer

This is the one piece of genuinely new engineering. It:

  • Ingests panel SVG fragments with their metadata envelopes
  • Positions panels according to layout + constraints (using metadata bounding boxes, not parsing SVG geometry)
  • Handles ID namespacing (prefix all IDs per panel to avoid collisions)
  • Attaches typed annotations (panel labels positioned relative to panel bounding boxes)
  • Applies style tokens (font injection, gap spacing)
  • Emits a single composed SVG with well-formed viewBox and namespace declarations

This is a focused module (~500-1000 LOC), not svgutils. It needs to handle: viewBox normalization, ID deduplication, unit consistency (mm throughout), <defs> merging, and clipping path management.

4. Build integration

Two modes:

  • Simple: figurespec build fig.yaml — sequential execution, file-level caching (hash inputs, skip unchanged panels).
  • DAG: emit Snakemake rules from the spec. Each panel and the composition step become Snakemake targets. Incremental rebuilds, parallel panel generation, integration with manuscript build.

The simple mode is the onboarding path. The Snakemake mode is the correct production path. FigureSpec doesn't implement dependency tracking — it generates rules for a system that already does.

5. Export and validation

  • Canonical output: composed SVG
  • Derived: PDF (via Inkscape or CairoSVG), PNG (rasterize at target DPI)
  • Validation checks against journal target profile: figure width, minimum font size, DPI, bounding box overflow, label collision detection

6. MCP server — the agent interface

FigureSpec exposes an MCP server so AI agents (Claude, etc.) can operate on figures through a structured protocol rather than generating code or editing files directly.

Why MCP and not just a Python API: - MCP is the emerging standard for tool integration with LLMs — adopted across Anthropic, OpenAI, Google - It provides typed tool definitions, structured inputs/outputs, and JSON-RPC transport - An MCP server makes FigureSpec accessible to any MCP-compatible agent without agent-specific integration code - It enforces the boundary: agents operate on spec structure, not on raw files or pixels

MCP tool surface:

Tools:
  figurespec/inspect
    → Returns: panel IDs, layout structure, current constraints,
      annotation list, resolved style tokens, last build status
    → Use: agent orients itself before making changes

  figurespec/build
    → Input: spec path (optional: panel subset to rebuild)
    → Returns: build result, validation warnings, output paths
    → Use: trigger rebuild after spec changes

  figurespec/validate
    → Input: spec path + target profile
    → Returns: pass/fail per check (font size, dimensions, overlap, etc.)
    → Use: pre-submission checks

  figurespec/edit_spec
    → Input: patch object (JSON merge-patch against current spec)
    → Returns: updated spec, diff summary
    → Use: agent modifies layout, swaps panels, changes annotations
    → Example patches:
        {"panels": {"psd": {"source": "panels/psd_v2.py"}}}
        {"layout": {"constraints": [{"equal_height": ["psd", "topo"]}]}}
        {"annotations": [{"type": "panel_label", "target": "psd", "text": "a"}]}

  figurespec/query_output
    → Input: query (e.g., "bounding_box of panel psd", "font sizes")
    → Returns: structured data from last build's metadata
    → Use: agent checks results without parsing SVG

Resources:
  figurespec://spec/{figure_id}
    → Current spec content

  figurespec://output/{figure_id}/svg
    → Last built SVG (as resource for agent inspection)

  figurespec://output/{figure_id}/metadata
    → Build metadata (panel positions, resolved constraints, warnings)

Agent workflow loop:

  1. inspect → understand current figure state
  2. edit_spec → apply a change (swap data source, adjust layout, fix annotation)
  3. build → recompile
  4. validate → check against journal target
  5. query_output → verify the change landed correctly
  6. Repeat or hand off to human review

The key property: every agent action goes through the spec. There is no "edit the SVG directly" or "call Matplotlib API" path. The spec is the single source of truth, and every change produces a full, deterministic rebuild. This makes agent edits auditable and reversible (diff the YAML).

Phase plan

Phase 1 — Buildable MVP: - FigureSpec YAML schema (JSON Schema for validation) - Matplotlib adapter with SVG normalization - Grid-based SVG composer (equal_height, align edges, fixed gaps) - Panel labels as typed annotations - figurespec build CLI with file-level caching - MCP server with inspect/build/edit_spec/validate tools - 3 journal target profiles (Nature single/double column, default)

Phase 2 — Practical completeness: - Additional adapters (Vega-Lite, Mermaid/Graphviz) - Richer annotations (callouts, arrows, brackets with semantic anchors) - Shared legends, shared colorbars across panels - Style token library (reusable across figures in a project) - Snakemake rule generation - Watch mode with live SVG preview

Phase 3 — Ecosystem integration: - Quarto/Pandoc integration (figures as compiled dependencies) - Richer constraint vocabulary (insets, overlays, aspect ratio locks) - MCP resource subscriptions (agent watches for upstream data changes) - Rubric-based validation (SciFig-style quality checks as postconditions)

What this is not

  • Not a plotting library. Panel internals are opaque.
  • Not a constraint solver. Layout delegates to backends or uses simple grid algebra.
  • Not an AI figure generator. Agents edit structured specs; they don't generate images.
  • Not a replacement for Inkscape. Final manual polish is fine — the point is that 90% of composition is automated and reproducible before that step.

Key risks and mitigations

Risk Mitigation
SVG assembly is finicky (IDs, namespaces, viewBox) Invest here — this is the core new engineering. Build a robust test suite against real Matplotlib/Vega SVG output.
Matplotlib SVG output is noisy and inconsistent Normalization pass in the adapter: strip boilerplate, deduplicate IDs, convert text-as-paths to actual text where possible.
Scope creep into panel internals Hard architectural boundary: panels are functions (inputs → SVG + metadata). The spec never configures axes, ticks, or colormaps.
Agent workflow is secondary to human workflow Design for human-authored YAML first. The MCP server is a thin layer over the same CLI operations. If the YAML workflow is good, the agent workflow follows.
Adoption friction (another tool to install) Pure Python package, pip install figurespec. Snakemake is optional. MCP server is optional. Minimum viable usage: write YAML, run figurespec build.