34  Architecture

Plotje has a five-stage pipeline: a pose becomes a draft, then a plan, a membrane, and finally a plot. Each stage is produced from the previous one by a single step. The user-facing functions pj/draft, pj/plan, and pj/plot are literal compositions of those steps, with pj/options folded in to inject pose-level options. Because the API is built as a composition, you can inspect each intermediate value and test each transition on its own.

This chapter introduces the single-step transitions, walks a small example through every stage, shows the user-facing functions as compositions, and explains how composite poses traverse the same pipeline through internal shape dispatch.

(ns plotje-book.architecture
  (:require
   ;; Clojure string -- splitting a message into sentences
   [clojure.string :as str]
   ;; Kindly -- notebook rendering protocol
   [scicloj.kindly.v4.kind :as kind]
   ;; Rdatasets -- standard datasets
   [scicloj.metamorph.ml.rdatasets :as rdatasets]
   ;; Plotje -- composable plotting
   [scicloj.plotje.api :as pj]
   ;; Plotje internals -- read in "How Panels Share an Axis" and "Poses rewritten before drafting"
   [scicloj.plotje.impl.pose :as pose]
   [scicloj.plotje.impl.plan :as plan]))

Pipeline Overview

graph LR X["Raw data"] -->|pj/->pose| B["Pose"] B -->|pj/options pj/lay-* ...| B B -->|pj/pose->draft| D["Draft"] D -->|pj/draft->plan| P["Plan"] P -->|pj/plan->membrane| M["Membrane"] M -->|pj/membrane->plot| F["Plot"] style X fill:#eee,stroke-dasharray:3 3 style B fill:#d1c4e9 style D fill:#e8f5e9 style P fill:#fff3e0 style M fill:#e3f2fd style F fill:#fce4ec

Two terms used throughout: data space is values in their original units (centimeters, dollars, dates, species names); drawing space is the drawing-unit coordinates inside the output canvas. The plan stage holds geometry in data space; the membrane stage holds geometry in drawing space.

The five stages:

  • Pose – the composable specification you write. Built by pj/pose, pj/lay-*, pj/options, pj/facet, pj/arrange, pj/scale, and pj/coord. Lifted from raw data by pj/->pose, so a dataset can flow through the pipeline without an explicit constructor call. No computation has happened yet.

  • Draft – the pose flattened. A LeafDraft record holds :layers (a vector of one map per applicable layer with all scope merged in – :data, :x, :y, :mark, :stat, and aesthetic keys) and :opts (the pose-level options that flow into the plan stage). A composite pose, a leaf with a facet over columns, which is drawn as one cell per column, and a leaf whose facet is wrapped into rows with {:cols n} produce a CompositeDraft instead, carrying per-leaf drafts (:sub-drafts), the resolved composite-framing geometry (:chrome-spec), the layout map from leaf path to rect (:layout), and the composite’s overall dimensions (:width, :height). A leaf draft holds no geometry. A composite draft already holds where each cell sits on the canvas, in drawing units, because the sizes of the cells are decided before the cells are planned. Produced by pj/pose->draft.

  • Plan – fully resolved geometry in data space (domains, ticks, legends, computed shapes), and the layout of the canvas in drawing units. A Plan record (composite plots use CompositePlan) holding panels as plain maps, layers as PlanLayer records, and numeric arrays as dtype-next buffers. Produced by pj/draft->plan. No rendering primitives yet.

  • Membrane – a PlotjeMembrane record carrying positioned drawing primitives (Translate, WithColor, Path, Label, …) in drawing space, sized to the output canvas. The record itself implements the Membrane library’s UI protocols, so it composes with other Membrane elements. Produced by pj/plan->membrane. The Membranes chapter walks this stage in depth.

  • Plot – rendered output (SVG hiccup or BufferedImage). Produced by pj/membrane->plot, dispatching on a :format keyword.

The composition shortcuts pj/draft, pj/plan, pj/membrane, and pj/plot run the chain from a pose up through the named stage. They are introduced one section down.

Most users only interact with the pose stage and never need to think about the others. The stages below matter when you are debugging unexpected output, building a custom renderer, or extending the library.

Why these stages?

A simpler library could go from data to a rendered image in one function. Plotje splits the work into five stages so each stage addresses a distinct concern, and each boundary between stages has a specific purpose:

  • The pose is what the user specifies.

  • The draft is the same specification flattened, with scope merged in. This boundary lets the layer engine run on a uniform input regardless of how the pose was built (single layer, faceted leaf, composite tree).

  • The plan holds geometry in data space – domains, ticks, computed shapes – before any drawing. This boundary lets you inspect and validate plot structure with Malli, and it lets multiple renderers share the same computed plan. It also carries the dimensions the layout settled on, so where each panel will sit on the canvas is decided here: pj/frames reports it, and the membrane stage draws into it.

  • The membrane holds drawing primitives in drawing space. This boundary decouples “what to draw, where” from the output format, so SVG and raster renderers consume the same membrane tree.

  • The plot is the format-specific output: SVG hiccup, a BufferedImage, or any other format a backend supports.

Every intermediate value can be inspected with kind/pprint. The same pipeline can be extended by registering new methods at several stages – mark, stat, position adjustment, output format – without modifying the core. The scale types and the coordinate systems are fixed sets. The Extensibility chapter walks each extension point.

The Single-Step Transitions

Each transition is its own public function. Walk the example below to see what enters and what leaves at each step. The per-function reference (arities, arguments, return types) lives in the API Reference.

Step 1: pj/->pose

Lift raw data (or a pose) to a pose. Polymorphic on input: a dataset becomes a leaf pose with :data set; an existing pose flows through unchanged (idempotent). This is the structural lift that lets every downstream function accept either raw data or a pose. It stops there – it sets no mapping. The default mapping that makes a bare dataset render is filled in when the pose is drafted, and pj/infer-mapping writes it onto the pose ahead of that (see “Where Inference Happens” below).

The example traced through every stage is iris petal measurements, with a scatter and per-species regression line. Pose-level :x, :y, and :color mappings; two layers on top of that mapping.

(def trace-pose
  (-> (rdatasets/datasets-iris)
      (pj/pose :petal-length :petal-width {:color :species})
      pj/lay-point
      (pj/lay-smooth {:stat :linear-model})))

The pose auto-renders as the plot it specifies:

trace-pose
petal widthpetal lengthspeciessetosaversicolorvirginica12345670.00.51.01.52.02.5

The underlying value is a plain Clojure map – kind/pprint reveals the structure (without it, the auto-render would re-show the plot):

(kind/pprint trace-pose)
{:mapping {:color :species, :x :petal-length, :y :petal-width},
 :layers
 [{:layer-type :point} {:layer-type :smooth, :stat :linear-model}],
 :data
 https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv [150 6]:

| :rownames | :sepal-length | :sepal-width | :petal-length | :petal-width |  :species |
|----------:|--------------:|-------------:|--------------:|-------------:|-----------|
|         1 |           5.1 |          3.5 |           1.4 |          0.2 |    setosa |
|         2 |           4.9 |          3.0 |           1.4 |          0.2 |    setosa |
|         3 |           4.7 |          3.2 |           1.3 |          0.2 |    setosa |
|         4 |           4.6 |          3.1 |           1.5 |          0.2 |    setosa |
|         5 |           5.0 |          3.6 |           1.4 |          0.2 |    setosa |
|         6 |           5.4 |          3.9 |           1.7 |          0.4 |    setosa |
|         7 |           4.6 |          3.4 |           1.4 |          0.3 |    setosa |
|         8 |           5.0 |          3.4 |           1.5 |          0.2 |    setosa |
|         9 |           4.4 |          2.9 |           1.4 |          0.2 |    setosa |
|        10 |           4.9 |          3.1 |           1.5 |          0.1 |    setosa |
|       ... |           ... |          ... |           ... |          ... |       ... |
|       140 |           6.9 |          3.1 |           5.4 |          2.1 | virginica |
|       141 |           6.7 |          3.1 |           5.6 |          2.4 | virginica |
|       142 |           6.9 |          3.1 |           5.1 |          2.3 | virginica |
|       143 |           5.8 |          2.7 |           5.1 |          1.9 | virginica |
|       144 |           6.8 |          3.2 |           5.9 |          2.3 | virginica |
|       145 |           6.7 |          3.3 |           5.7 |          2.5 | virginica |
|       146 |           6.7 |          3.0 |           5.2 |          2.3 | virginica |
|       147 |           6.3 |          2.5 |           5.0 |          1.9 | virginica |
|       148 |           6.5 |          3.0 |           5.2 |          2.0 | virginica |
|       149 |           6.2 |          3.4 |           5.4 |          2.3 | virginica |
|       150 |           5.9 |          3.0 |           5.1 |          1.8 | virginica |
}

Step 2: pj/pose->draft

Flatten a pose into a draft. For a leaf, returns a LeafDraft record carrying the merged layer maps and the pose-level options. The pose-level mapping (:x :petal-length, :y :petal-width, :color :species) appears inside each of the two layer maps, alongside layer-specific keys (:mark, :stat). The layer engine downstream receives a uniform shape regardless of where each mapping was originally specified. Keys prefixed with double underscores (such as :__panel-idx) are internal markers; they pass through the plan stage and follow the Clojure “do not consume” convention.

A draft layer’s :data carries one column the pose was not given: :scicloj.plotje/row, the row of the pose’s dataset each row came from. A facet splits the data and a series pivots it, and either numbers the resulting rows from zero again, so the row number is attached before both. :brush true reads it, which is how highlighting a point in one panel highlights the same row in the others. Like the double-underscore keys, the column is Plotje’s: no mapping can name it, and no list of available columns shows it.

(def trace-draft
  (pj/pose->draft trace-pose))
(kind/pprint trace-draft)
{:layers
 [{:color :species,
   :x :petal-length,
   :y :petal-width,
   :mark :point,
   :stat :identity,
   :layer-type :point,
   :data
   https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv [150 7]:

| :rownames | :sepal-length | :sepal-width | :petal-length | :petal-width |  :species | :scicloj.plotje/row |
|----------:|--------------:|-------------:|--------------:|-------------:|-----------|--------------------:|
|         1 |           5.1 |          3.5 |           1.4 |          0.2 |    setosa |                   0 |
|         2 |           4.9 |          3.0 |           1.4 |          0.2 |    setosa |                   1 |
|         3 |           4.7 |          3.2 |           1.3 |          0.2 |    setosa |                   2 |
|         4 |           4.6 |          3.1 |           1.5 |          0.2 |    setosa |                   3 |
|         5 |           5.0 |          3.6 |           1.4 |          0.2 |    setosa |                   4 |
|         6 |           5.4 |          3.9 |           1.7 |          0.4 |    setosa |                   5 |
|         7 |           4.6 |          3.4 |           1.4 |          0.3 |    setosa |                   6 |
|         8 |           5.0 |          3.4 |           1.5 |          0.2 |    setosa |                   7 |
|         9 |           4.4 |          2.9 |           1.4 |          0.2 |    setosa |                   8 |
|        10 |           4.9 |          3.1 |           1.5 |          0.1 |    setosa |                   9 |
|       ... |           ... |          ... |           ... |          ... |       ... |                 ... |
|       140 |           6.9 |          3.1 |           5.4 |          2.1 | virginica |                 139 |
|       141 |           6.7 |          3.1 |           5.6 |          2.4 | virginica |                 140 |
|       142 |           6.9 |          3.1 |           5.1 |          2.3 | virginica |                 141 |
|       143 |           5.8 |          2.7 |           5.1 |          1.9 | virginica |                 142 |
|       144 |           6.8 |          3.2 |           5.9 |          2.3 | virginica |                 143 |
|       145 |           6.7 |          3.3 |           5.7 |          2.5 | virginica |                 144 |
|       146 |           6.7 |          3.0 |           5.2 |          2.3 | virginica |                 145 |
|       147 |           6.3 |          2.5 |           5.0 |          1.9 | virginica |                 146 |
|       148 |           6.5 |          3.0 |           5.2 |          2.0 | virginica |                 147 |
|       149 |           6.2 |          3.4 |           5.4 |          2.3 | virginica |                 148 |
|       150 |           5.9 |          3.0 |           5.1 |          1.8 | virginica |                 149 |
,
   :__panel-idx 0}
  {:color :species,
   :x :petal-length,
   :y :petal-width,
   :mark :line,
   :stat :linear-model,
   :layer-type :smooth,
   :data
   https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv [150 7]:

| :rownames | :sepal-length | :sepal-width | :petal-length | :petal-width |  :species | :scicloj.plotje/row |
|----------:|--------------:|-------------:|--------------:|-------------:|-----------|--------------------:|
|         1 |           5.1 |          3.5 |           1.4 |          0.2 |    setosa |                   0 |
|         2 |           4.9 |          3.0 |           1.4 |          0.2 |    setosa |                   1 |
|         3 |           4.7 |          3.2 |           1.3 |          0.2 |    setosa |                   2 |
|         4 |           4.6 |          3.1 |           1.5 |          0.2 |    setosa |                   3 |
|         5 |           5.0 |          3.6 |           1.4 |          0.2 |    setosa |                   4 |
|         6 |           5.4 |          3.9 |           1.7 |          0.4 |    setosa |                   5 |
|         7 |           4.6 |          3.4 |           1.4 |          0.3 |    setosa |                   6 |
|         8 |           5.0 |          3.4 |           1.5 |          0.2 |    setosa |                   7 |
|         9 |           4.4 |          2.9 |           1.4 |          0.2 |    setosa |                   8 |
|        10 |           4.9 |          3.1 |           1.5 |          0.1 |    setosa |                   9 |
|       ... |           ... |          ... |           ... |          ... |       ... |                 ... |
|       140 |           6.9 |          3.1 |           5.4 |          2.1 | virginica |                 139 |
|       141 |           6.7 |          3.1 |           5.6 |          2.4 | virginica |                 140 |
|       142 |           6.9 |          3.1 |           5.1 |          2.3 | virginica |                 141 |
|       143 |           5.8 |          2.7 |           5.1 |          1.9 | virginica |                 142 |
|       144 |           6.8 |          3.2 |           5.9 |          2.3 | virginica |                 143 |
|       145 |           6.7 |          3.3 |           5.7 |          2.5 | virginica |                 144 |
|       146 |           6.7 |          3.0 |           5.2 |          2.3 | virginica |                 145 |
|       147 |           6.3 |          2.5 |           5.0 |          1.9 | virginica |                 146 |
|       148 |           6.5 |          3.0 |           5.2 |          2.0 | virginica |                 147 |
|       149 |           6.2 |          3.4 |           5.4 |          2.3 | virginica |                 148 |
|       150 |           5.9 |          3.0 |           5.1 |          1.8 | virginica |                 149 |
,
   :__panel-idx 0}],
 :opts {}}

Step 3: pj/draft->plan

Resolve the draft into computed geometry. Reads :opts from the draft to apply title, dimensions, axis labels, and so on. The smooth layer’s :linear-model stat resolves into per-species line segments here; the point layer keeps the raw observations grouped by species. The legend gets one entry per species.

(def trace-plan
  (pj/draft->plan trace-draft))

The plan – a Plan record carrying panels, total dimensions, ticks, the legend spec, and per-layer geometry (groups of dtype-next buffers):

(kind/pprint trace-plan)
{:panels
 [{:coord :cartesian,
   :y-domain [-0.01999999999999999 2.62],
   :x-scale {:type :linear},
   :x-domain [0.705 7.195],
   :x-ticks
   {:values [1.0 2.0 3.0 4.0 5.0 6.0 7.0],
    :labels ["1" "2" "3" "4" "5" "6" "7"],
    :categorical? false},
   :col 0,
   :layers
   [{:mark :point,
     :style {:opacity 0.75, :radius 3.0},
     :alpha-scale nil,
     :y-domain [0.1 2.5],
     :x-domain [1.0 6.9],
     :size-scale nil,
     :fixed-tooltip nil,
     :alpha-drawn? nil,
     :groups
     [{:color
       [0.8941176470588236
        0.10196078431372549
        0.10980392156862745
        1.0],
       :xs #tech.v3.dataset.column<float64>[50]
:petal-length
[1.400, 1.400, 1.300, 1.500, 1.400, 1.700, 1.400, 1.500, 1.400, 1.500, 1.500, 1.600, 1.400, 1.100, 1.200, 1.500, 1.300, 1.400, 1.700, 1.500...],
       :ys #tech.v3.dataset.column<float64>[50]
:petal-width
[0.2000, 0.2000, 0.2000, 0.2000, 0.2000, 0.4000, 0.3000, 0.2000, 0.2000, 0.1000, 0.2000, 0.2000, 0.1000, 0.1000, 0.2000, 0.4000, 0.4000, 0.3000, 0.3000, 0.3000...],
       :label "setosa",
       :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19...],
       :group-key "setosa"}
      {:color
       [0.21568627450980393
        0.49411764705882355
        0.7215686274509804
        1.0],
       :xs #tech.v3.dataset.column<float64>[50]
:petal-length
[4.700, 4.500, 4.900, 4.000, 4.600, 4.500, 4.700, 3.300, 4.600, 3.900, 3.500, 4.200, 4.000, 4.700, 3.600, 4.400, 4.500, 4.100, 4.500, 3.900...],
       :ys #tech.v3.dataset.column<float64>[50]
:petal-width
[1.400, 1.500, 1.500, 1.300, 1.500, 1.300, 1.600, 1.000, 1.300, 1.400, 1.000, 1.500, 1.000, 1.400, 1.300, 1.400, 1.500, 1.000, 1.500, 1.100...],
       :label "versicolor",
       :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69...],
       :group-key "versicolor"}
      {:color
       [0.30196078431372547 0.6862745098039216 0.2901960784313726 1.0],
       :xs #tech.v3.dataset.column<float64>[50]
:petal-length
[6.000, 5.100, 5.900, 5.600, 5.800, 6.600, 4.500, 6.300, 5.800, 6.100, 5.100, 5.300, 5.500, 5.000, 5.100, 5.300, 5.500, 6.700, 6.900, 5.000...],
       :ys #tech.v3.dataset.column<float64>[50]
:petal-width
[2.500, 1.900, 2.100, 1.800, 2.200, 2.100, 1.700, 1.800, 1.800, 2.500, 2.000, 1.900, 2.100, 2.000, 2.400, 2.300, 1.800, 2.200, 2.300, 1.500...],
       :label "virginica",
       :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119...],
       :group-key "virginica"}],
     :size-drawn? nil}
    {:mark :line,
     :style {:stroke-width 2.5, :opacity 1.0},
     :groups
     [{:color
       [0.8941176470588236
        0.10196078431372549
        0.10980392156862745
        1.0],
       :label "setosa",
       :x1 1.0,
       :y1 0.15302476654486402,
       :x2 1.9,
       :y2 0.33414535119772626}
      {:color
       [0.21568627450980393
        0.49411764705882355
        0.7215686274509804
        1.0],
       :label "versicolor",
       :x1 3.0,
       :y1 0.9088724584103528,
       :x2 5.1,
       :y2 1.6040850277264331}
      {:color
       [0.30196078431372547 0.6862745098039216 0.2901960784313726 1.0],
       :label "virginica",
       :x1 4.5,
       :y1 1.8573676029159527,
       :x2 6.9,
       :y2 2.2420802958833654}],
     :y-domain [0.1 2.5],
     :x-domain [1.0 6.9]}],
   :y-scale {:type :linear},
   :y-ticks
   {:values [-0.0 0.5 1.0 1.5 2.0 2.5],
    :labels ["0.0" "0.5" "1.0" "1.5" "2.0" "2.5"],
    :categorical? false},
   :row 0}],
 :width 600,
 :height 400,
 :shape-legend nil,
 :caption nil,
 :total-width 600.0,
 :legend-position :right,
 :layout-type :single,
 :layout
 {:subtitle-pad 0,
  :legend-w 102,
  :caption-pad 0,
  :y-label-pad 42.5,
  :legend-h 0.0,
  :title-pad 0,
  :strip-h 0,
  :x-label-pad 38,
  :strip-w 0.0},
 :grid {:rows 1, :cols 1},
 :legend
 {:title :species,
  :entries
  [{:label "setosa",
    :color
    [0.8941176470588236 0.10196078431372549 0.10980392156862745 1.0]}
   {:label "versicolor",
    :color
    [0.21568627450980393 0.49411764705882355 0.7215686274509804 1.0]}
   {:label "virginica",
    :color
    [0.30196078431372547 0.6862745098039216 0.2901960784313726 1.0]}]},
 :panel-height 362.0,
 :title nil,
 :y-label "petal width",
 :alpha-legend nil,
 :x-label "petal length",
 :y-label-lines ["petal width"],
 :subtitle nil,
 :panel-width 455.5,
 :x-label-lines ["petal length"],
 :size-legend nil,
 :total-height 400.0,
 :tooltip false,
 :margin 10}

The plan validates against a Malli schema:

(pj/valid-plan? trace-plan)
true

Step 4: pj/plan->membrane

Convert the plan into a PlotjeMembrane – a tree of membrane drawing primitives positioned in drawing-space coordinates, wrapped in a record that itself implements the Membrane UI protocols (IOrigin, IBounds, IChildren).

(def trace-membrane (pj/plan->membrane trace-plan))

The membrane carries the drawables plus the canvas size and title. The record fields :width and :height give the canvas size, the size the plan settled on, and :plotje/title carries the title when one is set. This plot sets none:

(select-keys trace-membrane [:width :height :plotje/title])
{:width 600, :height 400}

The drawables sit in the record’s :drawables field, a vector of Membrane primitives. Their class names:

(->> trace-membrane
     :drawables
     (mapv #(.getSimpleName (class %))))
["Translate"
 "Translate"
 "Translate"
 "Translate"
 "Translate"
 "Translate"
 "Translate"
 "Translate"]

The dedicated Membranes chapter walks the record’s protocols, the namespaced-attribute convention, and how a PlotjeMembrane composes with hand-built Membrane elements.

Step 5: pj/membrane->plot

Convert the membrane into the rendered output for a chosen format. Dispatches on the format keyword. This is the only stage whose output is format-native rather than portable: every earlier stage is the same across backends, while here the membrane becomes SVG hiccup, a BufferedImage, or another format’s bytes. The membrane carries its plan-derived dimensions as record fields (read via (membrane.ui/width m)/(height m)), so pj/membrane->plot does not need them written again in its options:

(def trace-plot
  (pj/membrane->plot trace-membrane :svg {}))

The plot is SVG hiccup, a vector starting with :svg. pj/svg-summary counts what it draws – one panel, a point per row of the data, and a regression line per species:

(pj/svg-summary trace-plot)
{:sizes #{3.0},
 :bold-texts 0,
 :italic-texts 0,
 :dash-patterns #{},
 :texts
 ["petal width"
  "petal length"
  "species"
  "setosa"
  "versicolor"
  "virginica"
  "1"
  "2"
  "3"
  "4"
  "5"
  "6"
  "7"
  "0.0"
  "0.5"
  "1.0"
  "1.5"
  "2.0"
  "2.5"],
 :width 600,
 :lines 3,
 :colors #{"none" "rgb(228,26,28)" "rgb(55,126,184)" "rgb(77,175,74)"},
 :points 150,
 :label-boxes 0,
 :alphas #{0.75},
 :dashed-lines 0,
 :clips 1,
 :tiles 0,
 :polygons 0,
 :visible-tiles 0,
 :panels 1,
 :height 400,
 :shapes #{:polyline :rect}}

Rendered:

(kind/hiccup trace-plot)
petal widthpetal lengthspeciessetosaversicolorvirginica12345670.00.51.01.52.02.5

Two formats are built in. :svg gives SVG hiccup, which renders in browsers and in notebooks through Kindly and Clay, and is written to .svg files. :bufimg gives a Java2D BufferedImage through Membrane’s Java2D backend, and is used for .png files. A new format that consumes the membrane registers a defmethod on the render multimethod scicloj.plotje.impl.render/membrane->plot, which the public pj/membrane->plot dispatches through. A format that goes from a plan directly to a figure, skipping the membrane – for example a Plotly-spec target – registers a defmethod on scicloj.plotje.impl.render/plan->plot instead, and is called with pj/plan->plot; pj/plot always renders through the membrane. The Extensibility chapter shows a worked backend registration.

Pipeline Shortcuts: pj/pose, pj/draft, pj/plan, pj/membrane, pj/plot, pj/save

Each pipeline stage has a user-facing convenience that runs the chain from raw input up through that stage; pj/save is the file-output sibling that runs the chain all the way and writes the result to disk:

  • pj/pose – not a literal composition. On raw data its one-argument form is pj/->pose. With more arguments it reads column arguments ((pj/pose data :x :y)) and a mapping, and on an existing pose it merges them into that pose’s mapping. It infers nothing: the default mapping is filled in when the pose is drafted.

  • pj/draft – from raw input to a draft.

  • pj/plan – from raw input to a plan.

  • pj/membrane – from raw input to a membrane tree.

  • pj/plot – from raw input to a rendered figure.

  • pj/save – from raw input to a file on disk (renders, then writes).

pj/draft->membrane and pj/draft->plot start the same chain from a draft rather than from raw input.

Each shortcut starts from raw input and stops at a different stage along the same chain. The solid line is the stage-to-stage pipeline; each dashed arrow is one shortcut, running from raw input up to the stage it produces:

graph LR X["Raw data"] --> B["Pose"] --> D["Draft"] --> P["Plan"] --> M["Membrane"] --> F["Plot"] X -.->|pj/pose| B X -.->|pj/draft| D X -.->|pj/plan| P X -.->|pj/membrane| M X -.->|pj/plot| F style X fill:#eee,stroke-dasharray:3 3 style B fill:#d1c4e9 style D fill:#e8f5e9 style P fill:#fff3e0 style M fill:#e3f2fd style F fill:#fce4ec

The shortcuts from pj/draft on are near-literal compositions of these steps: each lifts raw input with pj/->pose, applies the shared pj/infer-mapping step (which gives a pose that names no :x or :y column the default mapping of its data, and changes nothing on a pose whose mapping names one), and runs the chain up to its stage. Their source shows the pipeline directly. Two things are simplified below for clarity: the pj/->pose caller-name argument – which only tunes error messages – is elided, and pj/pose is shown at its 1-arity (its other arities also read column arguments and a mapping).

Pseudocode:

(defn pose         ; 1-arity; the other arities also read
  ([x]             ; column arguments and a mapping
   (-> x
       ->pose)))

(defn draft
  ([x]
   (-> x
       ->pose
       infer-mapping
       pose->draft))
  ([x opts]
   (-> x
       ->pose
       (options opts)
       draft)))

(defn plan
  ([x]
   (-> x
       ->pose
       infer-mapping
       pose->draft
       draft->plan))
  ([x opts]
   (-> x
       ->pose
       (options opts)
       plan)))

(defn membrane
  ([x]
   (let [pose (-> x
                  ->pose
                  infer-mapping)
         opts (:opts pose {})]
     (-> pose
         pose->draft
         draft->plan
         (plan->membrane opts))))
  ([x opts]
   (-> x
       ->pose
       (options opts)
       membrane)))

(defn plot
  ([x]
   (let [pose (-> x
                  ->pose
                  infer-mapping)
         opts (:opts pose {})
         fmt  (or (:format opts) :svg)]
     (-> pose
         pose->draft
         draft->plan
         (plan->membrane opts)
         (membrane->plot fmt opts))))
  ([x opts]
   (-> x
       ->pose
       (options opts)
       plot)))

(defn save
  ([x path] (save x path nil))
  ([x path opts]
   (let [pose (-> x
                  ->pose
                  infer-mapping)
         pose (if opts (options pose opts) pose)
         fmt  (resolve-format pose path)]  ; :format opt, else path extension
     ;; render the plan to fmt bytes, write them to path,
     ;; and return the java.io.File written
     (write-file path (plan->plot (plan pose) fmt (:opts pose))))))

In plot, the let binds pose, opts, and fmt for use in the subsequent -> thread, which runs the four transitions (pose->draft, draft->plan, plan->membrane, membrane->plot) in order. The membrane carries the dimensions the plan settled on as record fields, and the title under :plotje/title, so membrane->plot reads them without the plan.

pj/save is the file-output sibling: it lifts and infers the same way, resolves the file format (from a :format option or the path extension), renders the plan to those bytes, and writes them to disk, returning the file it wrote as a java.io.File (str of it is the path). resolve-format and write-file above stand in for that format-resolution and file-writing detail.

The 2-arity of each function folds the options map into the pose using pj/options before recursing into the 1-arity.

pj/membrane is the analogous shortcut for the membrane stage, useful for a caller that needs a membrane tree without choosing an output format yet – for example a custom backend, or a target that Membrane supports but Plotje has not wired in yet.

Because the compositions call the steps, redefining a step with with-redefs takes effect in pj/draft, pj/plan, pj/membrane and pj/plot. pj/save calls the render multimethods without going through the public step functions, so a with-redefs of pj/plan->membrane does not reach it.

The composition holds at runtime:

(let [pose-with-opts (-> trace-pose
                         (pj/options {:title "Iris Petals"
                                      :x-label "Petal length"
                                      :width 700}))
      via-plan (pj/plan pose-with-opts)
      via-arrows (-> pose-with-opts
                     pj/->pose
                     pj/pose->draft
                     pj/draft->plan)]
  {:title-match (= (:title via-plan) (:title via-arrows))
   :x-label-match (= (:x-label via-plan) (:x-label via-arrows))
   :width-match (= (:width via-plan) (:width via-arrows))
   :title (:title via-plan)
   :x-label (:x-label via-plan)
   :width (:width via-plan)})
{:title-match true,
 :x-label-match true,
 :width-match true,
 :title "Iris Petals",
 :x-label "Petal length",
 :width 700}

Plot-level options (title, x-label, width, …) are stored on the pose’s :opts, copied into the LeafDraft’s :opts, and read by pj/draft->plan. Calling the steps directly, without the user-facing convenience, produces the identical plan.

The same property allows inspection at any stage: stop the chain before the next step. A thread that ends in pj/pose->draft returns the draft, and kind/pprint after it shows the draft; a thread that ends in pj/draft->plan returns the plan.

Where Inference Happens

Two of the steps also infer: they fill in choices the user did not specify. Inference is what lets a dataset alone – with no mapping, no layers, no options – produce a complete plot. Most one-line examples in this book rely on inference at one or both of these stages. The Inference Rules chapter is the authoritative reference for what each stage infers.

(pj/pose {:x [1 2 3 4 5]
          :y [2 4 3 5 4]
          :g [:a :a :b :b :b]})
yxgab123452345

Where each kind of inference lives:

  • Mapping inference fills in :x, :y and :color on data of one to three columns: one column to :x, two columns to :x and :y, and a third column to :color. It runs in pj/pose->draft, after a facet over columns is expanded into cells, on every leaf, cells of a composite included. It writes the default in two cases: where no mapping and no layer names an :x or a :y column, and where the first layer names no column, whatever later layers name (the example below this list shows a later layer naming the default’s columns). Where a layer names :y alone and its layer type draws from two columns, :x is filled in with the first column nothing names, so (pj/lay-point data {:y :b}) draws :b against :a. pj/infer-mapping is the same step as a public function: it writes the default onto the pose, where it can be read before drafting. pj/->pose, pj/pose and the pj/lay-* functions read no column of the data when they are called: a layer naming no column is given its columns when the pose is drafted, and a layer left without a column reports an error then.

  • Layer-type inference is written into the draft. A pose with no pj/lay-* call is given one layer, of the type its columns choose: a categorical x with a numerical y gives a boxplot, a temporal x with a numerical y gives a line, a numerical x and y give a scatter, and one numerical column gives a histogram. pose/elaborate makes the choice, so the draft layer already holds its :mark and :stat. pj/draft->plan chooses the stat of a layer whose type leaves it open – a bar with no :y column counts – and gives a draft layer written by hand with no :mark the type its columns choose.

  • Column-type inference lives in pj/draft->plan. :x-type/:y-type/:color-type default from the data (numerical / categorical / temporal); a user-supplied :x-type overrides the default.

  • Geometry inference lives in pj/draft->plan. Domains default to data ranges; ticks default to evenly-spaced values; legend entries are derived from the layers’ aesthetic mappings; the coordinate system defaults to :cartesian.

Mapping inference and the choice of a layer type run in pj/pose->draft; the rest runs in pj/draft->plan. Every inferred default has an explicit override (pj/scale, pj/coord, pj/options, mapping keys like :x-type), so the user can opt out of any single inference without losing the others.

Below, pj/pose->draft gives a pose built by pj/->pose the default mapping of its three columns, with no pj/infer-mapping call in between. The points keep the color of the default beside a smooth that names the default’s :x and :y columns:

(->> (-> {:a [1 2 3 4] :b [2 4 3 5] :c ["u" "u" "v" "v"]}
         pj/->pose
         pj/lay-point
         (pj/lay-smooth :a :b)
         pj/pose->draft)
     :layers
     (mapv #(select-keys % [:mark :x :y :color])))
[{:mark :point, :x :a, :y :b, :color :c}
 {:mark :line, :x :a, :y :b, :color :c}]

What elaboration writes

Before it flattens a pose, pj/pose->draft writes into the pose what the draft would otherwise have to choose, in this order: the rows the pose keeps (its :where); a series pivoted into a key column and a value column; the default mapping of data with one to three columns; one layer, where the pose names columns and has no layer; the panel’s columns, into each layer that reads them; a layer type for each layer left to inference; and the flip of a column named on :y alone. The function that does this is pose/elaborate. Panels shows which of the panel’s columns a layer reads, and Layer Types shows the flip.

Where the layers of a panel draw different columns and no layer is colored by a column, pj/pose->draft also gives each layer that specifies no :color a color and a legend entry naming its columns, as Panels shows.

Composite Poses

A composite pose – one with :poses inside – flows through the same steps. Each step dispatches internally on shape: a leaf pose produces a LeafDraft; a composite pose, a leaf with a facet over columns, and a leaf whose facet is wrapped into rows with {:cols n} produce a CompositeDraft. The user-facing pipeline is unchanged.

(def composite-pose
  (-> (rdatasets/datasets-iris)
      (pj/arrange [{:x :petal-length :y :petal-width}
                   {:x :sepal-length :y :sepal-width}])
      (pj/pose {:color :species})))
composite-pose
petal widthpetal length2460.00.51.01.52.02.5sepal widthsepal length682.02.53.03.54.04.5speciessetosaversicolorvirginica

pj/draft (the composition of ->pose, infer-mapping and pose->draft) returns a CompositeDraft – wrap in kind/pprint to inspect:

(-> composite-pose
    pj/draft
    kind/pprint)
{:width 600,
 :height 400,
 :sub-drafts
 [{:path [0 0],
   :rect [0.0 0.0 240.0 400.0],
   :draft
   [{:y :petal-width,
     :stat :identity,
     :color :species,
     :__panel-idx 0,
     :mark :point,
     :color-scale
     {:values
      {"setosa" "#E41A1CFF",
       "versicolor" "#377EB8FF",
       "virginica" "#4DAF4AFF"},
      :type :linear},
     :layer-type :point,
     :__scale
     {:color
      {:values
       {"setosa" "#E41A1CFF",
        "versicolor" "#377EB8FF",
        "virginica" "#4DAF4AFF"}}},
     :x :petal-length,
     :__color-order ["setosa" "versicolor" "virginica"],
     :data
     https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv [150 7]:

| :rownames | :sepal-length | :sepal-width | :petal-length | :petal-width |  :species | :scicloj.plotje/row |
|----------:|--------------:|-------------:|--------------:|-------------:|-----------|--------------------:|
|         1 |           5.1 |          3.5 |           1.4 |          0.2 |    setosa |                   0 |
|         2 |           4.9 |          3.0 |           1.4 |          0.2 |    setosa |                   1 |
|         3 |           4.7 |          3.2 |           1.3 |          0.2 |    setosa |                   2 |
|         4 |           4.6 |          3.1 |           1.5 |          0.2 |    setosa |                   3 |
|         5 |           5.0 |          3.6 |           1.4 |          0.2 |    setosa |                   4 |
|         6 |           5.4 |          3.9 |           1.7 |          0.4 |    setosa |                   5 |
|         7 |           4.6 |          3.4 |           1.4 |          0.3 |    setosa |                   6 |
|         8 |           5.0 |          3.4 |           1.5 |          0.2 |    setosa |                   7 |
|         9 |           4.4 |          2.9 |           1.4 |          0.2 |    setosa |                   8 |
|        10 |           4.9 |          3.1 |           1.5 |          0.1 |    setosa |                   9 |
|       ... |           ... |          ... |           ... |          ... |       ... |                 ... |
|       140 |           6.9 |          3.1 |           5.4 |          2.1 | virginica |                 139 |
|       141 |           6.7 |          3.1 |           5.6 |          2.4 | virginica |                 140 |
|       142 |           6.9 |          3.1 |           5.1 |          2.3 | virginica |                 141 |
|       143 |           5.8 |          2.7 |           5.1 |          1.9 | virginica |                 142 |
|       144 |           6.8 |          3.2 |           5.9 |          2.3 | virginica |                 143 |
|       145 |           6.7 |          3.3 |           5.7 |          2.5 | virginica |                 144 |
|       146 |           6.7 |          3.0 |           5.2 |          2.3 | virginica |                 145 |
|       147 |           6.3 |          2.5 |           5.0 |          1.9 | virginica |                 146 |
|       148 |           6.5 |          3.0 |           5.2 |          2.0 | virginica |                 147 |
|       149 |           6.2 |          3.4 |           5.4 |          2.3 | virginica |                 148 |
|       150 |           5.9 |          3.0 |           5.1 |          1.8 | virginica |                 149 |
}],
   :opts {:width 240, :height 400, :suppress-color-legend true}}
  {:path [0 1],
   :rect [240.0 0.0 240.0 400.0],
   :draft
   [{:y :sepal-width,
     :stat :identity,
     :color :species,
     :__panel-idx 0,
     :mark :point,
     :color-scale
     {:values
      {"setosa" "#E41A1CFF",
       "versicolor" "#377EB8FF",
       "virginica" "#4DAF4AFF"},
      :type :linear},
     :layer-type :point,
     :__scale
     {:color
      {:values
       {"setosa" "#E41A1CFF",
        "versicolor" "#377EB8FF",
        "virginica" "#4DAF4AFF"}}},
     :x :sepal-length,
     :__color-order ["setosa" "versicolor" "virginica"],
     :data
     https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv [150 7]:

| :rownames | :sepal-length | :sepal-width | :petal-length | :petal-width |  :species | :scicloj.plotje/row |
|----------:|--------------:|-------------:|--------------:|-------------:|-----------|--------------------:|
|         1 |           5.1 |          3.5 |           1.4 |          0.2 |    setosa |                   0 |
|         2 |           4.9 |          3.0 |           1.4 |          0.2 |    setosa |                   1 |
|         3 |           4.7 |          3.2 |           1.3 |          0.2 |    setosa |                   2 |
|         4 |           4.6 |          3.1 |           1.5 |          0.2 |    setosa |                   3 |
|         5 |           5.0 |          3.6 |           1.4 |          0.2 |    setosa |                   4 |
|         6 |           5.4 |          3.9 |           1.7 |          0.4 |    setosa |                   5 |
|         7 |           4.6 |          3.4 |           1.4 |          0.3 |    setosa |                   6 |
|         8 |           5.0 |          3.4 |           1.5 |          0.2 |    setosa |                   7 |
|         9 |           4.4 |          2.9 |           1.4 |          0.2 |    setosa |                   8 |
|        10 |           4.9 |          3.1 |           1.5 |          0.1 |    setosa |                   9 |
|       ... |           ... |          ... |           ... |          ... |       ... |                 ... |
|       140 |           6.9 |          3.1 |           5.4 |          2.1 | virginica |                 139 |
|       141 |           6.7 |          3.1 |           5.6 |          2.4 | virginica |                 140 |
|       142 |           6.9 |          3.1 |           5.1 |          2.3 | virginica |                 141 |
|       143 |           5.8 |          2.7 |           5.1 |          1.9 | virginica |                 142 |
|       144 |           6.8 |          3.2 |           5.9 |          2.3 | virginica |                 143 |
|       145 |           6.7 |          3.3 |           5.7 |          2.5 | virginica |                 144 |
|       146 |           6.7 |          3.0 |           5.2 |          2.3 | virginica |                 145 |
|       147 |           6.3 |          2.5 |           5.0 |          1.9 | virginica |                 146 |
|       148 |           6.5 |          3.0 |           5.2 |          2.0 | virginica |                 147 |
|       149 |           6.2 |          3.4 |           5.4 |          2.3 | virginica |                 148 |
|       150 |           5.9 |          3.0 |           5.1 |          1.8 | virginica |                 149 |
}],
   :opts {:width 240, :height 400, :suppress-color-legend true}}],
 :chrome-spec
 {:caption nil,
  :block-titles [],
  :y-ticks-inferred-off #{},
  :legend-w 120,
  :row-labels [],
  :layout {[0 0] [0.0 0.0 240.0 400.0], [0 1] [240.0 0.0 240.0 400.0]},
  :axis-title-axes #{},
  :shared-aesthetics #{:color},
  :matrix? false,
  :title nil,
  :n-cols 0,
  :shared-categories {:color ["setosa" "versicolor" "virginica"]},
  :subtitle nil,
  :n-rows 0,
  :col-labels [],
  :strip-h 0,
  :grid-rect [0.0 0.0 480.0 400.0],
  :shared? true,
  :strip-w 0,
  :title-band-h 0},
 :layout {[0 0] [0.0 0.0 240.0 400.0], [0 1] [240.0 0.0 240.0 400.0]}}

Each entry of :sub-drafts is a map. It holds the cell’s place on the canvas (:path in the grid, and :rect in drawing units), the options the arrangement decided for the cell under :opts – its size, and which legends, ticks and axis titles it suppresses – and the cell’s draft layers under :draft, the vector a LeafDraft holds as :layers. With the draft layers counted, so that the entry prints readably:

(-> composite-pose
    pj/draft
    :sub-drafts
    first
    (update :draft count))
{:path [0 0],
 :rect [0.0 0.0 240.0 400.0],
 :draft 1,
 :opts {:width 240, :height 400, :suppress-color-legend true}}

The draft layers of a cell include the composite’s own layers. A pj/lay-* call on a composite keeps its layer at the root, and pj/pose->draft draws that layer in each cell it fits, under the cell’s own layers, or in every cell where it fits none. The function that does this is pose/fit-layers; the rule is stated in Rule LP2 and Rule LP3 of Pose Rules. The arrangement above has no layers, so each cell’s one draft layer is the one inference chose for it.

pj/plan returns a CompositePlan:

(pj/plan composite-pose)
{:width 600,
 :height 400,
 :sub-plots
 [{:path [0 0],
   :rect [0.0 0.0 240.0 400.0],
   :opts
   {:width 240,
    :height 400,
    :suppress-color-legend true,
    :laid-out-height 400.0},
   :plan
   {:panels
    [{:coord :cartesian,
      :y-domain [-0.01999999999999999 2.62],
      :x-scale {:type :linear},
      :x-domain [0.705 7.195],
      :x-ticks
      {:values [2.0 4.0 6.0],
       :labels ["2" "4" "6"],
       :categorical? false},
      :col 0,
      :layers
      [{:mark :point,
        :style {:opacity 0.75, :radius 3.0},
        :alpha-scale nil,
        :y-domain [0.1 2.5],
        :x-domain [1.0 6.9],
        :size-scale nil,
        :fixed-tooltip nil,
        :alpha-drawn? nil,
        :groups
        [{:color
          [0.8941176470588236
           0.10196078431372549
           0.10980392156862745
           1.0],
          :xs #tech.v3.dataset.column<float64>[50]
:petal-length
[1.400, 1.400, 1.300, 1.500, 1.400, 1.700, 1.400, 1.500, 1.400, 1.500, 1.500, 1.600, 1.400, 1.100, 1.200, 1.500, 1.300, 1.400, 1.700, 1.500...],
          :ys #tech.v3.dataset.column<float64>[50]
:petal-width
[0.2000, 0.2000, 0.2000, 0.2000, 0.2000, 0.4000, 0.3000, 0.2000, 0.2000, 0.1000, 0.2000, 0.2000, 0.1000, 0.1000, 0.2000, 0.4000, 0.4000, 0.3000, 0.3000, 0.3000...],
          :label "setosa",
          :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19...],
          :group-key "setosa"}
         {:color
          [0.21568627450980393
           0.49411764705882355
           0.7215686274509804
           1.0],
          :xs #tech.v3.dataset.column<float64>[50]
:petal-length
[4.700, 4.500, 4.900, 4.000, 4.600, 4.500, 4.700, 3.300, 4.600, 3.900, 3.500, 4.200, 4.000, 4.700, 3.600, 4.400, 4.500, 4.100, 4.500, 3.900...],
          :ys #tech.v3.dataset.column<float64>[50]
:petal-width
[1.400, 1.500, 1.500, 1.300, 1.500, 1.300, 1.600, 1.000, 1.300, 1.400, 1.000, 1.500, 1.000, 1.400, 1.300, 1.400, 1.500, 1.000, 1.500, 1.100...],
          :label "versicolor",
          :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69...],
          :group-key "versicolor"}
         {:color
          [0.30196078431372547
           0.6862745098039216
           0.2901960784313726
           1.0],
          :xs #tech.v3.dataset.column<float64>[50]
:petal-length
[6.000, 5.100, 5.900, 5.600, 5.800, 6.600, 4.500, 6.300, 5.800, 6.100, 5.100, 5.300, 5.500, 5.000, 5.100, 5.300, 5.500, 6.700, 6.900, 5.000...],
          :ys #tech.v3.dataset.column<float64>[50]
:petal-width
[2.500, 1.900, 2.100, 1.800, 2.200, 2.100, 1.700, 1.800, 1.800, 2.500, 2.000, 1.900, 2.100, 2.000, 2.400, 2.300, 1.800, 2.200, 2.300, 1.500...],
          :label "virginica",
          :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119...],
          :group-key "virginica"}],
        :size-drawn? nil}],
      :y-scale {:type :linear},
      :y-ticks
      {:values [-0.0 0.5 1.0 1.5 2.0 2.5],
       :labels ["0.0" "0.5" "1.0" "1.5" "2.0" "2.5"],
       :categorical? false},
      :row 0}],
    :width 240,
    :height 400,
    :shape-legend nil,
    :caption nil,
    :total-width 240.0,
    :legend-position :none,
    :layout-type :single,
    :layout
    {:subtitle-pad 0,
     :legend-w 0,
     :caption-pad 0,
     :y-label-pad 42.5,
     :legend-h 0.0,
     :title-pad 0,
     :strip-h 0,
     :x-label-pad 38,
     :strip-w 0.0},
    :grid {:rows 1, :cols 1},
    :legend nil,
    :panel-height 362.0,
    :title nil,
    :y-label "petal width",
    :alpha-legend nil,
    :x-label "petal length",
    :y-label-lines ["petal width"],
    :subtitle nil,
    :panel-width 197.5,
    :x-label-lines ["petal length"],
    :size-legend nil,
    :total-height 400.0,
    :tooltip false,
    :margin 10}}
  {:path [0 1],
   :rect [240.0 0.0 240.0 400.0],
   :opts
   {:width 240,
    :height 400,
    :suppress-color-legend true,
    :laid-out-height 400.0},
   :plan
   {:panels
    [{:coord :cartesian,
      :y-domain [1.88 4.5200000000000005],
      :x-scale {:type :linear},
      :x-domain [4.12 8.08],
      :x-ticks
      {:values [6.0 8.0], :labels ["6" "8"], :categorical? false},
      :col 0,
      :layers
      [{:mark :point,
        :style {:opacity 0.75, :radius 3.0},
        :alpha-scale nil,
        :y-domain [2.0 4.4],
        :x-domain [4.3 7.9],
        :size-scale nil,
        :fixed-tooltip nil,
        :alpha-drawn? nil,
        :groups
        [{:color
          [0.8941176470588236
           0.10196078431372549
           0.10980392156862745
           1.0],
          :xs #tech.v3.dataset.column<float64>[50]
:sepal-length
[5.100, 4.900, 4.700, 4.600, 5.000, 5.400, 4.600, 5.000, 4.400, 4.900, 5.400, 4.800, 4.800, 4.300, 5.800, 5.700, 5.400, 5.100, 5.700, 5.100...],
          :ys #tech.v3.dataset.column<float64>[50]
:sepal-width
[3.500, 3.000, 3.200, 3.100, 3.600, 3.900, 3.400, 3.400, 2.900, 3.100, 3.700, 3.400, 3.000, 3.000, 4.000, 4.400, 3.900, 3.500, 3.800, 3.800...],
          :label "setosa",
          :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19...],
          :group-key "setosa"}
         {:color
          [0.21568627450980393
           0.49411764705882355
           0.7215686274509804
           1.0],
          :xs #tech.v3.dataset.column<float64>[50]
:sepal-length
[7.000, 6.400, 6.900, 5.500, 6.500, 5.700, 6.300, 4.900, 6.600, 5.200, 5.000, 5.900, 6.000, 6.100, 5.600, 6.700, 5.600, 5.800, 6.200, 5.600...],
          :ys #tech.v3.dataset.column<float64>[50]
:sepal-width
[3.200, 3.200, 3.100, 2.300, 2.800, 2.800, 3.300, 2.400, 2.900, 2.700, 2.000, 3.000, 2.200, 2.900, 2.900, 3.100, 3.000, 2.700, 2.200, 2.500...],
          :label "versicolor",
          :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69...],
          :group-key "versicolor"}
         {:color
          [0.30196078431372547
           0.6862745098039216
           0.2901960784313726
           1.0],
          :xs #tech.v3.dataset.column<float64>[50]
:sepal-length
[6.300, 5.800, 7.100, 6.300, 6.500, 7.600, 4.900, 7.300, 6.700, 7.200, 6.500, 6.400, 6.800, 5.700, 5.800, 6.400, 6.500, 7.700, 7.700, 6.000...],
          :ys #tech.v3.dataset.column<float64>[50]
:sepal-width
[3.300, 2.700, 3.000, 2.900, 3.000, 3.000, 2.500, 2.900, 2.500, 3.600, 3.200, 2.700, 3.000, 2.500, 2.800, 3.200, 3.000, 3.800, 2.600, 2.200...],
          :label "virginica",
          :row-indices #tech.v3.dataset.column<int64>[50]
:__row-idx
[100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119...],
          :group-key "virginica"}],
        :size-drawn? nil}],
      :y-scale {:type :linear},
      :y-ticks
      {:values [2.0 2.5 3.0 3.5 4.0 4.5],
       :labels ["2.0" "2.5" "3.0" "3.5" "4.0" "4.5"],
       :categorical? false},
      :row 0}],
    :width 240,
    :height 400,
    :shape-legend nil,
    :caption nil,
    :total-width 240.0,
    :legend-position :none,
    :layout-type :single,
    :layout
    {:subtitle-pad 0,
     :legend-w 0,
     :caption-pad 0,
     :y-label-pad 42.5,
     :legend-h 0.0,
     :title-pad 0,
     :strip-h 0,
     :x-label-pad 38,
     :strip-w 0.0},
    :grid {:rows 1, :cols 1},
    :legend nil,
    :panel-height 362.0,
    :title nil,
    :y-label "sepal width",
    :alpha-legend nil,
    :x-label "sepal length",
    :y-label-lines ["sepal width"],
    :subtitle nil,
    :panel-width 197.5,
    :x-label-lines ["sepal length"],
    :size-legend nil,
    :total-height 400.0,
    :tooltip false,
    :margin 10}}],
 :chrome
 {:caption nil,
  :block-titles [],
  :y-ticks-inferred-off #{},
  :legend-w 120,
  :row-labels [],
  :layout {[0 0] [0.0 0.0 240.0 400.0], [0 1] [240.0 0.0 240.0 400.0]},
  :axis-title-axes #{},
  :shared-aesthetics #{:color},
  :matrix? false,
  :title nil,
  :n-cols 0,
  :shared-categories {:color ["setosa" "versicolor" "virginica"]},
  :subtitle nil,
  :n-rows 0,
  :col-labels [],
  :strip-h 0,
  :grid-rect [0.0 0.0 480.0 400.0],
  :axis-titles nil,
  :strip-w 0,
  :shared-legend
  {:legend
   {:title :species,
    :entries
    [{:label "setosa",
      :color
      [0.8941176470588236 0.10196078431372549 0.10980392156862745 1.0]}
     {:label "versicolor",
      :color
      [0.21568627450980393 0.49411764705882355 0.7215686274509804 1.0]}
     {:label "virginica",
      :color
      [0.30196078431372547
       0.6862745098039216
       0.2901960784313726
       1.0]}]}},
  :title-band-h 0},
 :composite? true,
 :total-width 600,
 :total-height 400,
 :title nil}

Each entry of :sub-plots holds the same place and options, and the cell’s leaf Plan under :plan:

(-> composite-pose
    pj/plan
    :sub-plots
    first
    :plan
    pj/leaf-plan?)
true

pj/membrane returns a PlotjeMembrane whose :drawables carry one Translate per leaf plus composite framing (column strip labels, shared legend, title if any). Plan-derived width and height are record fields, and the title is :plotje/title.

(-> composite-pose
    pj/membrane
    (select-keys [:width :height :plotje/title]))
{:width 600, :height 400}
(->> composite-pose
     pj/membrane
     :drawables
     (mapv #(.getSimpleName (class %))))
["Translate"
 "Translate"
 "Translate"
 "Translate"
 "Translate"
 "Translate"]

pj/plot (the full pipeline) returns the SVG hiccup that composite-pose auto-renders to at the top of this section:

(-> composite-pose
    pj/plot
    pj/svg-summary)
{:sizes #{3.0},
 :bold-texts 0,
 :italic-texts 0,
 :dash-patterns #{},
 :texts
 ["petal width"
  "petal length"
  "2"
  "4"
  "6"
  "0.0"
  "0.5"
  "1.0"
  "1.5"
  "2.0"
  "2.5"
  "sepal width"
  "sepal length"
  "6"
  "8"
  "2.0"
  "2.5"
  "3.0"
  "3.5"
  "4.0"
  "4.5"
  "species"
  "setosa"
  "versicolor"
  "virginica"],
 :width 600,
 :lines 0,
 :colors #{"none" "rgb(228,26,28)" "rgb(55,126,184)" "rgb(77,175,74)"},
 :points 300,
 :label-boxes 0,
 :alphas #{0.75},
 :dashed-lines 0,
 :clips 2,
 :tiles 0,
 :polygons 0,
 :visible-tiles 0,
 :panels 2,
 :height 400,
 :shapes #{:rect}}

The composition holds for both leaf and composite poses – pj/plan runs the same steps either way, ->pose, infer-mapping, pose->draft and draft->plan, as in the pseudocode above – because each step dispatches on shape at the bottom of its call.

The composite path also performs work across leaves that a single leaf has no counterpart for: deciding which cells share an axis, and what that axis covers. The next section follows it through the stages.

How Panels Share an Axis

Panels that are meant to be compared are drawn on one axis: the same numbers, or the same categories, in every panel. Plotje draws several panels in two ways, and both follow one rule: a shared axis covers what the panels sharing it draw.

  • A facet by categories – the categories of one column in one row, or a grid crossing the categories of two columns under :col and :row – is one plan with several panels.

  • An arrangement, a facet over columns, and a facet wrapped into rows with {:cols n} are each a CompositePlan: one plan per cell.

The same two clinics drawn by each path:

(def north
  {:week ["w1" "w1" "w2" "w2"]
   :age ["adults" "children" "adults" "children"]
   :visits [30 20 25 15]})
(def south
  {:week ["w1" "w1" "w2" "w2"]
   :age ["adults" "children" "adults" "children"]
   :visits [60 45 50 40]})
(def arranged-clinics
  (pj/arrange [(pj/lay-bar north :week :visits {:color :age :position :stack})
               (pj/lay-bar south :week :visits {:color :age :position :stack})]
              {:share-scales #{:y}}))
arranged-clinics
w1w20102030405060708090100110w1w2weekvisitsageadultschildren
(def faceted-clinics
  (-> {:clinic ["north" "north" "north" "north" "south" "south" "south" "south"]
       :week ["w1" "w1" "w2" "w2" "w1" "w1" "w2" "w2"]
       :age ["adults" "children" "adults" "children"
             "adults" "children" "adults" "children"]
       :visits [30 20 25 15 60 45 50 40]}
      (pj/lay-bar :week :visits {:color :age :position :stack})
      (pj/facet :clinic)))
faceted-clinics
visitsweekageadultschildrenw1w20102030405060708090100110w1w2northsouth

The pose stage decides which cells share

pj/pose->draft reads the composite’s :share-scales and puts the cells into groups. It measures no extent to do this. It reads the type of each column, to report cells that cannot share, and for a column of categories the order the data holds them in, which the group carries as :order. Each sub-draft carries the groups of its cell under :share, by axis:

(->> arranged-clinics
     pj/draft
     :sub-drafts
     (mapv :share))
[{:y {:group 0}} {:y {:group 0}}]

Two cells are placed in one group where they draw the same thing along an axis: the values of one column, or what a stat computes from one column – a count, a density – which the stat declares (see What a stat computes). A count of a column and the values of that column are never in one group. Which axes are shared by default, and the other values :share-scales takes, are in Composition.

The plan stage decides what the axis covers

What a cell draws is known only after its stats have run. A stack of bars reaches the sum of its parts, which no row of the :visits column holds. So pj/draft->plan first measures every sharing cell: plan/draft->extents runs the cell’s stats and returns the extent the cell draws on each axis, before any padding.

(->> arranged-clinics
     pj/draft
     :sub-drafts
     (mapv (fn [sub-draft]
             (plan/draft->extents (:draft sub-draft) (:opts sub-draft)))))
[{:x ["w1" "w2"], :y [0.0 50.0]} {:x ["w1" "w2"], :y [0.0 105.0]}]

The extents of a group are pooled: numbers by the lowest low and the highest high, categories by listing every category of every cell. Each cell is then planned with the pooled extent in place of its own. The pooled extent is in the cell’s options in the plan:

(->> arranged-clinics
     pj/plan
     :sub-plots
     (mapv (fn [sub-plot] (:y-scale-domain (:opts sub-plot)))))
[[0.0 105.0] [0.0 105.0]]

Every cell pads that one extent the same way, so the panels end on one domain:

(->> arranged-clinics
     pj/plan
     :sub-plots
     (mapv (fn [sub-plot] (:y-domain (first (:panels (:plan sub-plot)))))))
[[0.0 110.25] [0.0 110.25]]

Two details complete the grouping. A cell whose layers draw two columns on an axis belongs to the groups of both columns, and the two groups become one: three cells sharing :y, drawing :adults, :children, and both, are one group. And a cell’s numeric domain is widened after planning so that its text marks are drawn in full; cells of a group that end on different domains are planned again with the widest domain as a floor, so that they agree.

A sharing cell’s stats run twice: once to measure it, and once to plan it with the pooled extent. A cell that shares with no other cell is not measured.

A facet in one plan follows the same steps inside pj/draft->plan: its panels are measured without padding, pooled, and each panel is then given the pooled extent and padded. The facet of the two clinics ends on the domain the arrangement ends on:

(->> faceted-clinics
     pj/plan
     :panels
     (mapv :y-domain))
[[0.0 110.25] [0.0 110.25]]

Poses rewritten before drafting

Some poses are rewritten into the cells they stand for before any of the above runs, so that one spelling cannot be drawn differently from another. pj/pose->draft does this in pose/expand-facet-nodes.

A facet over columns is kept on the pose as written, and expanded into a composite of one cell per column. A series faceted by the key column the series pivot adds draws the same panels as a facet over the columns of the series. pj/pose->draft first rewrites the series into that facet over columns, and then expands the facet like any other, here on weekly counts of visits:

(def weekly-visits
  {:week [1 2 3 4]
   :adults [30 25 28 35]
   :children [20 15 18 22]})
(def series-by-key
  (-> weekly-visits
      (pj/lay-line :week [:adults :children])
      (pj/facet :series)))
series-by-key
adults12342526272829303132333435children12341516171819202122week

The expanded pose is a composite. The column the layer drew along :x has moved to the root, and each cell names one of the series’ columns:

(-> series-by-key
    pose/expand-facet-nodes
    (dissoc :data)
    kind/pprint)
{:layers [{:layer-type :line, :mapping {}}],
 :mapping {:x :week},
 :layout {:direction :vertical},
 :poses
 [{:layout {:direction :horizontal},
   :poses
   [{:plotje/facet-cell true, :mapping {:y :adults}}
    {:plotje/facet-cell true, :mapping {:y :children}}]}]}

From here the pose is an ordinary composite: its cells are grouped, measured and planned as the two clinics above were. Cells that split the data of one pose share by default each axis on which they draw one column. In the series faceted by its key, each cell draws a column of its own on :y, so the two cells share :x and not :y:

(->> series-by-key
     pj/draft
     :sub-drafts
     (mapv :share))
[{:x {:group 0}} {:x {:group 0}}]

Cells written by hand, each keeping the rows of one value of a column with :where, that together keep every value of the column are rewritten the other way, into one pose faceted by that column (pose/collapse-regular-split).

Where a Mistake Is Reported

When a pose cannot be drawn, which step reports the error depends on what the error depends on. A call reports what its own arguments decide. pj/pose->draft reports what depends on which columns the data has, and on how the layers of a panel or the cells of a facet relate. pj/draft->plan reports what is known once a layer’s stat and mark meet the values of its columns – a bar given two categorical columns, a mark polar coordinates have no drawing for – and panels too small for the canvas. So pj/lay-point naming a column the data lacks returns a pose, and drafting that pose reports the column.

The table below builds a pose for each mistake and runs the steps in order. It shows the first step that reports an error, and the first sentence of the message:

(kind/table
 {:column-names ["Mistake" "Reported by" "First sentence of the message"]
  :row-vectors
  (for [[mistake build]
        [["A second column given to pj/lay-histogram"
          #(pj/lay-histogram weekly-visits :adults :children)]
         ["Pairs of columns given to pj/pose"
          #(pj/pose weekly-visits [[:week :adults] [:week :children]])]
         ["A key pj/facet's map does not accept"
          #(-> weekly-visits
               (pj/lay-point :week :adults)
               (pj/facet {:y [:adults :children] :z [:week]}))]
         ["A column the data lacks"
          #(pj/lay-point weekly-visits :week :teens)]
         ["A layer left without a column, on data of four columns"
          #(pj/lay-point (assoc weekly-visits :seniors [5 6 7 8]))]
         ["A category and a number on one axis"
          #(-> north
               (pj/lay-point :visits :week)
               (pj/lay-point :age :week))]
         ["A layer naming a column no cell has, on a facet over columns or an arrangement"
          #(-> weekly-visits
               (pj/pose {:x :week})
               (pj/lay-point {:y :week})
               (pj/facet {:y [:adults :children]}))]
         ["A bar of two categorical columns"
          #(pj/lay-bar north :week :age)]
         ["A mark polar coordinates cannot draw"
          #(-> weekly-visits
               (pj/lay-line :week :adults)
               (pj/coord :polar))]
         ["Panels too small for the canvas"
          #(-> north
               (pj/lay-point :week :visits)
               (pj/facet :age)
               (pj/options {:width 60}))]]]
    (let [report (fn [step f]
                   (try (f)
                        nil
                        (catch Exception e
                          [mistake step (first (str/split (ex-message e) #"(?<=\.) "))])))
          pose (try (build) (catch Exception _ nil))]
      (or (report "the call" build)
          (report "pj/pose->draft" #(pj/pose->draft pose))
          (report "pj/draft->plan" #(pj/draft->plan (pj/pose->draft pose))))))})
Mistake Reported by First sentence of the message
A second column given to pj/lay-histogram the call pj/lay-histogram draws from one column, and was given :adults on :x and :children on :y.
Pairs of columns given to pj/pose the call pj/pose was given pairs of columns, [[:week :adults] [:week :children]].
A key pj/facet's map does not accept the call pj/facet was given {:y [:adults :children], :z [:week]}, and the map has unexpected key(s): [:z].
A column the data lacks pj/pose->draft Column :teens (from :y) not found in dataset.
A layer left without a column, on data of four columns pj/pose->draft pj/lay-point draws from two columns, one on :x and one on :y, and this layer has no column on :x or :y.
A category and a number on one axis pj/pose->draft The layers of this pose share one panel, and on :x they draw columns of different kinds: pj/lay-point draws :visits (numerical), pj/lay-point draws :age (categorical).
A layer naming a column no cell has, on a facet over columns or an arrangement pj/pose->draft pj/lay-point names :week on :y, and no panel draws it: the panels draw :adults and :children on :y.
A bar of two categorical columns pj/draft->plan pj/lay-bar reads the length of a bar from a numerical column, but :week and :age are both categorical.
A mark polar coordinates cannot draw pj/draft->plan Mark :line is not supported with polar coordinates.
Panels too small for the canvas pj/draft->plan Each panel would be 11 wide, which is below the smallest panel size, 20.

Pipeline Summary

Stage Type Coordinates
Pose Plain map (leaf or composite) None (declarative)
Draft LeafDraft (:layers, :opts) or CompositeDraft (:sub-drafts, :chrome-spec, :layout, :width, :height) record None for a leaf; drawing units for where a composite’s cells sit (:layout)
Plan Plan or CompositePlan record (with PlanLayer records and dtype buffers) Data space (marks); drawing units (layout)
Membrane Record tree (membrane.ui primitives in a vector) Drawing units
Plot Hiccup vector (:svg) or BufferedImage (:bufimg) Drawing units

Dependencies

Plotje builds on several Clojure libraries:

What’s Next

  • Exploring Plans – a hands-on tour of the plan stage, building intuition for the data shape that the pipeline produces
  • Extensibility – add custom marks, statistics, position adjustments and output formats by extending the multimethods of the pipeline stages
source: notebooks/plotje_book/architecture.clj