19  Relationships

Scatter plots, regression, smoothing, density estimation, and heatmaps – revealing structure between two variables.

Scatter is the foundation. Each row becomes a point in the plane, and clusters or trends become visible. Regression and smoothing draw trend lines through it; 2D density and contours reveal where points are dense or sparse; the scatter plot matrix (SPLOM) at the end shows every pair of columns at once.

(ns plotje-book.relationships
  (:require
   ;; Kindly -- notebook rendering protocol
   [scicloj.kindly.v4.kind :as kind]
   ;; Rdatasets -- standard datasets
   [scicloj.metamorph.ml.rdatasets :as rdatasets]
   ;; Plotje -- composable plotting
   [scicloj.plotje.api :as pj]
   ;; Fastmath -- random number generation
   [fastmath.random :as rng]))

Basic Scatter

Sepal dimensions, no color – the default mark.

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width))
sepal widthsepal length4.55.05.56.06.57.07.58.02.02.53.03.54.04.5

Colored by Species

Adding :color :species groups points by species with distinct colors.

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width {:color :species}))
sepal widthsepal lengthspeciessetosaversicolorvirginica4.55.05.56.06.57.07.58.02.02.53.03.54.04.5

Petal Dimensions

Petal length against petal width – a strongly correlated pair, set up here as the running example for the regression sections below.

(-> (rdatasets/datasets-iris)
    (pj/lay-point :petal-length :petal-width {:color :species}))
petal widthpetal lengthspeciessetosaversicolorvirginica12345670.00.51.01.52.02.5

Linear Regression

A single regression line through all data.

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width)
    (pj/lay-smooth {:stat :linear-model}))
sepal widthsepal length4.55.05.56.06.57.07.58.02.02.53.03.54.04.5

Per-Group Regression

Fit a regression line per group.

(-> (rdatasets/datasets-iris)
    (pj/pose :petal-length :petal-width {:color :species})
    pj/lay-point
    (pj/lay-smooth {:stat :linear-model}))
petal widthpetal lengthspeciessetosaversicolorvirginica12345670.00.51.01.52.02.5

Regression with Confidence Ribbon

Pass {:confidence-band true} to show a 95% confidence band around the line.

(-> (rdatasets/datasets-iris)
    (pj/pose :sepal-length :sepal-width {:color :species})
    pj/lay-point
    (pj/lay-smooth {:stat :linear-model :confidence-band true}))
sepal widthsepal lengthspeciessetosaversicolorvirginica4.55.05.56.06.57.07.58.02.02.53.03.54.04.5

Pass :level to widen or narrow the band. A 99% interval covers more of the regression’s uncertainty than the default 95%; an 80% interval covers less:

(-> (rdatasets/datasets-iris)
    (pj/pose :sepal-length :sepal-width)
    pj/lay-point
    (pj/lay-smooth {:stat :linear-model :confidence-band true :level 0.80}))
sepal widthsepal length4.55.05.56.06.57.07.58.02.02.53.03.54.04.5
(-> (rdatasets/datasets-iris)
    (pj/pose :sepal-length :sepal-width)
    pj/lay-point
    (pj/lay-smooth {:stat :linear-model :confidence-band true :level 0.99}))
sepal widthsepal length4.55.05.56.06.57.07.58.02.02.53.03.54.04.5

Tips with Regression

Do smokers and non-smokers tip differently?

(-> (rdatasets/reshape2-tips)
    (pj/pose :total-bill :tip {:color :smoker})
    pj/lay-point
    (pj/lay-smooth {:stat :linear-model}))
tiptotal billsmokerNoYes102030405012345678910

LOESS Smoothing

A smooth curve through noisy data.

(-> (let [r (rng/rng :jdk 42)
          xs (vec (range 50))]
      {:x xs
       :y (mapv #(+ (Math/sin (* % 0.2))
                    (* 0.3 (- (rng/drandom r) 0.5)))
                xs)})
    (pj/lay-point :x :y)
    (pj/lay-smooth {:bandwidth 0.2}))
yx05101520253035404550-1.2-1.0-0.8-0.6-0.4-0.20.00.20.40.60.81.0

Heatmap (Auto-Binned)

Bin x and y into a grid, count points per cell.

(-> (rdatasets/datasets-iris)
    (pj/lay-tile :sepal-length :sepal-width))
sepal widthsepal lengthcount024684.55.05.56.06.57.07.58.02.02.53.03.54.04.5

Heatmap (Pre-Computed)

Use a numeric column for tile color. On a numeric axis, the axis reaches the outer edges of the tiles, so the first and last rows and columns are drawn at full size.

(def grid-data
  (let [r (rng/rng :jdk 99)]
    {:x (for [i (range 5) _j (range 5)] i)
     :y (for [_i (range 5) j (range 5)] j)
     :value (repeatedly 25 #(rng/irandom r 100))}))
(-> grid-data
    (pj/lay-tile :x :y {:fill :value}))
yxvalue0204060800123401234

Density 2D

KDE-smoothed 2D density heatmap.

(-> (rdatasets/datasets-iris)
    (pj/lay-density-2d :sepal-length :sepal-width))
sepal widthsepal lengthrelative density051015202534567891.52.02.53.03.54.04.55.0

Density 2D with Points

Scatter points drawn on top of the density heatmap.

(-> (rdatasets/datasets-iris)
    (pj/lay-density-2d :sepal-length :sepal-width)
    (pj/lay-point {:alpha 0.5}))
sepal widthsepal lengthrelative density051015202534567891.52.02.53.03.54.04.55.0

Contour Lines

Iso-density contour lines from 2D KDE.

(-> (rdatasets/datasets-iris)
    (pj/lay-contour :sepal-length :sepal-width))
sepal widthsepal lengthrelative density051015202534567891.52.02.53.03.54.04.55.0

Contour with Points

Contour lines drawn over scatter points.

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width {:alpha 0.3})
    (pj/lay-contour {:levels 8}))
sepal widthsepal lengthrelative density051015202534567891.52.02.53.03.54.04.55.0

Scatter with a Marginal Distribution

A scatter shows how two columns move together and says little about how either is spread on its own. pj/marginal puts that spread back: a distribution of one of the pose’s columns, drawn in a thin panel against one edge of the scatter and sharing the scatter’s axis for that column. :top describes the :x column.

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width)
    (pj/marginal :top))
density0.00.20.4sepal width4.55.05.56.06.57.07.58.02.02.53.03.54.04.5sepal length

The marginal panel keeps its own value scale, its duplicate ticks and axis title are dropped, and the two drawing areas are aligned so a value sits at the same place in both. :histogram draws bars instead of a curve, and :size sets the marginal panel’s share of the height.

:right describes the :y column instead, in a marginal panel beside the scatter. The distribution is drawn under pj/coord :flip, so it runs up the panel with its baseline against the scatter:

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width)
    (pj/marginal :right :histogram))
sepal length4.55.05.56.06.57.07.58.02.02.53.03.54.04.5count02040sepal width

A pose takes one marginal: pj/marginal returns a composite, and a composite has no single panel for a second marginal panel to sit against. For a distribution on both edges, pj/matrix with nil in each list of columns draws the scatter and both marginals. A nil leaves a place with no column on that axis, so that panel draws the distribution of the other column, and the corner, nil on both axes, is left empty. The marginals are histograms, where pj/marginal draws a density by default, and the weights draw them a third the size of the scatter:

(pj/matrix (rdatasets/datasets-iris)
           [:sepal-length nil]
           [nil :sepal-width]
           {:col-weights [3 1] :row-weights [1 3]})
count020sepal widthsepal length4.55.05.56.06.57.07.58.02.02.53.03.54.04.5count02040

A :color mapping does not reach the marginal panel that pj/marginal adds – the marginal panel describes the column as a whole. Faceting does reach the marginal panel, giving one distribution per panel, so facet first and add the marginal after:

(-> (rdatasets/datasets-iris)
    (pj/lay-point :sepal-length :sepal-width)
    (pj/facet :species)
    (pj/marginal :top)
    (pj/options {:width 900 :height 480}))
density0.00.51.0setosaversicolorvirginicasepal width56782.02.53.03.54.04.556785678sepal length

:col is a mapping, and the marginal carries it across to the distribution it builds, so both rows are divided by species. The strip label is drawn once, at the top of each column, above the marginal.

Scatter Plot Matrix (SPLOM)

pj/matrix given a list of columns draws a grid with a panel for every pair of those columns – a scatter plot matrix. With no layer written, each panel draws the layer type inferred from its columns: a histogram where a column meets itself, on the diagonal, and a scatter elsewhere.

Start small: two variables crossed with themselves give a 2x2 grid.

(def small-cols [:sepal-length :petal-length])
(-> (rdatasets/datasets-iris)
    (pj/pose {:color :species})
    (pj/matrix small-cols))
567868246246sepal lengthpetal lengthsepal lengthpetal lengthspeciessetosaversicolorvirginica

pj/matrix writes a facet over columns: the columns across under :col and the columns down under :row, so (pj/matrix data cols) is (pj/facet data {:col {:x cols} :row {:y cols}}), as Faceting shows.

The full 4x4 SPLOM follows the same pattern with iris’s four numeric columns:

(def cols [:sepal-length :sepal-width :petal-length :petal-width])
(-> (rdatasets/datasets-iris)
    (pj/pose {:color :species})
    (pj/matrix cols))
6823424668012234246012sepal lengthsepal widthpetal lengthpetal widthsepal lengthsepal widthpetal lengthpetal widthspeciessetosaversicolorvirginica

Inference picks the layer type for each panel: diagonal panels, where a column meets itself, get histograms; panels off the diagonal get scatter plots. All panels share the color aesthetic set on the pose.

A layer written on the matrix is drawn in each panel that fits it: a panel fits a layer when the panel’s columns, together with any the layer names, are a number of columns the layer type draws from (Rule LP2 in Pose Rules). A scatter draws two columns, so pj/lay-point is drawn in the panels off the diagonal, and each diagonal panel, which names one column, keeps the histogram that inference gives it:

(-> (rdatasets/datasets-iris)
    (pj/pose {:color :species})
    (pj/matrix small-cols)
    pj/lay-point)
567868246246sepal lengthpetal lengthsepal lengthpetal lengthspeciessetosaversicolorvirginica

A density draws one column, so pj/lay-density is drawn on the diagonal, and the diagonal shows densities in place of histograms:

(-> (rdatasets/datasets-iris)
    (pj/pose {:color :species})
    (pj/matrix small-cols)
    pj/lay-point
    pj/lay-density)
567868246246sepal lengthpetal lengthsepal lengthpetal lengthspeciessetosaversicolorvirginica

A layer that names its columns is drawn only in the panel of those columns. Here a trend line is added to one panel of the matrix:

(-> (rdatasets/datasets-iris)
    (pj/pose {:color :species})
    (pj/matrix small-cols)
    pj/lay-point
    pj/lay-density
    (pj/lay-smooth (first small-cols) (second small-cols) {:color "black"}))
567868246246sepal lengthpetal lengthsepal lengthpetal lengthspeciessetosaversicolorvirginica

A matrix of two different lists of columns has no diagonal, so every panel names two columns. A histogram draws from one column, so it fits none of the panels, and a layer that fits no panel is drawn in every panel, as written, as it would be on a single pose (Rule LP3 in Pose Rules). Here every panel draws a histogram of its :x column in place of the scatter that inference would give it:

(-> (rdatasets/datasets-iris)
    (pj/matrix [:sepal-length :sepal-width] [:petal-length :petal-width])
    pj/lay-histogram)
010200102030405056780102023401020304050sepal lengthsepal widthpetal lengthpetal width

See the Gallery chapter for more SPLOM variations, and the Interactivity chapter for brush selection.

See Also

  • Composition – composite poses, the marginal among them, and shared scales
  • Distributions – one-variable shape and spread

What’s Next

  • Cookbook – recipes for common multi-layer plots
  • Faceting – divide a chart into panels by category, or give each of several columns a panel of its own
  • Polar Coordinates – radial charts and pie-style visualizations
  • Customization – mark styling, palettes, and themes
source: notebooks/plotje_book/relationships.clj