12 Visualizing correlation matrices (experimental 🛠) - DRAFT
This tutorial explores various ways to visualize a correlation matrix as a heatmap. It is inspired by the discussion at the Clojurians Zulip chat: #data-science > correlation matrix plot ?
author: Daniel Slutsky
(ns noj-book.visualizing-correlation-matrices
(:require [fastmath.stats]
[fastmath.core :as fastmath]
[tablecloth.api :as tc]
[noj-book.datasets]
[scicloj.kindly.v4.kind :as kind]
[clojure.math :as math]
[clojure.string :as str]))12.1 Auxiliary functions
Rounding numbers:
(defn round
[n scale rm]
(.setScale ^java.math.BigDecimal (bigdec n)
(int scale)
^RoundingMode (if (instance? java.math.RoundingMode rm)
rm
(java.math.RoundingMode/valueOf
(str (if (ident? rm) (symbol rm) rm))))))For example (see RoundingMode)
(round (/ 2.0 3) 2 :DOWN)0.66M(round (/ 2.0 3) 2 :UP)0.67M(round (/ 2.0 3) 2 :HALF_EVEN)0.67M12.2 Computing a correlation matrix and representing it as a dataset:
(defn correlations-dataset [data columns-to-use]
(let [matrix (->> columns-to-use
(mapv #(get data %))
fastmath.stats/correlation-matrix)]
(->> matrix
(map-indexed
(fn [i row]
(let [coli (columns-to-use i)]
(->> row
(map-indexed
(fn [j corr]
(let [colj (columns-to-use j)]
{:i i
:j j
:coli coli
:colj colj
:corr corr
:corr-round (round corr 2 :HALF_EVEN)})))))))
(apply concat)
tc/dataset)))For example:
(-> noj-book.datasets/iris
(correlations-dataset [:sepal-length :sepal-width :petal-length :petal-width]))_unnamed [16 6]:
| :i | :j | :coli | :colj | :corr | :corr-round |
|---|---|---|---|---|---|
| 0 | 0 | :sepal-length | :sepal-length | 1.00000000 | 1.000 |
| 0 | 1 | :sepal-length | :sepal-width | -0.11756978 | -0.1200 |
| 0 | 2 | :sepal-length | :petal-length | 0.87175378 | 0.8700 |
| 0 | 3 | :sepal-length | :petal-width | 0.81794113 | 0.8200 |
| 1 | 0 | :sepal-width | :sepal-length | -0.11756978 | -0.1200 |
| 1 | 1 | :sepal-width | :sepal-width | 1.00000000 | 1.000 |
| 1 | 2 | :sepal-width | :petal-length | -0.42844010 | -0.4300 |
| 1 | 3 | :sepal-width | :petal-width | -0.36612593 | -0.3700 |
| 2 | 0 | :petal-length | :sepal-length | 0.87175378 | 0.8700 |
| 2 | 1 | :petal-length | :sepal-width | -0.42844010 | -0.4300 |
| 2 | 2 | :petal-length | :petal-length | 1.00000000 | 1.000 |
| 2 | 3 | :petal-length | :petal-width | 0.96286543 | 0.9600 |
| 3 | 0 | :petal-width | :sepal-length | 0.81794113 | 0.8200 |
| 3 | 1 | :petal-width | :sepal-width | -0.36612593 | -0.3700 |
| 3 | 2 | :petal-width | :petal-length | 0.96286543 | 0.9600 |
| 3 | 3 | :petal-width | :petal-width | 1.00000000 | 1.000 |
12.3 Drawing a heatmap using Echarts
The following function is inspired by an Apache Echarts heatmap tutorial.
(defn echarts-heatmap [{:keys [xyz-data xs ys
min max
series-name]
:or {series-name ""}}]
(kind/echarts
{:tooltip {}
:xAxis {:type :category
:data xs}
:yAxis {:type :category
:data ys}
:visualMap {:min min
:max max
:calculable true
:splitNumber 8
:inRange {:color
["#313695" "#4575b4" "#74add1"
"#abd9e9" "#e0f3f8" "#ffffbf"
"#fee090" "#fdae61" "#f46d43"
"#d73027" "#a50026"]}}
:series [{:name series-name
:type :heatmap
:data xyz-data
:itemStyle {:emphasis {:borderColor "#333"
:borderWidth 2}}
:progressive 1000
:animation false}]}))Here is an example using synthetic data:
(let [n 30]
(echarts-heatmap
{:xyz-data (for [i (range n)
j (range n)]
[i j (fastmath/logistic (* (+ (- i j))
(rand)
(/ 2 (double n))))])
:x-data (range n)
:y-data (range n)
:min 0
:max 1}))Note the slider control and the tooltips.
Here is an example with an actual correlation matrix.
(let [columns-for-correlations [:sepal-length :sepal-width
:petal-length :petal-width]
correlations (-> noj-book.datasets/iris
(correlations-dataset columns-for-correlations)
(tc/select-columns [:coli :colj :corr-round])
tc/rows)]
(echarts-heatmap {:xyz-data correlations
:xs columns-for-correlations
:ys columns-for-correlations
:min -1
:max 1
:series-name "correlation"}))TODO: Improve the layout so that the slider control does not overlap the labels.
12.4 Drawing a heatmap using cljplot
coming soon
12.5 Drawing a heatmap using Vega
coming soon