10 Introduction to Linear Regression
last update: 2024-12-29
In this tutorial, we introduce the fundamentals of linear regression, guided by the In Depth: Linear Regression chapter of the Python Data Science Handbook by Jake VanderPlas.
10.1 Setup
ns noj-book.linear-regression-intro
(:require
(:as ds]
[tech.v3.dataset :as tc]
[tablecloth.api :as tcc]
[tablecloth.column.api :as datetime]
[tech.v3.datatype.datetime :as ds-mod]
[tech.v3.dataset.modelling :as reg]
[fastmath.ml.regression :as kind]
[scicloj.kindly.v4.kind :as rand]
[fastmath.random :as plotly])) [scicloj.tableplot.v1.plotly
10.2 Simple Linear Regression
We begin with the classic straight-line model: for data points \((x, y)\), we assume there is a linear relationship allowing us to predict \(y\) as \[y = ax + b.\] In this formulation, \(a\) is the slope and \(b\) is the intercept, the point where our line would cross the \(y\) axis.
To illustrate, we’ll use Fastmath and Tablecloth to create synthetic data in which the relationship is known to hold with \(a=2\) and \(b=-5\).
For each row in the dataset below, we draw \(x\) uniformly from 0 to 10 and compute \(y = ax + b\) plus an extra random noise term (drawn from a standard Gaussian distribution). This noise is added independently for every row.
def simple-linear-data
(let [rng (rand/rng 1234)
(50
n 2
a 5]
b --> {:x (repeatedly n #(rand/frandom rng 0 10))}
(
tc/dataset:y
(tc/map-columns :x]
[fn [x]
(+ (* a x)
(
b (rand/grandom rng)))))))
simple-linear-data
_unnamed [50 2]:
:x | :y |
---|---|
7.82639551 | 11.39657880 |
9.88273144 | 15.95902529 |
8.14601707 | 11.18658419 |
5.06452560 | 4.50154209 |
8.82313347 | 12.24456668 |
7.30451632 | 9.26730510 |
2.29433346 | 0.10903138 |
3.28054380 | -0.22858143 |
0.61168969 | -4.16880557 |
0.51301837 | -3.73796728 |
… | … |
5.49086523 | 6.02278638 |
4.84104443 | 3.36497550 |
3.24252248 | 1.56584607 |
6.29732704 | 9.54682717 |
5.68643951 | 8.54097107 |
3.21027803 | 2.05249271 |
8.12725163 | 12.86243696 |
0.87016940 | -3.99094271 |
3.81351900 | 3.10835397 |
5.48738384 | 6.28797664 |
9.24097919 | 14.16718955 |
Let’s plot these points using Tableplot’s Plotly API.
-> simple-linear-data
( plotly/layer-point)
10.2.1 Regression using Fastmath
We can now fit a linear model to the data using the Fastmath library.
def simple-linear-data-model
(
(reg/lm;; ys - a "column" sequence of `y` values:
:y)
(simple-linear-data ;; xss - a sequence of "rows", each containing `x` values:
;; (one `x` per row, in our case):
-> simple-linear-data
(:x])
(tc/select-columns [
tc/rows);; options
:names ["x"]})) {
type simple-linear-data-model) (
fastmath.ml.regression.LMData
simple-linear-data-model
:model :ols,
{:intercept? true,
:offset? false,
:transformer nil,
:xtxinv
0x9a735e2 "Array2DRowRealMatrix{{0.0732200272,-0.0115199569},{-0.0115199569,0.002493599}}"],
#object[org.apache.commons.math3.linear.Array2DRowRealMatrix :intercept -5.009631160534331,
:beta [2.0501082309505883],
:coefficients
:estimate -5.009631160534331,
[{:stderr 0.27459774026325406,
:t-value -18.243526533509158,
:p-value 3.158002880662819E-23,
:confidence-interval [-5.5617469214727 -4.457515399595962]}
:estimate 2.0501082309505883,
{:stderr 0.05067518288721502,
:t-value 40.455862498087114,
:p-value 0.0,
:confidence-interval [1.9482189468885893 2.1519975150125874]}],
:offset
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
:weights
1.0
[1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0],
:residuals
:weighted
{0.3612521016440926
[0.7079873867662414
0.5040013081872328
-0.8716523783836756
-0.8341807027704089
-0.6981127586513516
-0.4150306328253658
1.9444201265213537
-0.41320447488202205
-0.2199206970466796
0.9100989974806004
1.8569978822643893
1.3348786315269994
-0.7286041339021478
-0.2576255439635031
2.3031600754150583
-0.5497866292305629
-0.8167151380895525
-0.2884927501055299
0.29089222075703525
0.8484392631738071
-0.23149227234657666
1.7185056819553686
0.9920589483409916
0.03304625406500872
2.0586599748796637
-1.014219275504825
0.5392992543299764
-0.8757692687660887
0.8554055867440131
0.02946366579718962
0.2717202110613339
-0.19442393632580757
-0.6749722739671813
-0.25788775338705827
-1.4857464383018026
-0.5513432107618428
1.4330654709594874
1.0893843554795986
0.22445046404134406
-1.5500583641561878
-0.07204479326470947
-1.6462563270738464
1.8927857802530923
0.48070645307794324
1.210322664596303
0.7652529980813467
-0.29985843351798236
0.04787701691587021
0.23181320352136048],
:raw
0.3612521016440926
[0.7079873867662414
0.5040013081872328
-0.8716523783836756
-0.8341807027704089
-0.6981127586513516
-0.4150306328253658
1.9444201265213537
-0.41320447488202205
-0.2199206970466796
0.9100989974806004
1.8569978822643893
1.3348786315269994
-0.7286041339021478
-0.2576255439635031
2.3031600754150583
-0.5497866292305629
-0.8167151380895525
-0.2884927501055299
0.29089222075703525
0.8484392631738071
-0.23149227234657666
1.7185056819553686
0.9920589483409916
0.03304625406500872
2.0586599748796637
-1.014219275504825
0.5392992543299764
-0.8757692687660887
0.8554055867440131
0.02946366579718962
0.2717202110613339
-0.19442393632580757
-0.6749722739671813
-0.25788775338705827
-1.4857464383018026
-0.5513432107618428
1.4330654709594874
1.0893843554795986
0.22445046404134406
-1.5500583641561878
-0.07204479326470947
-1.6462563270738464
1.8927857802530923
0.48070645307794324
1.210322664596303
0.7652529980813467
-0.29985843351798236
0.04787701691587021
0.23181320352136048]},
:fitted
11.035326696527326
(15.251037904133943
11.69058549353633
5.37319446659459
13.078747386275401
9.965417860903498
0.30599925385231774
-1.7158386943858774
3.7556010988888913
-3.957887978235413
-0.3340133960152736
6.178822739962399
3.745403338379761
6.032063421425679
3.5787138744604077
-8.805859216617359
3.083646648318119
2.8776633585830047
-2.2392150615599276
-1.9752250635370325
0.945930765593932
-8.37979314143435
1.1505809150555066
-9.446803488582011
3.4630786033861813
-14.064243243849672
6.977465313953921
6.998744031883334
1.3164631701684963
-2.9689413063876624
-3.767589745785883
-11.426172959629623
5.984029650847299
3.3318596798935123
-2.9922086415793476
-10.36748464701741
3.8033535565474246
13.721281183454975
9.960181030290743
6.2472368436777455
4.91503386353266
1.637890860868235
7.900570840490801
6.648185292248069
1.5717862610398843
11.65211429100984
3.225689708876782
-2.8084955323167478
6.240099621295117
13.935376348442555),
:df {:residual 48, :model 1, :intercept 1},
:observations 50,
:names ["Intercept" "x"],
:r-squared 0.9715078882181706,
:adjusted-r-squared 0.9709143025560492,
:sigma2 1.0298264269092297,
:sigma 1.0148036395821753,
:tss 1734.9247002171178,
:rss 49.43166849164302,
:regss 1685.4930317254748,
:msreg 1685.4930317254748,
:qt 2.010634757624228,
:f-statistic 1636.676810464135,
:p-value 0.0,
:ll
:log-likelihood -70.66113355915249,
{:aic 147.32226711830498,
:bic 153.0583361345894,
:aic-rss 3.4284137978377234,
:bic-rss 7.252459808694016},
:analysis #<Delay@75ac47e7: :not-delivered>}
Printing the model gives a tabular summary: We’ll capture the printed output and display it via Kindly for cleaner formatting.
(kind/codewith-out-str
(println
( simple-linear-data-model)))
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|----------+-----------+----------+----------+----------|2.30316 | -0.705736 | 0.040462 | 0.744842 | 1.892786 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-----------+-----------+----------+------------+----------+-----------------------|5.009631 | 0.274598 | -18.243527 | 0.0 | [-5.561747 -4.457515] |
| Intercept | -2.050108 | 0.050675 | 40.455862 | 0.0 | [1.948219 2.151998] |
| x |
1636.676810464135 on degrees of freedom: {:residual 48, :model 1, :intercept 1}
F-statistic: 0.0
p-value:
0.9715078882181706
R2: 0.9709143025560492
Adjusted R2: 1.0148036395821753 on 48 degrees of freedom
Residual standard error: 147.32226711830498
AIC:
As you can see, the estimated coefficients match our intercept \(b\) and slope \(a\) (the coefficient of \(x\)).
10.2.2 Dataset ergonomics
Below are a couple of helper functions that simplify how we use regression with datasets and display model summaries. We have similar ideas under development in the Tablemath library, but it is still in an experimental stage and not part of Noj yet.
defn lm
("Compute a linear regression model for `dataset`.
The first column marked as target is the target.
All the columns unmarked as target are the features.
The resulting model is of type `fastmath.ml.regression.LMData`,
created via [Fastmath](https://github.com/generateme/fastmath).
See [fastmath.ml.regression.lm](https://generateme.github.io/fastmath/clay/ml.html#lm)
for `options`."
([dataset]nil))
(lm dataset
([dataset options]let [inference-column-name (-> dataset
(
ds-mod/inference-target-column-namesfirst)
-> dataset
ds-without-target (
(tc/drop-columns [inference-column-name]))]
(reg/lm;; ys
get dataset inference-column-name)
(;; xss
(tc/rows ds-without-target);; options
merge {:names (-> ds-without-target
(
tc/column-namesvec)}
options)))))
defn summary
("Generate a summary of a linear model."
[lmdata]
(kind/codewith-out-str
(println
( lmdata))))
-> simple-linear-data
(:y)
(ds-mod/set-inference-target
lm summary)
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|----------+-----------+----------+----------+----------|2.30316 | -0.705736 | 0.040462 | 0.744842 | 1.892786 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-----------+-----------+----------+------------+----------+-----------------------|5.009631 | 0.274598 | -18.243527 | 0.0 | [-5.561747 -4.457515] |
| Intercept | -:x | 2.050108 | 0.050675 | 40.455862 | 0.0 | [1.948219 2.151998] |
|
1636.676810464135 on degrees of freedom: {:residual 48, :model 1, :intercept 1}
F-statistic: 0.0
p-value:
0.9715078882181706
R2: 0.9709143025560492
Adjusted R2: 1.0148036395821753 on 48 degrees of freedom
Residual standard error: 147.32226711830498
AIC:
10.2.3 Prediction
Once we have a linear model, we can generate new predictions. For instance, let’s predict \(y\) when \(x=3\):
3]) (simple-linear-data-model [
1.1406935323174334
10.2.4 Displaying the regression line
We can visualize the fitted line by adding a smooth layer to our scatter plot. Tableplot makes this convenient:
-> simple-linear-data
("data"})
(plotly/layer-point {:=name "prediction"})) (plotly/layer-smooth {:=name
Alternatively, we can build the regression line explicitly. We’ll obtain predictions and then plot them:
-> simple-linear-data
(:prediction
(tc/map-columns :x]
[
simple-linear-data-model)"data"})
(plotly/layer-point {:=name :prediction
(plotly/layer-smooth {:=y "prediction"})) :=name
10.3 Multiple linear regression
We can easily extend these ideas to multiple linear predictors.
def multiple-linear-data
(let [rng (rand/rng 1234)
(50
n 2
a0 3
a1 -5]
b --> {:x0 (repeatedly n #(rand/frandom rng 0 10))
(:x1 (repeatedly n #(rand/frandom rng 0 10))}
tc/dataset:y
(tc/map-columns :x0 :x1]
[fn [x0 x1]
(+ (* a0 x0)
(* a1 x1)
(
b (rand/grandom rng)))))))
def multiple-linear-data-model
(-> multiple-linear-data
(:y)
(ds-mod/set-inference-target lm))
(summary multiple-linear-data-model)
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|-----------+-----------+-----------+----------+----------|2.474281 | -0.771093 | -0.053932 | 0.755103 | 2.382538 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-----------+-----------+----------+------------+----------+-----------------------|6.432147 | 0.4166 | -15.439634 | 0.0 | [-7.270238 -5.594057] |
| Intercept | -:x0 | 2.06835 | 0.052566 | 39.347644 | 0.0 | [1.962601 2.1741] |
| :x1 | -2.866676 | 0.060302 | -47.538623 | 0.0 | [-2.987988 -2.745364] |
|
1555.4136204883068 on degrees of freedom: {:residual 47, :model 2, :intercept 1}
F-statistic: 0.0
p-value:
0.9851163485481035
R2: 0.98448300167781
Adjusted R2: 1.0469780323724478 on 47 degrees of freedom
Residual standard error: 151.39087815167295
AIC:
Visualizing multiple dimensions is more involved. In the case of two features, we can use a 3D scatterplot and a 3D surface. Let us do that using Tableplot’s Plotly API.
-> multiple-linear-data
(:3d
(plotly/layer-point {:=coordinates :x0
:=x :x1
:=y :y})
:=z let [xs (range 11)
(plotly/layer-surface {:=dataset (range 11)]
ys (
(tc/dataset:x xs
{:y ys
:z (for [y ys]
for [x xs]
(
(multiple-linear-data-model
[x y])))}))0.5})) :=mark-opacity
10.4 Coming soon: Polynomial regression 🛠
10.5 Coming soon: One-hot encoding 🛠
10.6 Coming soon: Regularization 🛠
10.7 Example: Predicting Bicycle Traffic
As in the Python Data Science Handbook, we’ll try predicting the daily number of bicycle trips across the Fremont Bridge in Seattle. The features will include weather, season, day of week, and related factors.
10.7.1 Reading and parsing data
def column-name-mapping
("Fremont Bridge Sidewalks, south of N 34th St" :total
{"Fremont Bridge Sidewalks, south of N 34th St Cyclist West Sidewalk" :west
"Fremont Bridge Sidewalks, south of N 34th St Cyclist East Sidewalk" :east
"Date" :datetime})
(column-name-mapping"Fremont Bridge Sidewalks, south of N 34th St")
:total
def counts
("data/seattle-bikes-and-weather/Fremont_Bridge_Bicycle_Counter.csv.gz"
(tc/dataset :key-fn column-name-mapping
{:parser-fn {"Date" [:local-date-time "MM/dd/yyyy hh:mm:ss a"]}}))
counts
data/seattle-bikes-and-weather/Fremont_Bridge_Bicycle_Counter.csv.gz [106608 4]:
:datetime | :total | :west | :east |
---|---|---|---|
2012-10-02T13:00 | 55 | 7 | 48 |
2012-10-02T14:00 | 130 | 55 | 75 |
2012-10-02T15:00 | 152 | 81 | 71 |
2012-10-02T16:00 | 278 | 167 | 111 |
2012-10-02T17:00 | 563 | 393 | 170 |
2012-10-02T18:00 | 381 | 236 | 145 |
2012-10-02T19:00 | 175 | 104 | 71 |
2012-10-02T20:00 | 86 | 51 | 35 |
2012-10-02T21:00 | 63 | 35 | 28 |
2012-10-02T22:00 | 42 | 27 | 15 |
… | … | … | … |
2024-11-30T13:00 | 147 | 62 | 85 |
2024-11-30T14:00 | 154 | 73 | 81 |
2024-11-30T15:00 | 118 | 57 | 61 |
2024-11-30T16:00 | 88 | 51 | 37 |
2024-11-30T17:00 | 46 | 11 | 35 |
2024-11-30T18:00 | 46 | 18 | 28 |
2024-11-30T19:00 | 69 | 14 | 55 |
2024-11-30T20:00 | 18 | 8 | 10 |
2024-11-30T21:00 | 49 | 15 | 34 |
2024-11-30T22:00 | 14 | 4 | 10 |
2024-11-30T23:00 | 10 | 5 | 5 |
def weather
("data/seattle-bikes-and-weather/BicycleWeather.csv.gz"
(tc/dataset :key-fn keyword})) {
weather
data/seattle-bikes-and-weather/BicycleWeather.csv.gz [1340 26]:
:STATION | :STATION_NAME | :DATE | :PRCP | :SNWD | :SNOW | :TMAX | :TMIN | :AWND | :WDF2 | :WDF5 | :WSF2 | :WSF5 | :FMTM | :WT14 | :WT01 | :WT17 | :WT05 | :WT02 | :WT22 | :WT04 | :WT13 | :WT16 | :WT08 | :WT18 | :WT03 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120101 | 0 | 0 | 0 | 128 | 50 | 47 | 100 | 90 | 89 | 112 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120102 | 109 | 0 | 0 | 106 | 28 | 45 | 180 | 200 | 130 | 179 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120103 | 8 | 0 | 0 | 117 | 72 | 23 | 180 | 170 | 54 | 67 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120104 | 203 | 0 | 0 | 122 | 56 | 47 | 180 | 190 | 107 | 148 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120105 | 13 | 0 | 0 | 89 | 28 | 61 | 200 | 220 | 107 | 165 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120106 | 25 | 0 | 0 | 44 | 22 | 22 | 180 | 180 | 45 | 63 | -9999 | 1 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120107 | 0 | 0 | 0 | 72 | 28 | 23 | 170 | 180 | 54 | 63 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120108 | 0 | 0 | 0 | 100 | 28 | 20 | 160 | 200 | 45 | 63 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120109 | 43 | 0 | 0 | 94 | 50 | 34 | 200 | 200 | 67 | 89 | -9999 | 1 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120110 | 10 | 0 | 0 | 61 | 6 | 34 | 20 | 30 | 89 | 107 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
… | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150822 | 0 | 0 | 0 | 267 | 122 | 25 | 20 | 20 | 63 | 76 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150823 | 0 | 0 | 0 | 278 | 139 | 18 | 10 | 10 | 67 | 81 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150824 | 0 | 0 | 0 | 239 | 122 | 23 | 190 | 190 | 54 | 67 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150825 | 0 | 0 | 0 | 256 | 122 | 34 | 350 | 360 | 63 | 76 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150826 | 0 | 0 | 0 | 283 | 139 | 17 | 30 | 40 | 58 | 67 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150827 | 0 | 0 | 0 | 294 | 144 | 21 | 230 | 200 | 45 | 63 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150828 | 5 | 0 | 0 | 233 | 156 | 26 | 230 | 240 | 81 | 103 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150829 | 325 | 0 | 0 | 222 | 133 | 58 | 210 | 210 | 157 | 206 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150830 | 102 | 0 | 0 | 200 | 128 | 47 | 200 | 200 | 89 | 112 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150831 | 0 | 0 | 0 | 189 | 161 | 58 | 210 | 210 | 112 | 134 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150901 | 58 | 0 | 0 | 194 | 139 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
10.7.2 Preprocessing
The bike counts come in hourly data, but our weather information is daily. We’ll need to aggregate the hourly counts into daily totals before combining the datasets.
In the Python handbook, one does:
= counts.resample('d').sum() daily
Since Tablecloth’s time series features are still evolving, we’ll be a bit more explicit:
def daily-totals
(-> counts
(fn [{:keys [datetime]}]
(tc/group-by (:date (datetime/local-date-time->local-date
{
datetime)})):total :west :east]
(tc/aggregate-columns [ tcc/sum)))
daily-totals
_unnamed [4443 4]:
:date | :total | :west | :east |
---|---|---|---|
2012-10-02 | 1938.0 | 1165.0 | 773.0 |
2012-10-03 | 3521.0 | 1761.0 | 1760.0 |
2012-10-04 | 3475.0 | 1767.0 | 1708.0 |
2012-10-05 | 3148.0 | 1590.0 | 1558.0 |
2012-10-06 | 2006.0 | 926.0 | 1080.0 |
2012-10-07 | 2142.0 | 951.0 | 1191.0 |
2012-10-08 | 3537.0 | 1708.0 | 1829.0 |
2012-10-09 | 3501.0 | 1742.0 | 1759.0 |
2012-10-10 | 3235.0 | 1587.0 | 1648.0 |
2012-10-11 | 3047.0 | 1468.0 | 1579.0 |
… | … | … | … |
2024-11-20 | 2300.0 | 787.0 | 1513.0 |
2024-11-21 | 2382.0 | 775.0 | 1607.0 |
2024-11-22 | 1473.0 | 536.0 | 937.0 |
2024-11-23 | 1453.0 | 652.0 | 801.0 |
2024-11-24 | 727.0 | 311.0 | 416.0 |
2024-11-25 | 1483.0 | 486.0 | 997.0 |
2024-11-26 | 2173.0 | 727.0 | 1446.0 |
2024-11-27 | 1522.0 | 548.0 | 974.0 |
2024-11-28 | 631.0 | 261.0 | 370.0 |
2024-11-29 | 833.0 | 368.0 | 465.0 |
2024-11-30 | 1178.0 | 509.0 | 669.0 |
10.7.3 Prediction by day-of-week
Next, we’ll explore a simple regression by day of week.
def days-of-week
(:Mon :Tue :Wed :Thu :Fri :Sat :Sun]) [
We’ll convert the numeric day-of-week to the corresponding keyword:
def idx->day-of-week
(comp days-of-week dec)) (
For example,
1) (idx->day-of-week
:Mon
7) (idx->day-of-week
:Sun
Now, let’s build our dataset:
def totals-with-day-of-week
(-> daily-totals
(:day-of-week
(tc/add-column fn [ds]
(map idx->day-of-week
(
(datetime/long-temporal-field:day-of-week
:date ds)))))
(:total :day-of-week]))) (tc/select-columns [
totals-with-day-of-week
_unnamed [4443 2]:
:total | :day-of-week |
---|---|
1938.0 | :Tue |
3521.0 | :Wed |
3475.0 | :Thu |
3148.0 | :Fri |
2006.0 | :Sat |
2142.0 | :Sun |
3537.0 | :Mon |
3501.0 | :Tue |
3235.0 | :Wed |
3047.0 | :Thu |
… | … |
2300.0 | :Wed |
2382.0 | :Thu |
1473.0 | :Fri |
1453.0 | :Sat |
727.0 | :Sun |
1483.0 | :Mon |
2173.0 | :Tue |
1522.0 | :Wed |
631.0 | :Thu |
833.0 | :Fri |
1178.0 | :Sat |
def totals-with-one-hot-days-of-week
(-> (reduce (fn [dataset day-of-week]
(-> dataset
(
(tc/add-column day-of-week-> (:day-of-week %)
#(
(tcc/eq day-of-week);; convert booleans to 0/1
1)))))
(tcc/*
totals-with-day-of-week
days-of-week):day-of-week])
(tc/drop-columns [:total))) (ds-mod/set-inference-target
-> totals-with-one-hot-days-of-week
( (tc/select-columns ds-mod/inference-column?))
_unnamed [0 0]
Since the binary columns sum to 1, they’re collinear, and we won’t use an intercept. This way, each coefficient directly reflects the expected bike count for that day of week.
def days-of-week-model
(
(lm totals-with-one-hot-days-of-week:intercept? false})) {
Let’s take a look at the results:
-> days-of-week-model
(println
with-out-str
kind/code)
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|--------------+-------------+------------+------------+-------------|3034.944882 | -841.755906 | -90.179528 | 828.820472 | 3745.244094 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-------+-------------+-----------+-----------+----------+---------------------------|:Mon | 2785.741325 | 44.521658 | 62.570476 | 0.0 | [2698.456664 2873.025986] |
| :Tue | 3115.527559 | 44.486587 | 70.032964 | 0.0 | [3028.311653 3202.743465] |
| :Wed | 3109.944882 | 44.486587 | 69.907472 | 0.0 | [3022.728976 3197.160788] |
| :Thu | 2961.179528 | 44.486587 | 66.563423 | 0.0 | [2873.963622 3048.395433] |
| :Fri | 2623.755906 | 44.486587 | 58.978583 | 0.0 | [2536.54 2710.971811] |
| :Sat | 1688.029921 | 44.486587 | 37.944693 | 0.0 | [1600.814015 1775.245827] |
| :Sun | 1576.178233 | 44.521658 | 35.402506 | 0.0 | [1488.893572 1663.462894] |
|
3472.719277560978 on degrees of freedom: {:residual 4436, :model 7, :intercept 0}
F-statistic: 0.0
p-value:
0.8456776967289252
R2: 0.8454341764126724
Adjusted R2: 1121.0266948292917 on 4436 degrees of freedom
Residual standard error: 75015.17638480052
AIC:
We can clearly see weekend versus weekday differences.