9 Introduction to Linear Regression
last update: 2024-12-29
In this tutorial, we introduce the fundamentals of linear regression, guided by the In Depth: Linear Regression chapter of the Python Data Science Handbook by Jake VanderPlas.
9.1 Setup
ns noj-book.linear-regression-intro
(:require
(:as ds]
[tech.v3.dataset :as tc]
[tablecloth.api :as tcc]
[tablecloth.column.api :as datetime]
[tech.v3.datatype.datetime :as ds-mod]
[tech.v3.dataset.modelling :as reg]
[fastmath.ml.regression :as kind]
[scicloj.kindly.v4.kind :as rand]
[fastmath.random :as plotly])) [scicloj.tableplot.v1.plotly
9.2 Simple Linear Regression
We begin with the classic straight-line model: for data points \((x, y)\), we assume there is a linear relationship allowing us to predict \(y\) as \[y = ax + b.\] In this formulation, \(a\) is the slope and \(b\) is the intercept, the point where our line would cross the \(y\) axis.
To illustrate, we’ll use Fastmath and Tablecloth to create synthetic data in which the relationship is known to hold with \(a=2\) and \(b=-5\).
For each row in the dataset below, we draw \(x\) uniformly from 0 to 10 and compute \(y = ax + b\) plus an extra random noise term (drawn from a standard Gaussian distribution). This noise is added independently for every row.
def simple-linear-data
(let [rng (rand/rng 1234)
(50
n 2
a 5]
b --> {:x (repeatedly n #(rand/frandom rng 0 10))}
(
tc/dataset:y
(tc/map-columns :x]
[fn [x]
(+ (* a x)
(
b (rand/grandom rng)))))))
simple-linear-data
_unnamed [50 2]:
:x | :y |
---|---|
6.03440666 | 8.05202814 |
9.03218937 | 13.06317204 |
3.61249399 | 1.67806430 |
8.93980598 | 12.26076280 |
8.18699551 | 11.16178930 |
5.55749226 | 5.67785089 |
4.08149433 | 1.95876215 |
6.48022079 | 7.18676642 |
3.35672855 | 2.17185976 |
3.09243441 | 0.82392731 |
… | … |
0.33738434 | -5.67425411 |
5.36782312 | 6.14683293 |
2.72949219 | -0.60467043 |
3.29467726 | 1.63686159 |
4.32346392 | 3.09767743 |
9.27556229 | 13.94324328 |
8.12368488 | 12.51588689 |
3.13823271 | 1.79607263 |
1.17235839 | -2.73687074 |
7.61648846 | 11.41161444 |
3.32321119 | 2.48004498 |
Let’s plot these points using Tableplot’s Plotly API.
-> simple-linear-data
( plotly/layer-point)
9.2.1 Regression using Fastmath
We can now fit a linear model to the data using the Fastmath library.
def simple-linear-data-model
(
(reg/lm;; ys - a "column" sequence of `y` values:
:y)
(simple-linear-data ;; xss - a sequence of "rows", each containing `x` values:
;; (one `x` per row, in our case):
-> simple-linear-data
(:x])
(tc/select-columns [
tc/rows);; options
:names ["x"]})) {
type simple-linear-data-model) (
fastmath.ml.regression.LMData
simple-linear-data-model
:model :ols,
{:intercept? true,
:offset? false,
:transformer nil,
:xtxinv
0x7e3b2f59 "Array2DRowRealMatrix{{0.1182880912,-0.0167213395},{-0.0167213395,0.0028447312}}"],
#object[org.apache.commons.math3.linear.Array2DRowRealMatrix :intercept -5.522274656289029,
:beta [2.0904098521181855],
:coefficients
:estimate -5.522274656289029,
[{:stderr 0.34084329409981673,
:t-value -16.20179933677034,
:p-value 4.308415286624282E-21,
:confidence-interval [-6.207586030309257 -4.8369632822688]}
:estimate 2.0904098521181855,
{:stderr 0.05285728565914854,
:t-value 39.54818765379371,
:p-value 0.0,
:confidence-interval [1.9841331563782287 2.196686547858142]}],
:offset
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
:weights
1.0
[1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0],
:residuals
:weighted
{0.9599196601227211
[0.2955309457801718
-0.35125407945781095
-0.9048210490667241
-0.4301121045308314
-0.4173110148539987
-1.0509591561629519
-0.8372763143881734
-0.6771959783217039
0.11825338091231385
-0.634276656180436
-1.3627672709252225
-1.1815590487471566
1.29984848143877
0.6790898364962157
-0.2982248877650049
-0.7227215276166357
0.3043864588308445
1.3921361752488086
-0.1614431066325368
0.09678276215922388
-1.434408067922564
-1.3347698000923671
0.947844468129599
-0.22766656302475496
-1.1250099128337663
-1.0706302850022116
2.3331333824284464
0.15440858317563055
0.48119442368840026
0.4185396269071653
0.8897334808891095
1.914325736101656
-0.43513467547342
1.222980221838652
1.648533281586623
0.6904642901430407
-0.7140173067549114
-2.323127803415707
-0.8572510108065963
-0.44815724684133684
0.7881531321373942
-0.27191044505861206
0.4178594760233487
-0.07579115216747567
1.0563306326348236
0.7581547117165783
0.33469438379648775
1.0123065916852632
1.0554462145283514],
:raw
0.9599196601227211
[0.2955309457801718
-0.35125407945781095
-0.9048210490667241
-0.4301121045308314
-0.4173110148539987
-1.0509591561629519
-0.8372763143881734
-0.6771959783217039
0.11825338091231385
-0.634276656180436
-1.3627672709252225
-1.1815590487471566
1.29984848143877
0.6790898364962157
-0.2982248877650049
-0.7227215276166357
0.3043864588308445
1.3921361752488086
-0.1614431066325368
0.09678276215922388
-1.434408067922564
-1.3347698000923671
0.947844468129599
-0.22766656302475496
-1.1250099128337663
-1.0706302850022116
2.3331333824284464
0.15440858317563055
0.48119442368840026
0.4185396269071653
0.8897334808891095
1.914325736101656
-0.43513467547342
1.222980221838652
1.648533281586623
0.6904642901430407
-0.7140173067549114
-2.323127803415707
-0.8572510108065963
-0.44815724684133684
0.7881531321373942
-0.27191044505861206
0.4178594760233487
-0.07579115216747567
1.0563306326348236
0.7581547117165783
0.33469438379648775
1.0123065916852632
1.0554462145283514]},
:fitted
7.0921084816168465
(13.358702987287263
2.02931837499574
13.165583849728762
11.591901409299837
6.095161909067872
3.0097213053501557
8.024042736806443
1.4946637834023315
0.9421806936290169
11.141967569931797
6.432158003787602
4.59250716269613
-2.308991352566302
3.8534209195196363
-0.8030334817357971
4.361015010739046
13.881843714267223
1.8247880416884117
7.9592417378868605
13.39100081788244
4.717597932543376
0.7728518258938326
10.452205243530223
6.045920725455697
11.97085717252472
9.744664258203414
14.744335939142537
9.260002426182577
9.93530439123472
12.702788492301046
9.027225196899504
12.881936689990958
3.7888464121568886
13.539788935198672
13.074033126043666
14.383633304547335
8.460745250845175
11.230935623439986
4.817003101423537
-5.698675686405999
0.18348270374058906
1.3649511424177998
3.5155369106223615
13.867452131089237
11.459556258883733
1.0379179147062416
3.0715651199496325
-10.39930785219559
1.4245987624046927),
:df {:residual 48, :model 1, :intercept 1},
:observations 50,
:names ["Intercept" "x"],
:r-squared 0.9702244175728484,
:adjusted-r-squared 0.9696040929389494,
:sigma2 0.982128884935045,
:sigma 0.9910241596121888,
:tss 1583.2498521974987,
:rss 47.14218647688216,
:regss 1536.1076657206165,
:msreg 1536.1076657206165,
:qt 2.010634757624228,
:f-statistic 1564.0591466996818,
:p-value 0.0,
:ll
:log-likelihood -69.47555850093158,
{:aic 144.95111700186317,
:bic 150.6871860181476,
:aic-rss 1.057263681395916,
:bic-rss 4.8813096922522075},
:analysis #<Delay@45266dae: :not-delivered>}
Printing the model gives a tabular summary: We’ll capture the printed output and display it via Kindly for cleaner formatting.
(kind/codewith-out-str
(println
( simple-linear-data-model)))
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|-----------+-----------+-----------+----------+----------|2.323128 | -0.732551 | -0.010496 | 0.791049 | 2.333133 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-----------+-----------+----------+------------+----------+-----------------------|5.522275 | 0.340843 | -16.201799 | 0.0 | [-6.207586 -4.836963] |
| Intercept | -2.09041 | 0.052857 | 39.548188 | 0.0 | [1.984133 2.196687] |
| x |
1564.0591466996818 on degrees of freedom: {:residual 48, :model 1, :intercept 1}
F-statistic: 0.0
p-value:
0.9702244175728484
R2: 0.9696040929389494
Adjusted R2: 0.9910241596121888 on 48 degrees of freedom
Residual standard error: 144.95111700186317
AIC:
As you can see, the estimated coefficients match our intercept \(b\) and slope \(a\) (the coefficient of \(x\)).
9.2.2 Dataset ergonomics
Below are a couple of helper functions that simplify how we use regression with datasets and display model summaries. We have similar ideas under development in the Tablemath library, but it is still in an experimental stage and not part of Noj yet.
defn lm
("Compute a linear regression model for `dataset`.
The first column marked as target is the target.
All the columns unmarked as target are the features.
The resulting model is of type `fastmath.ml.regression.LMData`,
created via [Fastmath](https://github.com/generateme/fastmath).
See [fastmath.ml.regression.lm](https://generateme.github.io/fastmath/clay/ml.html#lm)
for `options`."
([dataset]nil))
(lm dataset
([dataset options]let [inference-column-name (-> dataset
(
ds-mod/inference-target-column-namesfirst)
-> dataset
ds-without-target (
(tc/drop-columns [inference-column-name]))]
(reg/lm;; ys
get dataset inference-column-name)
(;; xss
(tc/rows ds-without-target);; options
merge {:names (-> ds-without-target
(
tc/column-namesvec)}
options)))))
defn summary
("Generate a summary of a linear model."
[lmdata]
(kind/codewith-out-str
(println
( lmdata))))
-> simple-linear-data
(:y)
(ds-mod/set-inference-target
lm summary)
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|-----------+-----------+-----------+----------+----------|2.323128 | -0.732551 | -0.010496 | 0.791049 | 2.333133 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-----------+-----------+----------+------------+----------+-----------------------|5.522275 | 0.340843 | -16.201799 | 0.0 | [-6.207586 -4.836963] |
| Intercept | -:x | 2.09041 | 0.052857 | 39.548188 | 0.0 | [1.984133 2.196687] |
|
1564.0591466996818 on degrees of freedom: {:residual 48, :model 1, :intercept 1}
F-statistic: 0.0
p-value:
0.9702244175728484
R2: 0.9696040929389494
Adjusted R2: 0.9910241596121888 on 48 degrees of freedom
Residual standard error: 144.95111700186317
AIC:
9.2.3 Prediction
Once we have a linear model, we can generate new predictions. For instance, let’s predict \(y\) when \(x=3\):
3]) (simple-linear-data-model [
0.7489549000655282
9.2.4 Displaying the regression line
We can visualize the fitted line by adding a smooth layer to our scatter plot. Tableplot makes this convenient:
-> simple-linear-data
("data"})
(plotly/layer-point {:=name "prediction"})) (plotly/layer-smooth {:=name
Alternatively, we can build the regression line explicitly. We’ll obtain predictions and then plot them:
-> simple-linear-data
(:prediction
(tc/map-columns :x]
[
simple-linear-data-model)"data"})
(plotly/layer-point {:=name :prediction
(plotly/layer-smooth {:=y "prediction"})) :=name
9.3 Multiple linear regression
We can easily extend these ideas to multiple linear predictors.
def multiple-linear-data
(let [rng (rand/rng 1234)
(50
n 2
a0 3
a1 -5]
b --> {:x0 (repeatedly n #(rand/frandom rng 0 10))
(:x1 (repeatedly n #(rand/frandom rng 0 10))}
tc/dataset:y
(tc/map-columns :x0 :x1]
[fn [x0 x1]
(+ (* a0 x0)
(* a1 x1)
(
b (rand/grandom rng)))))))
def multiple-linear-data-model
(-> multiple-linear-data
(:y)
(ds-mod/set-inference-target lm))
(summary multiple-linear-data-model)
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|----------+-----------+----------+----------+----------|2.16068 | -0.848273 | 0.103585 | 0.671132 | 2.910758 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-----------+-----------+----------+------------+----------+-----------------------|4.802654 | 0.455012 | -10.555001 | 0.0 | [-5.718021 -3.887288] |
| Intercept | -:x0 | 1.980637 | 0.059387 | 33.351222 | 0.0 | [1.861165 2.100109] |
| :x1 | -3.027942 | 0.054026 | -56.04627 | 0.0 | [-3.136627 -2.919256] |
|
2233.2259214006704 on degrees of freedom: {:residual 47, :model 2, :intercept 1}
F-statistic: 0.0
p-value:
0.9895866840642241
R2: 0.9891435642371699
Adjusted R2: 1.1137695540668593 on 47 degrees of freedom
Residual standard error: 157.5751087956777
AIC:
Visualizing multiple dimensions is more involved. In the case of two features, we can use a 3D scatterplot and a 3D surface. Let us do that using Tableplot’s Plotly API.
-> multiple-linear-data
(:3d
(plotly/layer-point {:=coordinates :x0
:=x :x1
:=y :y})
:=z let [xs (range 11)
(plotly/layer-surface {:=dataset (range 11)]
ys (
(tc/dataset:x xs
{:y ys
:z (for [y ys]
for [x xs]
(
(multiple-linear-data-model
[x y])))}))0.5})) :=mark-opacity
9.4 Coming soon: Polynomial regression 🛠
9.5 Coming soon: One-hot encoding 🛠
9.6 Coming soon: Regularization 🛠
9.7 Example: Predicting Bicycle Traffic
As in the Python Data Science Handbook, we’ll try predicting the daily number of bicycle trips across the Fremont Bridge in Seattle. The features will include weather, season, day of week, and related factors.
9.7.1 Reading and parsing data
def column-name-mapping
("Fremont Bridge Sidewalks, south of N 34th St" :total
{"Fremont Bridge Sidewalks, south of N 34th St Cyclist West Sidewalk" :west
"Fremont Bridge Sidewalks, south of N 34th St Cyclist East Sidewalk" :east
"Date" :datetime})
(column-name-mapping"Fremont Bridge Sidewalks, south of N 34th St")
:total
def counts
("data/seattle-bikes-and-weather/Fremont_Bridge_Bicycle_Counter.csv.gz"
(tc/dataset :key-fn column-name-mapping
{:parser-fn {"Date" [:local-date-time "MM/dd/yyyy hh:mm:ss a"]}}))
counts
data/seattle-bikes-and-weather/Fremont_Bridge_Bicycle_Counter.csv.gz [106608 4]:
:datetime | :total | :west | :east |
---|---|---|---|
2012-10-02T13:00 | 55 | 7 | 48 |
2012-10-02T14:00 | 130 | 55 | 75 |
2012-10-02T15:00 | 152 | 81 | 71 |
2012-10-02T16:00 | 278 | 167 | 111 |
2012-10-02T17:00 | 563 | 393 | 170 |
2012-10-02T18:00 | 381 | 236 | 145 |
2012-10-02T19:00 | 175 | 104 | 71 |
2012-10-02T20:00 | 86 | 51 | 35 |
2012-10-02T21:00 | 63 | 35 | 28 |
2012-10-02T22:00 | 42 | 27 | 15 |
… | … | … | … |
2024-11-30T13:00 | 147 | 62 | 85 |
2024-11-30T14:00 | 154 | 73 | 81 |
2024-11-30T15:00 | 118 | 57 | 61 |
2024-11-30T16:00 | 88 | 51 | 37 |
2024-11-30T17:00 | 46 | 11 | 35 |
2024-11-30T18:00 | 46 | 18 | 28 |
2024-11-30T19:00 | 69 | 14 | 55 |
2024-11-30T20:00 | 18 | 8 | 10 |
2024-11-30T21:00 | 49 | 15 | 34 |
2024-11-30T22:00 | 14 | 4 | 10 |
2024-11-30T23:00 | 10 | 5 | 5 |
def weather
("data/seattle-bikes-and-weather/BicycleWeather.csv.gz"
(tc/dataset :key-fn keyword})) {
weather
data/seattle-bikes-and-weather/BicycleWeather.csv.gz [1340 26]:
:STATION | :STATION_NAME | :DATE | :PRCP | :SNWD | :SNOW | :TMAX | :TMIN | :AWND | :WDF2 | :WDF5 | :WSF2 | :WSF5 | :FMTM | :WT14 | :WT01 | :WT17 | :WT05 | :WT02 | :WT22 | :WT04 | :WT13 | :WT16 | :WT08 | :WT18 | :WT03 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120101 | 0 | 0 | 0 | 128 | 50 | 47 | 100 | 90 | 89 | 112 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120102 | 109 | 0 | 0 | 106 | 28 | 45 | 180 | 200 | 130 | 179 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120103 | 8 | 0 | 0 | 117 | 72 | 23 | 180 | 170 | 54 | 67 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120104 | 203 | 0 | 0 | 122 | 56 | 47 | 180 | 190 | 107 | 148 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120105 | 13 | 0 | 0 | 89 | 28 | 61 | 200 | 220 | 107 | 165 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120106 | 25 | 0 | 0 | 44 | 22 | 22 | 180 | 180 | 45 | 63 | -9999 | 1 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120107 | 0 | 0 | 0 | 72 | 28 | 23 | 170 | 180 | 54 | 63 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120108 | 0 | 0 | 0 | 100 | 28 | 20 | 160 | 200 | 45 | 63 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120109 | 43 | 0 | 0 | 94 | 50 | 34 | 200 | 200 | 67 | 89 | -9999 | 1 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | 1 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20120110 | 10 | 0 | 0 | 61 | 6 | 34 | 20 | 30 | 89 | 107 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 |
… | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … | … |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150822 | 0 | 0 | 0 | 267 | 122 | 25 | 20 | 20 | 63 | 76 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150823 | 0 | 0 | 0 | 278 | 139 | 18 | 10 | 10 | 67 | 81 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | 1 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150824 | 0 | 0 | 0 | 239 | 122 | 23 | 190 | 190 | 54 | 67 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150825 | 0 | 0 | 0 | 256 | 122 | 34 | 350 | 360 | 63 | 76 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150826 | 0 | 0 | 0 | 283 | 139 | 17 | 30 | 40 | 58 | 67 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150827 | 0 | 0 | 0 | 294 | 144 | 21 | 230 | 200 | 45 | 63 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150828 | 5 | 0 | 0 | 233 | 156 | 26 | 230 | 240 | 81 | 103 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150829 | 325 | 0 | 0 | 222 | 133 | 58 | 210 | 210 | 157 | 206 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150830 | 102 | 0 | 0 | 200 | 128 | 47 | 200 | 200 | 89 | 112 | -9999 | -9999 | 1 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150831 | 0 | 0 | 0 | 189 | 161 | 58 | 210 | 210 | 112 | 134 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
GHCND:USW00024233 | SEATTLE TACOMA INTERNATIONAL AIRPORT WA US | 20150901 | 58 | 0 | 0 | 194 | 139 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 | -9999 |
9.7.2 Preprocessing
The bike counts come in hourly data, but our weather information is daily. We’ll need to aggregate the hourly counts into daily totals before combining the datasets.
In the Python handbook, one does:
= counts.resample('d').sum() daily
Since Tablecloth’s time series features are still evolving, we’ll be a bit more explicit:
def daily-totals
(-> counts
(fn [{:keys [datetime]}]
(tc/group-by (:date (datetime/local-date-time->local-date
{
datetime)})):total :west :east]
(tc/aggregate-columns [ tcc/sum)))
daily-totals
_unnamed [4443 4]:
:date | :total | :west | :east |
---|---|---|---|
2012-10-02 | 1938.0 | 1165.0 | 773.0 |
2012-10-03 | 3521.0 | 1761.0 | 1760.0 |
2012-10-04 | 3475.0 | 1767.0 | 1708.0 |
2012-10-05 | 3148.0 | 1590.0 | 1558.0 |
2012-10-06 | 2006.0 | 926.0 | 1080.0 |
2012-10-07 | 2142.0 | 951.0 | 1191.0 |
2012-10-08 | 3537.0 | 1708.0 | 1829.0 |
2012-10-09 | 3501.0 | 1742.0 | 1759.0 |
2012-10-10 | 3235.0 | 1587.0 | 1648.0 |
2012-10-11 | 3047.0 | 1468.0 | 1579.0 |
… | … | … | … |
2024-11-20 | 2300.0 | 787.0 | 1513.0 |
2024-11-21 | 2382.0 | 775.0 | 1607.0 |
2024-11-22 | 1473.0 | 536.0 | 937.0 |
2024-11-23 | 1453.0 | 652.0 | 801.0 |
2024-11-24 | 727.0 | 311.0 | 416.0 |
2024-11-25 | 1483.0 | 486.0 | 997.0 |
2024-11-26 | 2173.0 | 727.0 | 1446.0 |
2024-11-27 | 1522.0 | 548.0 | 974.0 |
2024-11-28 | 631.0 | 261.0 | 370.0 |
2024-11-29 | 833.0 | 368.0 | 465.0 |
2024-11-30 | 1178.0 | 509.0 | 669.0 |
9.7.3 Prediction by day-of-week
Next, we’ll explore a simple regression by day of week.
def days-of-week
(:Mon :Tue :Wed :Thu :Fri :Sat :Sun]) [
We’ll convert the numeric day-of-week to the corresponding keyword:
def idx->day-of-week
(comp days-of-week dec)) (
For example,
1) (idx->day-of-week
:Mon
7) (idx->day-of-week
:Sun
Now, let’s build our dataset:
def totals-with-day-of-week
(-> daily-totals
(:day-of-week
(tc/add-column fn [ds]
(map idx->day-of-week
(
(datetime/long-temporal-field:day-of-week
:date ds)))))
(:total :day-of-week]))) (tc/select-columns [
totals-with-day-of-week
_unnamed [4443 2]:
:total | :day-of-week |
---|---|
1938.0 | :Tue |
3521.0 | :Wed |
3475.0 | :Thu |
3148.0 | :Fri |
2006.0 | :Sat |
2142.0 | :Sun |
3537.0 | :Mon |
3501.0 | :Tue |
3235.0 | :Wed |
3047.0 | :Thu |
… | … |
2300.0 | :Wed |
2382.0 | :Thu |
1473.0 | :Fri |
1453.0 | :Sat |
727.0 | :Sun |
1483.0 | :Mon |
2173.0 | :Tue |
1522.0 | :Wed |
631.0 | :Thu |
833.0 | :Fri |
1178.0 | :Sat |
def totals-with-one-hot-days-of-week
(-> (reduce (fn [dataset day-of-week]
(-> dataset
(
(tc/add-column day-of-week-> (:day-of-week %)
#(
(tcc/eq day-of-week);; convert booleans to 0/1
1)))))
(tcc/*
totals-with-day-of-week
days-of-week):day-of-week])
(tc/drop-columns [:total))) (ds-mod/set-inference-target
-> totals-with-one-hot-days-of-week
( (tc/select-columns ds-mod/inference-column?))
_unnamed [0 0]
Since the binary columns sum to 1, they’re collinear, and we won’t use an intercept. This way, each coefficient directly reflects the expected bike count for that day of week.
def days-of-week-model
(
(lm totals-with-one-hot-days-of-week:intercept? false})) {
Let’s take a look at the results:
-> days-of-week-model
(println
with-out-str
kind/code)
Residuals:
:min | :q1 | :median | :q3 | :max |
|
|--------------+-------------+------------+------------+-------------|3034.944882 | -841.755906 | -90.179528 | 828.820472 | 3745.244094 |
| -
Coefficients:
:name | :estimate | :stderr | :t-value | :p-value | :confidence-interval |
|
|-------+-------------+-----------+-----------+----------+---------------------------|:Mon | 2785.741325 | 44.521658 | 62.570476 | 0.0 | [2698.456664 2873.025986] |
| :Tue | 3115.527559 | 44.486587 | 70.032964 | 0.0 | [3028.311653 3202.743465] |
| :Wed | 3109.944882 | 44.486587 | 69.907472 | 0.0 | [3022.728976 3197.160788] |
| :Thu | 2961.179528 | 44.486587 | 66.563423 | 0.0 | [2873.963622 3048.395433] |
| :Fri | 2623.755906 | 44.486587 | 58.978583 | 0.0 | [2536.54 2710.971811] |
| :Sat | 1688.029921 | 44.486587 | 37.944693 | 0.0 | [1600.814015 1775.245827] |
| :Sun | 1576.178233 | 44.521658 | 35.402506 | 0.0 | [1488.893572 1663.462894] |
|
3472.719277560978 on degrees of freedom: {:residual 4436, :model 7, :intercept 0}
F-statistic: 0.0
p-value:
0.8456776967289252
R2: 0.8454341764126724
Adjusted R2: 1121.0266948292917 on 4436 degrees of freedom
Residual standard error: 75015.17638480052
AIC:
We can clearly see weekend versus weekday differences.