This is part of the Scicloj Clojure Data Tutorials.

(comment
  (require '[scicloj.clay.v2.api :as clay])
  (clay/start!)
  (clay/make! {:source-path "notebooks/index.clj"
               :show false
               }))

The following code shows how to perform text classification from a Kaggle dataset and make a submission file, ready to get uploaded to Kaggle for scoring.

It makes use of the tidy text / TFIDF functionality present in metamorph.ml and the ability of the xgboost model to handle tidy text data as input.

First we need a fn to tokenize a line of text The simplest such function is:

(defn- tokenize-fn [text]
  (str/split text #" "))

#'index/tokenize-fn

It does not do any text normalization, which is always required in NLP tasks in order to have a more general model.

The following reads line-by-line a file from disk and converts it on the fly to the tidy text representation, it which each word is a row in a dataset.

line-parse-fn needs to split an input line into [text meta], and the text is then further handled by tokenize-fn and split into tokens. The format of the data has the text in field 4 and the label in 5. We ignore all other columns so far:

(defn- line-parse-fn [line]
  [(nth line 3)
   (Integer/parseInt (nth line 4))])

#'index/line-parse-fn

This triggers the parsing and produces a (seq of) “long” datasets (1 for our small text) and the vocabulary obtained during parsing.

(def tidy-train
  (text/->tidy-text (csv/read-csv (io/reader "train.csv"))
                    seq
                    line-parse-fn
                    tokenize-fn
                    :skip-lines 1))

(def tidy-train-ds 
  (-> tidy-train :datasets first))

The combination of columns :document, :token-pos and :token-index together with the vocabulary table is an exact representation of the text Unless we normalize it as part of hte tokenize-fn

meta is any other information of a row to be kept, usualy the “label” in case of training data.

tidy-train-ds

_unnamed [113650 4]:

:token-idx	:token-pos	:document	:meta
1	0	0	1
2	1	0	1
3	2	0	1
4	3	0	1
5	4	0	1
6	5	0	1
7	6	0	1
8	7	0	1
9	8	0	1
10	9	0	1
…	…	…	…
5529	2	7612	1
12372	3	7612	1
25359	4	7612	1
30	5	7612	1
2552	6	7612	1
44	7	7612	1
25361	8	7612	1
69	9	7612	1
11698	10	7612	1
3844	11	7612	1
32017	12	7612	1

The lookup table allow to convert from :token-idx to words and back if needed.

(def train--token-lookup-table (:token-lookup-table tidy-train))

(map str (take 20 train--token-lookup-table))

("attack.=>2828"
 "Ercjmnea:=>22642"
 "#failure.\n#annonymous=>23860"
 "concluded.=>27252"
 "criminal=>16677"
 "http://t.co/gzTolLl5Wo‰Û_=>21789"
 "FOR:=>31985"
 "http://t.co/qp6q8RS8ON=>30308"
 "#watch=>12433"
 "exercised=>27047"
 "http://t.co/zCKXtFc9PT=>7775"
 "HEAR=>7573"
 "@fa07af174a71408=>26138"
 "ended=>1477"
 "@Coach_Keith44=>27742"
 "http://t.co/Q0X7e84R4e=>26341"
 "http://t.co/dVONWIv3l1=>9946"
 "plummeting=>22166"
 "heated=>15565"
 "architect=>14412")

As we can see, the tokens are not cleaned / standardized at all. This gives as well a large vocabulary size of

(count train--token-lookup-table)

32018

Now we convert the text into a bag-of-words format, which looses any word order and calculates a metric which is known to work well for text classification, the so called TFIDF score.

(def train-tfidf
  (text/->tfidf tidy-train-ds))

The resulting table represent conceptually well three “sparse matrices” where :document and :token-idx are x,y coordinates and matrix cell values are :token-count, term-frequency (:tf) or TFIDF

Not present rows (the large majority) are 0 values. A subset of machine learning algorithms can deal with sparse matrices, without then need to convert them into dense matrices first, which is in most cases impossible due to the memory consumption The train-tfidf dataset represents therefore 3 sparse matrices with dimensions

(tcc/reduce-max (:document train-tfidf))

7612

times

(tcc/reduce-max (:token-idx train-tfidf))

32017

time 3

(* (tcc/reduce-max (:document train-tfidf))
   (tcc/reduce-max (:token-idx train-tfidf))
   3)

731140212

while only having shape:

(tc/shape train-tfidf)

[109209 6]

This is because most matrix elements are 0, as any text does “not contain” most words.

As TFIDF (and its variants) are one of the most common numeric representations for text, “sparse matrixes” and models supporting them is a prerequisite for NLP.

Only since a few years we have “dense text representations” based on “embeddings”, which will not be discussed here today, Now we get the data ready for training.

(def train-ds
  (-> train-tfidf
      (tc/rename-columns {:meta :label})
      (tc/select-columns [:document :token-idx :tfidf :label]) ;; we only need those
      (ds-mod/set-inference-target [:label])))

train-ds

_unnamed [109209 4]:

:document	:token-idx	:tfidf	:label
2048	1748	0.11541514	0
2048	188	0.06370677	0
2048	101	0.11088574	0
2048	24	0.03104085	0
2048	88	0.04050986	0
2048	437	0.06051489	0
2048	182	0.05415272	0
2048	5207	0.28932598	0
2048	3836	0.14466299	0
2048	4010	0.21482199	0
…	…	…	…
2046	11031	0.27329135	0
2047	11081	0.34044346	0
2047	11084	0.38815558	0
2047	11082	0.34044346	0
2047	11083	0.38815558	0
2047	6	0.06926274	0
2047	10964	0.31825858	0
2047	11079	0.38815558	0
2047	11080	0.38815558	0
2047	87	0.08122578	0
2047	190	0.11817181	0

(def n-sparse-columns (inc (tcc/reduce-max (train-ds :token-idx))))

The model used is from library scicloj.ml.xgboost which is the well known xgboost model behind a wrapper to make it work with tidy text data.

We use :tfidf column as the “feature”.

(require '[scicloj.ml.xgboost])

registers the mode under key :xgboost/classification

(def model
  (ml/train train-ds {:model-type :xgboost/classification
                         :sparse-column :tfidf
                         :seed 123
                         :num-class 2
                         :n-sparse-columns n-sparse-columns}))

Now we have a trained model, which we can use for prediction on the test data. This time we do parsing and tfidf in one go.

Important here:

We pass the vocabulary “obtained before” in order to be sure, that :token-idx maps to the same words in both datasets. In case of “new tokens”, we ignore them and map them to a special token, “[UNKNOWN]”

(def tfidf-test-ds
  (->
   (text/->tidy-text (csv/read-csv (io/reader "test.csv"))
                     seq
                     (fn [line]
                       [(nth line 3) {:id (first line)}])
                     tokenize-fn
                     :skip-lines 1
                     :new-token-behaviour :as-unknown
                     :token->index-map train--token-lookup-table)
   :datasets
   first
   text/->tfidf
   (tc/select-columns [:document :token-idx :tfidf :meta]) 
   ;; he :id for Kaggle
   (tc/add-column
    :id (fn [df] (map
                  #(:id %)
                  (:meta df))))
   (tc/drop-columns [:meta])))

This gives the dataset which can be passed into the predict function of metamorph.ml

tfidf-test-ds

_unnamed [39633 4]:

:document	:token-idx	:tfidf	:id
0	151	0.34977397	0
0	45	0.36523294	0
0	3824	0.48525953	0
0	152	0.34977397	0
0	778	0.45591098	0
0	56	0.12008759	0
1024	0	0.01644585	3358
1024	2467	0.39458153	3358
1024	77	0.41893619	3358
1024	12379	0.45591098	3358
…	…	…	…
1022	209	0.19479382	3353
1022	49	0.10779242	3353
1023	0	0.01096390	3354
1023	920	0.24262977	3354
1023	12387	0.26771560	3354
1023	8702	0.23455392	3354
1023	101	0.10735898	3354
1023	6064	0.25304133	3354
1023	12385	0.29280144	3354
1023	12388	0.20601872	3354
1023	12499	0.29280144	3354

(def prediction
  (ml/predict tfidf-test-ds model))

The raw predictions contain the “document” each prediction is about. This we can use to match predictions and the input “ids” in order to produce teh format required by Kaggle

prediction

:_unnamed [3263 4]:

0	1	:label	:document
0.53059942	0.46940058	0.0	0
0.62585568	0.37414426	0.0	1024
0.59270400	0.40729600	0.0	2048
0.63987494	0.36012506	0.0	1
0.61812049	0.38187948	0.0	2049
0.51093853	0.48906144	0.0	2
0.11743236	0.88256764	1.0	2050
0.62585568	0.37414426	0.0	3
0.22072305	0.77927697	1.0	4
0.73616028	0.26383975	0.0	2051
…	…	…	…
0.58601689	0.41398314	0.0	1013
0.53192580	0.46807417	0.0	1014
0.55812025	0.44187969	0.0	1015
0.62585568	0.37414426	0.0	1016
0.82627547	0.17372458	0.0	1017
0.62585568	0.37414426	0.0	1018
0.67966670	0.32033330	0.0	1019
0.80898130	0.19101864	0.0	1020
0.83320946	0.16679053	0.0	1021
0.56721741	0.43278253	0.0	1022
0.69006103	0.30993894	0.0	1023

(->
 (tc/right-join prediction tfidf-test-ds :document)
 (tc/unique-by [:id :label])
 (tc/select-columns [:id :label])
 (tc/update-columns {:label (partial map int)})
 (tc/rename-columns {:label :target})
 (tc/write-csv! "submission.csv"))

3264

The produced CVS file can be uploaded to Kaggle for scoring.

(->>
 (io/reader "submission.csv")
 line-seq
 (take 10))

("id,target"
 "0,0"
 "3358,0"
 "6887,0"
 "2,0"
 "6888,0"
 "3,0"
 "6889,1"
 "9,0"
 "11,1")

source: projects/ml/text-classification/notebooks/index.clj