Adds informations about observations to a dataset

Potential row names are these: https://raw.githubusercontent.com/scicloj/metamorph.ml/main/resources/columms-augment.edn

No other row names should be used.

Each model will only return a small subset of possible rows.

A model might not implement this function, and then the dataset is returned unchanged.

Given a datset which must have exactly 1 inference target column return a default loss fn. If column is categorical, loss is tech.v3.ml.loss/classification-loss, else the loss is tech.v3.ml.loss/mae (mean average error).

Create a model definition. An ml model is a function that takes a dataset and an options map and returns a model. A model is something that, combined with a dataset, produces a inferred dataset.

Atom controlling strict prediction validation behavior.

When set to true (via reset! or swap!), enables validation that throws exceptions during prediction if:

• Target categorical maps don’t match between training and prediction datasets
• Predicted values are not present in the prediction categorical map

Defaults to false for backward compatibility. Set to true to catch potential prediction inconsistencies early.

Example: (reset! enable-strict-prediction-validations true)

Creates an ensemble pipeline from multiple pipelines using majority voting.

pipes - Sequence of metamorph pipeline functions

Returns a single metamorph pipeline function that trains all sub-pipelines in :fit mode and combines their predictions via majority voting in :transform mode. Each pipeline is trained independently on the same data.

In :fit mode, stores all fitted pipeline contexts. In :transform mode, runs predictions from all pipelines and selects the most common prediction for each observation.

The ensemble pipeline can be used anywhere a regular pipeline is accepted (e.g., in evaluate-pipelines).

See also: scicloj.metamorph.ml/evaluate-pipelines

Evaluates the performance of a seq of metamorph pipelines, which are suposed to have a model as last step under key :model, which behaves correctly in mode :fit and :transform. The function scicloj.metamorph.ml/model is such function behaving correctly.

This function calculates the accuracy or loss, given as metric-fn of each pipeline in pipeline-fn-seq using all the train-test splits given in train-test-split-seq.

It runs the pipelines in mode :fit and in mode :transform for each pipeline-fn in pipe-fn-seq for each split in train-test-split-seq.

The function returns a seq of seqs of evaluation results per pipe-fn per train-test split. Each of the evaluation results is a context map, which is specified in the malli schema attached to this function.

• pipeline-fn-or-decl-seq need to be sequence of pipeline functions or pipline declarations which follow the metamorph approach. These type of functions get produced typically by calling scicloj.metamorph/pipeline. Documentation is here:

• train-test-split-seq need to be a sequence of maps containing the train and test dataset (being tech.ml.dataset) at keys :train and :test. tablecloth.api/split->seq produces such splits. Supervised models require both keys (:train and :test), while unsupervised models only use :train

• metric-fn Metric function to use. Typically comming from tech.v3.ml.loss. For supervised models the metric-fn receives the trueth and predicted values and should return a single double number. The metric fns receives a a seq without categorical maps. These get reverse-applied to the prediction , if present, before passing the values to the metriic fn. For unsupervised models the function receives the fitted ctx and should return a singel double number as well. This metric will be used to sort and eventualy filter the result, depending on the options (:return-best-pipeline-only and :return-best-crossvalidation-only). The notion of best comes from metric-fn combined with loss-and-accuracy

• loss-or-accuracy If the metric-fn is a loss or accuracy calculation. Can be :loss or :accuracy. Decided the notion of best model. In case of :loss pipelines with lower metric are better, in case of :accuracy pipelines with higher value are better.

• options map controls some mainly performance related parameters. This function can potentialy result in a large ammount of data, able to bring the JVM into out-of-memory. We can control memory consumption / paralellism by the below options. The defaults are quite aggresive in removing details, and this can be tweaked further into more or less details via:

  • :return-best-pipeline-only - Only return information of the best performing pipeline. Default is true.
  • :return-best-crossvalidation-only - Only return information of the best crossvalidation (per pipeline returned). Default is true.
  • :map-fn - Controls parallelism, so decides if we use map , pmap , ppmap or mapv to map over different pipelines. Default is :map Giving :run!, :prun!, pprun! executes the pipelines based on run!. In this case this function returns nothing, and it assumes that a site-effect making :evaluation-handler-fn is given. (which can then return nil)
  • :evaluation-handler-fn - Gets called once with the complete result of an individual pipeline evaluation. It can be used to adapt the data returned for each evaluation and / or to make side effects using the evaluation data. The result of this function is taken as evaluation result. It need to contain as a minumum this 2 key paths:
    • [:train-transform :metric]
    • [:test-transform :metric]

    All other evalution data can be removed, if desired. The evaluation-handler-fn fn can as well return nil, and then it should be used together with map-fn :run!, :prun! or pprun!, so we execute it for side-efects only, which means as well that memory consumption is minimal

    It can be used for side effects, like experiment tracking on disk or in a database. The passed in evaluation result is a map with all information on the current evaluation, including the datasets used.

    The default handler function is: scicloj.metamorph.ml/default-result-dissoc--in-fn which removes the often large model object and the training data.

    identity can be use to retain all evaluation data incl. data

    scicloj.metamorph.ml/result-dissoc-in-seq--all is availble and reduces even more agressively, just keeping the metrices.

  • :other-metrics Specifies other metrices to be calculated during evaluation

This function expects as well the ground truth of the target variable into a specific key in the context at key :model :scicloj.metamorph.ml/target-ds See here for the simplest way to set this up: https://github.com/behrica/metamorph.ml/blob/main/README.md The function scicloj.ml.metamorph/model does this correctly.

Explain (if possible) an ml model. A model explanation is a model-specific map of data that usually indicates some level of mapping between features and importance

Gives a glance on the model, returning a dataset with model information about the entire model.

Potential row names are these: https://raw.githubusercontent.com/scicloj/metamorph.ml/main/resources/columms-glance.edn

No other row names should be used.

Each model will only return a small subset of possible rows.

The list of allowed row names might change over time.

A model might not implement this function, and then an empty dataset will be returned.

Retrieves the hyperparameters definition for a model type.

model-kwd - Keyword identifying the model type (e.g., :smile.classification/random-forest)

Returns the hyperparameters map specified during model registration, or nil if no hyperparameters were defined. Hyperparameters describe tunable options for the model.

Used for introspection and hyperparameter tuning/grid search.

See also: define-model!, options->model-def

Calculates the log-likelihood for the given model and predictions.

• model - Trained model map containing :options with model definition
• y - Actual target values (ground truth)
• yhat - Predicted values from the model

Returns the log-likelihood value by calling the model’s :loglik-fn function. The specific log-likelihood function used depends on the model type.

See also: scicloj.metamorph.ml/tidy, scicloj.metamorph.ml/glance

Executes a machine learning model in train/predict (depending on :mode) from the metamorph.ml model registry.

The model is passed between both invocation via the shared context ctx in a key (a step indentifier) which is passed in key :metamorph/id and guarantied to be unique for each pipeline step.

The function writes and reads into this common context key.

Options: - :model-type - Keyword for the model to use

Further options get passed to train functions and are model specific.

See here for an overview for the models build into scicloj.ml:

https://scicloj.github.io/scicloj.ml-tutorials/userguide-models.html

Other libraries might contribute other models, which are documented as part of the library.

See as well:

• scicloj.metamorph.ml/train
• scicloj.metamorph.ml/predict

Returns a list of all registered model definition names.

Returns a sequence of keywords representing all model types that have been registered via define-model!. These can be used as the :model-type value when training models.

Example: [:metamorph.ml/dummy-classifier ...]

See also: define-model!, options->model-def

Map of model kwd to model definition

Finds optimal hyperparameters in a machine learning pipeline using cross validation.

It evaluates the performance of a seq of metamorph pipelines, which are suposed to have a model as last step under key :model, which behaves correctly in mode :fit and :transform. The function scicloj.metamorph.ml/model is such function behaving correctly.

This function calculates the accuracy or loss, given as metric-def of each pipeline in pipeline-fn-seq using all the train-test splits given in train-test-split-seq.

It runs the pipelines in mode :fit and in mode :transform for each pipeline-fn in pipe-fn-seq for each split in train-test-split-seq.

The function returns a seq of seqs of evaluation results per pipe-fn per train-test split. Each of the evaluation results is a context map, which is specified in the malli schema attached to this function.

• pipeline-fn-or-decl-seq need to be sequence of pipeline functions or pipline declarations which follow the metamorph approach. These type of functions get produced typically by calling scicloj.metamorph/pipeline. Each item in this list represents one hyper-pararmeter setting. scicloj.metamorph.ml.gridsearch/sobol-gridsearch can be used to generate model options vrainats from a search grid definition. The different pipelines can contain one model with several options, or multiple models with differentg options or even different preprocessing instructions per pipeline.

• train-test-split-seq need to be a sequence of maps containing the train and test dataset (being tech.ml.dataset) at keys :train and :test. tablecloth.api/split->seq produces such splits. Supervised models require both keys (:train and :test), while unsupervised models only use :train The nested loop over pipeline-fn-or-decl-seq and train-test-split-seq decides how many evaluations are run, and restults are returned in detailed and aggregated form.

• metric-def Metric definition to use. It’s a map with keys * :metric - a keyword specifying the metric fn to use, see scicloj.metamorph.ml.column-metric/classification-metric and scicloj.metamorph.ml.column-metric/regression-metric for available metric functions * :averaging - :macro or :micro , decides how to average over the results of the usualy binary classification functions * :loss-or-accuracy :accuracy or :loss , if teh fn is caluclating accuracy (higer values are better), or loss (lower values are better) * :options optional options for the metric fn

  This metric will be used to sort and eventualy filter the result, depending on the options (:return-best-pipeline-only and :return-best-crossvalidation-only). The notion of best comes from metric-fn combined with loss-and-accuracy

  • options map controls some mainly performance related parameters. This function can potentialy result in a large ammount of data, able to bring the JVM into out-of-memory. We can control memory consumption / paralellism by the below options.

  The defaults are quite aggresive in removing details, and this can be tweaked further into more or less details via:

  • :return-best-pipeline-only - Only return information of the best performing pipeline. Default is true.
  • :return-best-crossvalidation-only - Only return information of the best crossvalidation (per pipeline returned). Default is true.
  • :map-fn - Controls parallelism, so decides if we use map , pmap , ppmap or mapv to map over different pipelines. Default is :map Giving :run!, :prun!, pprun! executes the pipelines based on run!. In this case this function returns nothing, and it assumes that a site-effect making :evaluation-handler-fn is given. (which can then return nil)
  • :evaluation-handler-fn - Gets called once with the complete result of an individual pipeline evaluation. It can be used to adapt the data returned for each evaluation and / or to make side effects using the evaluatio data. The result of this function is taken as evaluation result. It need to contain as a minumum this 2 key paths: :train-transform :metric :test-transform :metric All other evalution data can be removed, if desired. The evaluation-handler-fn fn can as well return nil, and then it should be used together with map-fn :run!, :prun! or pprun!, so we execute it for side-efects only, which means as well that memory consumption is minimal

    It can be used for side effects, like experiment tracking on disk or in a database. The passed in evaluation result is a map with all information on the current evaluation, including the datasets used.

    The default handler function is: scicloj.metamorph.ml/default-result-dissoc--in-fn which removes the often large model object and the training data. identity can be use to retain all evaluation data incl. data scicloj.metamorph.ml/result-dissoc-in-seq--all is availble and reduces even more agressively, just keeping the metrices.

  • :other-metrics Specifies other metrices to be calculated during evaluation

  This function expects as well the ground truth of the target variable into a specific key in the context at key :model :scicloj.metamorph.ml/target-ds See here for the simplest way to set this up: https://github.com/behrica/metamorph.ml/blob/main/README.md The function scicloj.ml.metamorph/model does this correctly.

  (This function replaces the deprecated function evaluate-pipelines. It’s main difference is the changed handling of metric functions)

Retrieves the model definition corresponding to the :model-type option.

options - Map containing at minimum a :model-type keyword

Returns the model definition map registered for the given :model-type. Throws an exception if the model type is not found, suggesting a missing namespace require.

Used internally to look up train/predict functions and model metadata.

See also: define-model!, model-definition-names, hyperparameters

Predict returns a dataset with only the predictions in it.

• For regression, a single column dataset is returned with the column named after the target
• For classification, a dataset is returned with a float64 column for each target value and values that describe the probability distribution.

Each implementing model should construct its prediction in a shape expressed by

• :target-column
• :target-datatypes
• :target-categorical-maps

it is receiving.

Any implementing model need to behave symetric between the ‘datatype in the target columns of training data’ and the ’datatype of the prediction columns` A model can decide to not accept certain dataypes in the target columns of training data. (and fail with exception). But any model should try to minimize this and accept for categorical data:

• all numeric types ( :int32, :int64, :float32, :float64)
• string
• categorical maps

It NEED to be symetric, and return the same datatype in prediction as it receives in training:

• numeric in train -> same numeric in predict
• string in train -> string in predict
• categorical map in train -> equivalent categorical map in predict

ml/train passes the needed information of the train target column to the model implementaion to do this.

Thaws a frozen model for use in predictions.

Models returned from train may be ‘frozen’ (serialized) for storage efficiency. A ‘thaw’ operation deserializes the model for use. This happens automatically during predict, but you can manually thaw and cache the model under :thawed-model for faster repeated predictions on small datasets.

• model - Model map from train containing :model-data
• opts - Optional map with :thaw-fn to override the model’s thaw function

Returns the thawed model data ready for prediction. If already thawed and cached, returns the cached version.

See also: train, predict

summarizes information about model components. Returns a dataset with rows from this list: https://raw.githubusercontent.com/scicloj/metamorph.ml/main/resources/columms-tidy.edn

No other row names should be used.

Each model will only return a small subset of possible rows.

The list of allowed row names might change over time.

A model might not implement this function, and then an empty dataset will be returned.

Given a dataset and an options map produce a model. The model-type keyword in the options map selects which model definition to use to train the model. Returns a map containing at least:

• :model-data - the result of that definitions’s train-fn.
• :options - the options passed in.
• :id - new randomly generated UUID.
• :feature-columns - vector of column names.
• :target-columns - vector of column names.
• :target-datatypes - map of target columns names -> target columns type
• :target-categorical-maps - the categorical maps of the target columns, if present

A well behaving model implementaion should use

• :target-column
• ’:target-datatypes`
• :target-categorical-maps

to construct its prediction dataset so that its matches with the train data target column.

Controls , if train/predict invocations are cached. if ‘use-cache’ is true, the get-fn and set-fn functions ar called accorddngly