3 Test generation

(experimental 🛠)

Using Kindly annotations, Clay can automatically generate tests out of notebooks.

This allows one to make sure that a piece of documentation or tutorial remains correct after code changes, etc. This can also be seen as a literate way to create tests for a library.

3.1 The idea

Sometimes, we have some expectations regargings the forms we have in our notebook. For example, we know that (rand) should result in a number between zero and one, and we know that (filter pos? (range -4 4)) should result in a nonempty sequence.

We can express such expectations by adding Kindly-annotated forms following the forms of interest. Then, every time the notebook is rendered, Clay will generate tests to verify our assumptions. The tests will be collected in a regular clojure.test namespace with standard deftest forms as one may expect.

3.2 Expressing a test

Assume we have a couple of expressions in our notebook:

(def x 11)

(* x x)

If we think about it, we know the resulting number has to be more than 100. Thus, we may add a test for that by taking the result of the last form, infoking the > function with the additional argument 100, and making sure the result is truthy.

This can be expressed in any one of a few equivalent ways:

(kind/test-last [> 100])
^kind/test-last [> 100]
(kindly/check > 100)

In the rendered namespace, such forms will be hidden. However, as a side effect of rendering, we will get a test namespace with something like the following:

(def v1_l4 (def x 11))
(def v3_l8 (* x x))
(deftest t4_l10 (is (> v3_l8 100)))

Here, the forms of the notebook are added as def forms, except fot the test itself which is added as a deftest.

3.3 The different test modes

Clay currently supports two modes for test generation: :sequential (which is the default) and :simple.

The idea of :sequential test generation is that, in general, the original notebook may evolve a certain state sequentially. E.g., it may define some vars and mutate some atoms. In general, the correctness of the tests may rely on this state, so the test namespace has to go through all forms of the original notebook one by one, and interlace the deftest forms between them.

The idea of :simple test generation is that sometimes, the situation is simpler. The tests can be standalone invocations of certain library functions, so their correctness wuold not rely on anything else in the namespace. This allows us to write tests which are more readable. If all tests are simple, it makes the whole test namespace much simpler.

The testing mode can be specified in the :kind/options of the project-level configuration or the namespace level configuration. It can also be specified for a specific test. We’ll see that in the examples below.

3.4 Examples

3.4.1 Sequential tests

Here is a namespace with a few sequential tests, that depend on the state evolving throughout the notebook.

sequential.clj

(ns test-gen.sequential
  (:require [scicloj.kindly.v4.kind :as kind]
            [scicloj.kindly.v4.api :as kindly]))

(def x 9)

(def *a (atom 0))

(+ x (swap! *a inc))

;; Express a test by the
;; `kind/test-last` function:

(kind/test-last [= 10])

(+ x (swap! *a inc))

;; Express a test by the 
;; `^kind/test-last` metadata:

^kind/test-last [= 11]

(+ x (swap! *a inc))

;; Express a test by the 
;; `kindky/check` macro:

(kindly/check = 12)

sequential_generated_test.clj

(ns
 test-gen.sequential-generated-test
 (:require
  [scicloj.kindly.v4.kind :as kind]
  [scicloj.kindly.v4.api :as kindly]
  [clojure.test :refer [deftest is]]))


(def v1_l5 (def x 9))


(def v2_l7 (def *a (atom 0)))


(def v3_l9 (+ x (swap! *a inc)))


(deftest t5_l14 (is (= v3_l9 10)))


(def v6_l16 (+ x (swap! *a inc)))


(deftest t8_l21 (is (= v6_l16 11)))


(def v9_l23 (+ x (swap! *a inc)))


(deftest t11_l28 (is (= v9_l23 12)))

3.4.2 Mixed sequential and simple tests

Here is a namespace that involves both sequential and simple tests. This is expressed through :test-mode :simple in the kindly options part of the call to kind/test-last.

sequential_and_simple.clj

(ns test-gen.sequential-and-simple
  (:require [scicloj.kindly.v4.kind :as kind]))

(def x 9)

;; A sequential test:

(+ x 11)

(kind/test-last [= 20])

;; A simple test:

(+ 4 5)

(kind/test-last
 [= 9]
 {:test-mode :simple})

sequential_and_simple_generated_test.clj

(ns
 test-gen.sequential-and-simple-generated-test
 (:require
  [scicloj.kindly.v4.kind :as kind]
  [clojure.test :refer [deftest is]]))


(def v1_l4 (def x 9))


(def v3_l8 (+ x 11))


(deftest t4_l10 (is (= v3_l8 20)))


(def v6_l14 (+ 4 5))


(deftest t7_l16 (is (= (+ 4 5) 9)))

3.4.3 Only simple tests

Here is a namespace where all tests are simple. This can be expressed through :test-mode :simple, which can be specified either in the namespace-level :kindly/options (as we do here), or in each and every kind/test-last call.

simple.clj

^{:kindly/options {:test-mode :simple}}
(ns test-gen.simple
  (:require [scicloj.kindly.v4.kind :as kind]))

;; A simple test:

(+ 4 5)

(kind/test-last
 [= 9])

simple_generated_test.clj

(ns
 test-gen.simple-generated-test
 (:require
  [scicloj.kindly.v4.kind :as kind]
  [clojure.test :refer [deftest is]]))


(deftest t3_l9 (is (= (+ 4 5) 9)))

3.5 More examples

For a detailed example using this mechanism, see the source of the ClojisR tutorial.

source: notebooks/clay_book/test_generation.clj