3 Test generation
(experimental 🛠)
Using Kindly annotations, Clay can automatically generate tests out of notebooks.
This allows one to make sure that a piece of documentation or tutorial remains correct after code changes, etc. This can also be seen as a literate way to create tests for a library.
3.1 The idea
Sometimes, we have some expectations regargings the forms we have in our notebook. For example, we know that (rand)
should result in a number between zero and one, and we know that (filter pos? (range -4 4))
should result in a nonempty sequence.
We can express such expectations by adding Kindly-annotated forms following the forms of interest. Then, every time the notebook is rendered, Clay will generate tests to verify our assumptions. The tests will be collected in a regular clojure.test namespace with standard deftest
forms as one may expect.
3.2 Expressing a test
Assume we have a couple of expressions in our notebook:
def x 11) (
* x x) (
121
If we think about it, we know the resulting number has to be more than 100. Thus, we may add a test for that by taking the result of the last form, infoking the >
function with the additional argument 100
, and making sure the result is truthy.
This can be expressed in any one of a few equivalent ways:
(kind/test-last [> 100])
^kind/test-last [> 100]
(kindly/check > 100)
In the rendered namespace, such forms will be hidden. However, as a side effect of rendering, we will get a test namespace with something like the following:
def v1_l4 (def x 11))
(def v3_l8 (* x x))
(deftest t4_l10 (is (> v3_l8 100))) (
Here, the forms of the notebook are added as def
forms, except fot the test itself which is added as a deftest
.
3.3 The different test modes
Clay currently supports two modes for test generation: :sequential
(which is the default) and :simple
.
The idea of :sequential
test generation is that, in general, the original notebook may evolve a certain state sequentially. E.g., it may define some vars and mutate some atoms. In general, the correctness of the tests may rely on this state, so the test namespace has to go through all forms of the original notebook one by one, and interlace the deftest
forms between them.
The idea of :simple
test generation is that sometimes, the situation is simpler. The tests can be standalone invocations of certain library functions, so their correctness wuold not rely on anything else in the namespace. This allows us to write tests which are more readable. If all tests are simple, it makes the whole test namespace much simpler.
The testing mode can be specified in the :kind/options
of the project-level configuration or the namespace level configuration. It can also be specified for a specific test. We’ll see that in the examples below.
3.4 Examples
3.4.1 Sequential tests
Here is a namespace with a few sequential tests, that depend on the state evolving throughout the notebook.
sequential.clj
ns test-gen.sequential
(:require [scicloj.kindly.v4.kind :as kind]
(:as kindly]))
[scicloj.kindly.v4.api
def x 9)
(
def *a (atom 0))
(
+ x (swap! *a inc))
(
;; Express a test by the
;; `kind/test-last` function:
= 10])
(kind/test-last [
+ x (swap! *a inc))
(
;; Express a test by the
;; `^kind/test-last` metadata:
= 11]
^kind/test-last [
+ x (swap! *a inc))
(
;; Express a test by the
;; `kindky/check` macro:
= 12) (kindly/check
sequential_generated_test.clj
ns
(
test-gen.sequential-generated-test:require
(:as kind]
[scicloj.kindly.v4.kind :as kindly]
[scicloj.kindly.v4.api test :refer [deftest is]]))
[clojure.
def v1_l5 (def x 9))
(
def v2_l7 (def *a (atom 0)))
(
def v3_l9 (+ x (swap! *a inc)))
(
deftest t5_l14 (is (= v3_l9 10)))
(
def v6_l16 (+ x (swap! *a inc)))
(
deftest t8_l21 (is (= v6_l16 11)))
(
def v9_l23 (+ x (swap! *a inc)))
(
deftest t11_l28 (is (= v9_l23 12))) (
3.4.2 Mixed sequential and simple tests
Here is a namespace that involves both sequential and simple tests. This is expressed through :test-mode :simple
in the kindly options part of the call to kind/test-last
.
sequential_and_simple.clj
ns test-gen.sequential-and-simple
(:require [scicloj.kindly.v4.kind :as kind]))
(
def x 9)
(
;; A sequential test:
+ x 11)
(
= 20])
(kind/test-last [
;; A simple test:
+ 4 5)
(
(kind/test-last= 9]
[:test-mode :simple}) {
sequential_and_simple_generated_test.clj
ns
(
test-gen.sequential-and-simple-generated-test:require
(:as kind]
[scicloj.kindly.v4.kind test :refer [deftest is]]))
[clojure.
def v1_l4 (def x 9))
(
def v3_l8 (+ x 11))
(
deftest t4_l10 (is (= v3_l8 20)))
(
def v6_l14 (+ 4 5))
(
deftest t7_l16 (is (= (+ 4 5) 9))) (
3.4.3 Only simple tests
Here is a namespace where all tests are simple. This can be expressed through :test-mode :simple
, which can be specified either in the namespace-level :kindly/options
(as we do here), or in each and every kind/test-last
call.
simple.clj
:kindly/options {:test-mode :simple}}
^{ns test-gen.simple
(:require [scicloj.kindly.v4.kind :as kind]))
(
;; A simple test:
+ 4 5)
(
(kind/test-last= 9]) [
simple_generated_test.clj
ns
(
test-gen.simple-generated-test:require
(:as kind]
[scicloj.kindly.v4.kind test :refer [deftest is]]))
[clojure.
deftest t3_l9 (is (= (+ 4 5) 9))) (
3.5 More examples
For a detailed example using this mechanism, see the source of the ClojisR tutorial.