15 API Reference

Last modified: 2026-02-08

Setup

A few preparations for the code examples below:

(require '[scicloj.pocket :as pocket])

(def cache-dir "/tmp/pocket-demo-reference")

(pocket/set-base-cache-dir! cache-dir)

OUT

10:06:46.939 INFO scicloj.pocket - Cache dir set to: /tmp/pocket-demo-reference

"/tmp/pocket-demo-reference"

(pocket/cleanup!)

OUT

10:06:46.939 INFO scicloj.pocket - Cache cleanup: /tmp/pocket-demo-reference

{:dir "/tmp/pocket-demo-reference", :existed false}

(defn expensive-calculation
  "Simulates an expensive computation"
  [x y]
  (println (str "Computing " x " + " y " (this is expensive!)"))
  (Thread/sleep 400)
  (+ x y))

Reference

`base-cache-dir`

Base directory for cache storage.

Resolved with precedence: binding > set-base-cache-dir! > POCKET_BASE_CACHE_DIR env var > pocket.edn :base-cache-dir > pocket-defaults.edn (library default: .cache/pocket).

The current value:

pocket/*base-cache-dir*

"/tmp/pocket-demo-reference"

`set-base-cache-dir!`

[dir]

Set the base cache directory by altering *base-cache-dir*. Returns the directory path.

(pocket/set-base-cache-dir! "/tmp/pocket-demo-2")

OUT

10:06:46.942 INFO scicloj.pocket - Cache dir set to: /tmp/pocket-demo-2

"/tmp/pocket-demo-2"

pocket/*base-cache-dir*

"/tmp/pocket-demo-2"

Restore it for the rest of the notebook:

(pocket/set-base-cache-dir! cache-dir)

OUT

10:06:46.942 INFO scicloj.pocket - Cache dir set to: /tmp/pocket-demo-reference

"/tmp/pocket-demo-reference"

`config`

[]

Return the effective resolved configuration as a map. Useful for inspecting which cache directory, mem-cache policy, storage policy, and filename length limit are in effect after applying the precedence chain.

Inspect the current effective configuration:

(pocket/config)

{:base-cache-dir "/tmp/pocket-demo-reference",
 :mem-cache {:policy :lru, :threshold 256},
 :storage :mem+disk,
 :filename-length-limit 240}

`cached`

[func & args]

Create a cached computation (returns IDeref).

The computation is executed on first deref and cached to disk. Subsequent derefs load from cache if available.

func must be a var (e.g., #'my-fn) or keyword (e.g., :train) for stable cache keys. Keywords are useful for extracting from cached maps: (cached :train split-c).

Storage policy is controlled by *storage* (see set-storage!). Use caching-fn with an opts map for per-function overrides.

cached returns a Cached object — the computation is not yet executed:

(def my-result (pocket/cached #'expensive-calculation 100 200))

(type my-result)

scicloj.pocket.impl.cache.Cached

The computation runs when we deref:

(deref my-result)

OUT

10:06:46.945 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 100 + 200 (this is expensive!)

Derefing again loads from cache (no recomputation):

(deref my-result)

`caching-fn`

[f]

[f opts]

Wrap a function to automatically cache its results.

Returns a new function where each call returns a Cached object (IDeref). Deref the result to trigger computation or load from cache. f must be a var (e.g., #'my-fn) or keyword for stable cache keys.

Optionally accepts an options map to override configuration per-function:

:storage — :mem+disk, :mem, or :none (overrides *storage*)
:cache-dir — base cache directory (overrides *base-cache-dir*)
:mem-cache — in-memory cache options (overrides *mem-cache-options*)
:filename-length-limit — max filename length before SHA-1 fallback (overrides *filename-length-limit*)

caching-fn wraps a function so that every call returns a Cached object:

(def my-caching-fn (pocket/caching-fn #'expensive-calculation))

(deref (my-caching-fn 3 4))

OUT

10:06:47.364 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 3 + 4 (this is expensive!)

Same args hit the cache:

(deref (my-caching-fn 3 4))

caching-fn accepts an optional map to override per-function configuration:

(pocket/caching-fn #'f {:storage :mem})     ;; in-memory only
(pocket/caching-fn #'f {:storage :none})    ;; identity tracking only
(pocket/caching-fn #'f {:cache-dir "/tmp/alt"})  ;; alternate cache dir

See the Configuration chapter for details on storage modes and the full option map.

`maybe-deref`

[x]

Deref if x implements IDeref, otherwise return x as-is.

Useful in pipeline functions that may receive either Cached or plain values.

A plain value passes through unchanged:

(pocket/maybe-deref 42)

A Cached value gets derefed:

(pocket/maybe-deref (pocket/cached #'expensive-calculation 100 200))

`->id`

[x]

Return a cache key representation of a value. Dispatches via the PIdentifiable protocol.

For derefed Cached values, returns the same lightweight identity as the original Cached reference — the origin registry preserves the link automatically (see cache_keys notebook for details).

A var’s identity is its fully-qualified name:

(pocket/->id #'expensive-calculation)

pocket-book.api-reference/expensive-calculation

A map’s identity is itself (maps are deep-sorted later for stable cache paths):

(pocket/->id {:b 2 :a 1})

{:b 2, :a 1}

A Cached object’s identity captures the full computation — function name and argument identities — without running it:

(pocket/->id (pocket/cached #'expensive-calculation 100 200))

(pocket-book.api-reference/expensive-calculation 100 200)

nil is handled as well:

(pocket/->id nil)

nil

A derefed Cached value carries its origin identity (for maps, vectors, sets, and datasets):

(defn make-config [x y] {:x x :y y})

(let [c (pocket/cached #'make-config 100 200)]
  (= (pocket/->id (deref c)) (pocket/->id c)))

OUT

10:06:47.778 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/make-config

true

`set-mem-cache-options!`

[opts]

Configure the in-memory cache. Resets it, discarding any currently cached values.

Supported keys:

:policy — :lru, :fifo, :lu, :ttl, :lirs, :soft, or :basic
:threshold — max entries for :lru, :fifo, :lu
:ttl — time-to-live in ms for :ttl policy
:s-history-limit / :q-history-limit — for :lirs policy

Defaults come from pocket-defaults.edn.

Switch to a FIFO policy with 100 entries:

(pocket/set-mem-cache-options! {:policy :fifo :threshold 100})

OUT

10:06:47.781 INFO scicloj.pocket - Mem-cache options set: {:policy :fifo, :threshold 100}
10:06:47.781 INFO scicloj.pocket.impl.cache - Mem-cache reconfigured: {:policy :fifo, :threshold 100}

{:policy :fifo, :threshold 100}

Reset to default:

(pocket/set-mem-cache-options! {:policy :lru :threshold 256})

OUT

10:06:47.782 INFO scicloj.pocket - Mem-cache options set: {:policy :lru, :threshold 256}
10:06:47.782 INFO scicloj.pocket.impl.cache - Mem-cache reconfigured: {:policy :lru, :threshold 256}

{:policy :lru, :threshold 256}

`reset-mem-cache-options!`

[]

Reset the in-memory cache configuration to library defaults. Clears any options set by set-mem-cache-options! and reconfigures the mem-cache with the default policy from pocket-defaults.edn. Returns the default options.

Reset mem-cache configuration to library defaults:

(pocket/reset-mem-cache-options!)

OUT

10:06:47.783 INFO scicloj.pocket - Mem-cache options reset to defaults: {:policy :lru, :threshold 256}
10:06:47.783 INFO scicloj.pocket.impl.cache - Mem-cache reconfigured: {:policy :lru, :threshold 256}

{:policy :lru, :threshold 256}

`storage`

Storage policy for cached computations: :mem+disk, :mem, or :none.

:mem+disk (default) — in-memory cache backed by disk persistence
:mem — in-memory cache only, no disk I/O
:none — no shared cache; instance-local memoization only

Resolved with precedence: binding > set-storage! > POCKET_STORAGE env var > pocket.edn :storage > pocket-defaults.edn (library default: :mem+disk).

`set-storage!`

[storage]

Set the storage policy by altering *storage*. Valid values: :mem+disk, :mem, :none. Returns the storage policy.

Switch to memory-only storage:

(pocket/set-storage! :mem)

OUT

10:06:47.784 INFO scicloj.pocket - Storage policy set to: :mem

:mem

Reset to default:

(pocket/set-storage! nil)

OUT

10:06:47.785 INFO scicloj.pocket - Storage policy set to: nil

nil

`cleanup!`

[]

Delete the cache directory, removing all cached values. Also clears the in-memory cache. Returns a map with :dir and :existed indicating what happened.

(pocket/cleanup!)

OUT

10:06:47.787 INFO scicloj.pocket - Cache cleanup: /tmp/pocket-demo-reference

{:dir "/tmp/pocket-demo-reference", :existed true}

`clear-mem-cache!`

[]

Clear all entries from the in-memory cache and the origin registry, without deleting the disk cache. The next deref of a cached value will reload from disk if available and re-register origin identity. Useful for testing scenarios that need to simulate memory eviction.

Clear in-memory cache without touching disk:

(deref (pocket/cached #'expensive-calculation 10 20))

OUT

10:06:47.788 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 10 + 20 (this is expensive!)

(pocket/clear-mem-cache!)

nil

`invalidate!`

[func & args]

Invalidate a specific cached computation, removing it from both disk and memory. Takes the same arguments as cached: a function var (or keyword) and its arguments. Returns a map with :path and :existed.

Remove a specific cached entry:

(deref (pocket/cached #'expensive-calculation 10 20))

(pocket/invalidate! #'expensive-calculation 10 20)

OUT

10:06:48.194 INFO scicloj.pocket.impl.cache - Invalidated: /tmp/pocket-demo-reference/a3/(pocket-book.api-reference_expensive-calculation 10 20) existed= true

{:path
 "/tmp/pocket-demo-reference/a3/(pocket-book.api-reference_expensive-calculation 10 20)",
 :existed true}

Derefing again will recompute:

(deref (pocket/cached #'expensive-calculation 10 20))

OUT

10:06:48.195 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 10 + 20 (this is expensive!)

`invalidate-fn!`

[func]

Invalidate all cached entries for a given function var (or keyword), regardless of arguments. Removes matching entries from both disk and memory. Returns a map with :fn-name, :count, and :paths.

Cache a few entries, then invalidate them all:

(deref (pocket/cached #'expensive-calculation 1 2))

OUT

10:06:48.599 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 1 + 2 (this is expensive!)

(deref (pocket/cached #'expensive-calculation 3 4))

OUT

10:06:49.003 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 3 + 4 (this is expensive!)

(pocket/invalidate-fn! #'expensive-calculation)

OUT

10:06:49.407 INFO scicloj.pocket.impl.cache - Invalidated 3 entries for pocket-book.api-reference/expensive-calculation

{:fn-name "pocket-book.api-reference/expensive-calculation",
 :count 3,
 :paths
 ["/tmp/pocket-demo-reference/a3/(pocket-book.api-reference_expensive-calculation 10 20)"
  "/tmp/pocket-demo-reference/d2/(pocket-book.api-reference_expensive-calculation 1 2)"
  "/tmp/pocket-demo-reference/49/(pocket-book.api-reference_expensive-calculation 3 4)"]}

(pocket/cleanup!)

OUT

10:06:49.408 INFO scicloj.pocket - Cache cleanup: /tmp/pocket-demo-reference

{:dir "/tmp/pocket-demo-reference", :existed true}

`cache-entries`

[]

[fn-name]

Scan the cache directory and return a sequence of metadata maps. Each entry contains :path, :id, :fn-name, :args-str, and :created-at (when metadata is available — entries cached before metadata support will only have :path). Optionally filter by function name.

List all cached entries:

(deref (pocket/cached #'expensive-calculation 10 20))

OUT

10:06:49.410 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 10 + 20 (this is expensive!)

(deref (pocket/cached #'expensive-calculation 3 4))

OUT

10:06:49.813 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/expensive-calculation
Computing 3 + 4 (this is expensive!)

(pocket/cache-entries)

[{:path
  "/tmp/pocket-demo-reference/a3/(pocket-book.api-reference_expensive-calculation 10 20)",
  :id "(pocket-book.api-reference/expensive-calculation 10 20)",
  :fn-name "pocket-book.api-reference/expensive-calculation",
  :args-str "[10 20]",
  :created-at "2026-02-09T08:06:49.810916122Z"}
 {:path
  "/tmp/pocket-demo-reference/49/(pocket-book.api-reference_expensive-calculation 3 4)",
  :id "(pocket-book.api-reference/expensive-calculation 3 4)",
  :fn-name "pocket-book.api-reference/expensive-calculation",
  :args-str "[3 4]",
  :created-at "2026-02-09T08:06:50.214225562Z"}]

Filter by function name:

(pocket/cache-entries "pocket-book.api-reference/expensive-calculation")

[{:path
  "/tmp/pocket-demo-reference/a3/(pocket-book.api-reference_expensive-calculation 10 20)",
  :id "(pocket-book.api-reference/expensive-calculation 10 20)",
  :fn-name "pocket-book.api-reference/expensive-calculation",
  :args-str "[10 20]",
  :created-at "2026-02-09T08:06:49.810916122Z"}
 {:path
  "/tmp/pocket-demo-reference/49/(pocket-book.api-reference_expensive-calculation 3 4)",
  :id "(pocket-book.api-reference/expensive-calculation 3 4)",
  :fn-name "pocket-book.api-reference/expensive-calculation",
  :args-str "[3 4]",
  :created-at "2026-02-09T08:06:50.214225562Z"}]

`cache-stats`

[]

Return aggregate statistics about the cache. Returns a map with :total-entries, :total-size-bytes, and :entries-per-fn.

(pocket/cache-stats)

{:total-entries 2,
 :total-size-bytes 388,
 :entries-per-fn {"pocket-book.api-reference/expensive-calculation" 2}}

(pocket/cleanup!)

OUT

10:06:50.223 INFO scicloj.pocket - Cache cleanup: /tmp/pocket-demo-reference

{:dir "/tmp/pocket-demo-reference", :existed true}

`origin-story`

[x]

Given a value, return its computation DAG as a nested map.

For a Cached value, each node is {:fn <var> :args [<nodes>] :id <string>}, with :value included if the computation has been realized. Plain (non-Cached) arguments become {:value <val>} leaf nodes.

When the same Cached instance appears multiple times in the tree, subsequent occurrences are represented as {:ref <id>} pointing to the first occurrence’s :id. This enables proper DAG representation for diamond dependencies.

Does not trigger computation — only peeks at already-realized values. Works with all storage policies (:mem+disk, :mem, :none).

origin-story returns a nested map describing a computation’s DAG. Each cached step is {:fn <var> :args [...]}, with :value if realized. Plain arguments become {:value ...} leaves.

(defn step-a [x] (+ x 10))

(defn step-b [x y] (* x y))

(def a-c (pocket/cached #'step-a 5))

(def b-c (pocket/cached #'step-b a-c 3))

Before deref — no :value keys:

(pocket/origin-story b-c)

{:fn #'pocket-book.api-reference/step-b,
 :args
 [{:fn #'pocket-book.api-reference/step-a,
   :args [{:value 5}],
   :id "c2"}
  {:value 3}],
 :id "c1"}

Deref to trigger computation:

(deref b-c)

OUT

10:06:50.228 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/step-b
10:06:50.228 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/step-a

After deref — :value keys appear:

(pocket/origin-story b-c)

{:fn #'pocket-book.api-reference/step-b,
 :args
 [{:fn #'pocket-book.api-reference/step-a,
   :args [{:value 5}],
   :id "c2",
   :value 15}
  {:value 3}],
 :id "c1",
 :value 45}

`origin-story-mermaid`

[x]

Given a value, return a Mermaid flowchart string of its computation DAG.

Accepts a Cached value (walks it via origin-story) or a tree map previously returned by origin-story.

Returns a plain string. Wrap with (kind/mermaid ...) for Kindly rendering.

Returns a Mermaid flowchart with kindly metadata for notebook rendering:

(pocket/origin-story-mermaid b-c)

flowchart TD n0["step-b"] n1["step-a"] n2[/"5"/] n2 --> n1 n1 --> n0 n3[/"3"/] n3 --> n0

`origin-story-graph`

[x]

Given a value, return its computation DAG as a normalized graph.

Returns {:nodes {<id> <node-map>} :edges [[<from> <to>] ...]}.

Node maps contain :fn (for cached steps) or :value (for leaves), plus :value if the cached computation has been realized.

This is the fully normalized (Format B) representation of the DAG. Use origin-story for the tree-with-refs representation (Format A).

origin-story-graph returns a normalized {:nodes ... :edges ...} format, suitable for graph algorithms:

(pocket/origin-story-graph b-c)

{:nodes
 {"c1" {:fn #'pocket-book.api-reference/step-b, :value 45},
  "c2" {:fn #'pocket-book.api-reference/step-a, :value 15},
  "v3" {:value 5},
  "v4" {:value 3}},
 :edges [["c1" "c2"] ["c2" "v3"] ["c1" "v4"]]}

`compare-experiments`

[cached-values]

Compare multiple cached experiment results.

Takes a seq of Cached values (typically final metrics from different hyperparameter configurations). Walks each experiment’s origin-story to extract parameter maps, identifies which parameters vary across experiments, and returns a seq of maps containing the varying params plus the experiment result.

Only parameters that differ across experiments are included. The :result key contains the derefed value of each Cached.

compare-experiments extracts varying parameters from multiple experiments. This is useful for hyperparameter sweeps where you want to see which parameters differ across experiments.

(defn run-exp [config]
  {:rmse (* 0.1 (:lr config))})

(def exp1 (pocket/cached #'run-exp {:lr 0.01 :epochs 100}))

(def exp2 (pocket/cached #'run-exp {:lr 0.001 :epochs 100}))

(pocket/compare-experiments [exp1 exp2])

OUT

10:06:50.237 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/run-exp
10:06:50.238 INFO scicloj.pocket.impl.cache - Cache miss, computing: pocket-book.api-reference/run-exp

[{:lr 0.01, :result {:rmse 0.001}} {:lr 0.001, :result {:rmse 1.0E-4}}]

Note: :epochs is not shown because it’s the same (100) in both experiments. Only varying parameters appear in the comparison.

(pocket/cleanup!)

OUT

10:06:50.242 INFO scicloj.pocket - Cache cleanup: /tmp/pocket-demo-reference

{:dir "/tmp/pocket-demo-reference", :existed true}

Extending `PIdentifiable`

You can customize how your types contribute to cache keys by extending the PIdentifiable protocol. See the Extending Pocket chapter for a full walkthrough with examples.

source: notebooks/pocket_book/api_reference.clj

Setup

Reference

*base-cache-dir*

set-base-cache-dir!

config

cached

caching-fn

maybe-deref

->id

set-mem-cache-options!

reset-mem-cache-options!

*storage*

set-storage!

cleanup!

clear-mem-cache!

invalidate!

invalidate-fn!

cache-entries

cache-stats

origin-story

origin-story-mermaid

origin-story-graph

compare-experiments

Extending PIdentifiable

`base-cache-dir`

`set-base-cache-dir!`

`config`

`cached`

`caching-fn`

`maybe-deref`

`->id`

`set-mem-cache-options!`

`reset-mem-cache-options!`

`storage`

`set-storage!`

`cleanup!`

`clear-mem-cache!`

`invalidate!`

`invalidate-fn!`

`cache-entries`

`cache-stats`

`origin-story`

`origin-story-mermaid`

`origin-story-graph`

`compare-experiments`

Extending `PIdentifiable`