1    Introduction

1.1    Overview

std.lib.transform is part of the standard foundation library set. This page collects the public API reference for the namespace.

2    Walkthrough

2.1    Normalise flat keys into nested records

normalise takes data with slash-separated keys and a schema, and returns a nested record tree. It is the main entry point for the transform pipeline.

normalise flat keys against a schema

(let [sch (schema/schema [:Account
                          [:id   {:type :uuid :scope :-/id}]
                          [:name {:type :string :scope :-/data}]])]
  (normalise {:Account/name "Chris"
              :Account/age  10}
             {:schema sch}
             {}))
=> {:Account {:name "Chris" :age 10}}

normalise nested data with references

(let [sch (schema/schema [:Link
                          [:value {:type :string :scope :-/data}]
                          [:next  {:type :ref :ref {:ns :Link}}]])]
  (normalise {:Link/value "hello"
              :Link {:next/value "world"
                     :next/next {:value "!"}}}
             {:schema sch}
             {}))
=> {:Link {:value "hello"
           :next {:value "world"
                  :next {:value "!"}}}}

2.2    Submaps and wrappers

submaps extracts the configuration for a single key across multiple directives. The wrapper helpers are combined into the normalise pipeline to enable tracing and extension.

extract submaps for a key

(submaps {:allow  {:account :check}
          :ignore {:account :check}}
         #{:allow :ignore}
         :account)
=> {:allow :check :ignore :check}

allow additional attributes with wrap-plus

(let [sch (schema/schema [:Account
                          [:id   {:type :uuid :scope :-/id}]
                          [:name {:type :string :scope :-/data}]])]
  (normalise {:Account {:name "Main"
                        :+    {:Account {:name "Extra"}}}}
             {:schema sch}
             {:normalise [wrap-plus]}))
=> {:Account {:name "Main"
              :+    {:Account {:name "Extra"}}}}

trace key paths through normalisation

(let [sch (schema/schema [:Account
                          [:id   {:type :uuid :scope :-/id}]
                          [:name {:type :string :scope :-/data}]])]
  (normalise {:Account {:+ {:Account {:WRONG "Chris"}}}}
             {:schema sch}
             {:normalise [wrap-plus]
              :normalise-branch [wrap-key-path]
              :normalise-attr [wrap-key-path]}))
=> (throws-info {:key-path [:Account :+ :Account :WRONG]})

2.3    Lower-level normalisation loops

normalise-loop walks a nested data map, normalise-attr processes a single attribute, and normalise-single follows reference attributes into their target entities.

normalise-loop over a simple map

(normalise-loop {:name "Chris" :age 10}
                {:name [{:type :string :cardinality :one :ident :Account/name}]
                 :age  [{:type :long   :cardinality :one :ident :Account/age}]
                 :sex  [{:type :enum   :cardinality :one
                         :enum {:ns :Account.sex :values #{:m :f}}
                         :ident :Account/sex}]}
                [:Account]
                {}
                {:normalise normalise-loop
                 :normalise-single normalise-single
                 :normalise-attr normalise-attr}
                {})
=> {:name "Chris" :age 10}

normalise-attr on a single value

(normalise-attr "Chris"
                [{:type :string :cardinality :one :ident :Account/name}]
                [:Account :name]
                {}
                {:normalise-single normalise-single}
                {})
=> "Chris"

normalise-single follows a reference

(normalise-single {:value "world"}
                  [{:type :ref
                    :ident :Link/next
                    :cardinality :one
                    :ref {:ns :Link
                          :rval :prev
                          :type :forward
                          :key :Link/next
                          :val :next
                          :rkey :Link/_next
                          :rident :Link/prev}}]
                  [:Link :next]
                  {}
                  {:normalise-attr normalise-attr
                   :normalise normalise-loop
                   :normalise-single normalise-single}
                  {:schema (schema/schema [:Link
                                           [:value {:type :string :scope :-/data}]])})
=> {:value "world"}

2.4    End-to-end: build a schema, normalise, and inspect

Combine schema creation and normalisation to turn flat external data into a structured tree, then inspect the result.

build a schema and normalise flat input

(let [sch (schema/schema [:User
                          [:id    {:type :uuid :scope :-/id}]
                          [:name  {:type :string :scope :-/data}]
                          [:email {:type :string :scope :-/data}]])
      input {:User/id    #uuid "00000000-0000-0000-0000-000000000001"
             :User/name  "Ada"
             :User/email "[email protected]"}]
  (normalise input {:schema sch} {})
  => {:User {:id    #uuid "00000000-0000-0000-0000-000000000001"
             :name  "Ada"
             :email "[email protected]"}})

3    API

normalise ^

[data {:keys [pipeline], :as datasource}] [data datasource wrappers]
Added 3.0

base normalise function

v 3.0
(defn normalise
  ([data {:keys [pipeline] :as datasource}]
   (let [wrappers (normalise-wrappers datasource)]
     (normalise data datasource wrappers)))
  ([data datasource wrappers]
   (let [tdata (coll/tree-nestify:all data)
         tdata (if-let [pre-process-fn (-> datasource :pipeline :pre-process)]
                 (pre-process-fn tdata datasource)
                 tdata)
         output (normalise-base tdata datasource wrappers)]
     (if-let [post-process-fn (-> datasource :pipeline :post-process)]
       (post-process-fn output datasource)
       output))))
link
(normalise {:account/name "Chris"
            :account/age 10}
           {:schema (schema/schema examples/account-name-age-sex)}
           {})
=> {:account {:age 10, :name "Chris"}}

(normalise {:link/value "hello"}
           {:schema (schema/schema examples/link-value-next)}
           {})
=> {:link {:value "hello"}}

(normalise {:link/value "hello"
            :link {:next/value "world"
                   :next/next {:value "!"}}}
           {:schema (schema/schema examples/link-value-next)}
           {})

=> {:link {:next {:next {:value "!"}
                  :value "world"}
           :value "hello"}}

normalise-attr ^

[subdata [attr] nsv interim fns datasource]
Added 3.0

base function for treating attributes

v 3.0
(defn normalise-attr
  ([subdata [attr] nsv interim fns datasource]
   (cond (set? subdata)
         (-> (keep #((:normalise-single fns) % [attr] nsv interim fns datasource) subdata)
             (set))

         (and (vector? subdata) (not (vector? (first subdata))))
         (-> (keep #((:normalise-single fns) % [attr] nsv interim fns datasource) subdata)
             (vec))

         :else
         ((:normalise-single fns) subdata [attr] nsv interim fns datasource))))
link
(normalise-attr "Chris"
                [{:type :string, :cardinality :one, :ident :account/name}]
                [:account :name]
                {}
                {:normalise-single normalise-single}
                {})
=> "Chris"

normalise-base ^

[tdata datasource wrappers]
Added 3.0

base normalise function

v 3.0
(defn normalise-base
  ([tdata datasource wrappers]
   (let [tsch (-> datasource :schema :tree)
         interim (:pipeline datasource)
         fns {:normalise normalise-loop
              :normalise-nil normalise-nil
              :normalise-branch normalise-loop
              :normalise-attr normalise-attr
              :normalise-expression normalise-expression
              :normalise-single normalise-single}
         fns (normalise-wrap fns wrappers)]
     ((:normalise fns) tdata tsch [] interim fns datasource))))
link
(normalise-base {:account {:name "Chris"
                           :age 10}}
                {:schema (schema/schema examples/account-name-age-sex)}
                {})
=> {:account {:name "Chris", :age 10}}

normalise-expression ^

[subdata [attr] nsv interim datasource]
Added 3.0

normalises an expression

v 3.0
(defn normalise-expression
  ([subdata [attr] nsv interim datasource]
   subdata))
link
(resolve 'normalise-expression) => var?

normalise-loop ^

[tdata tsch nsv interim fns datasource]
Added 3.0

base loop for the normalise function

v 3.0
(defn normalise-loop
  ([tdata tsch nsv interim fns datasource]
   (reduce-kv (fn [output k subdata]
                (let [subsch (get tsch k)
                      pinterim (submaps interim tree-directives k)
                      val (cond (nil? subsch)
                                ((:normalise-nil fns)
                                 subdata nil (conj nsv k) pinterim datasource)

                                (coll/hash-map? subsch)
                                ((:normalise-branch fns)
                                 subdata subsch (conj nsv k) pinterim fns datasource)

                                (vector? subsch)
                                ((:normalise-attr fns)
                                 subdata subsch (conj nsv k) pinterim fns datasource)

                                :else
                                (let [nnsv (conj nsv k)]
                                  (h/error (str "NORMALISE_LOOP: In " nsv ", " subdata
                                                " needs to be a vector or hashmap.")
                                           {:id :wrong-input :data subdata :nsv nnsv :key-path (:key-path interim)})))]
                  (cond-> output val (assoc k val))))
              {} tdata)))
link
(normalise-loop {:name "Chris", :age 10}
                {:name [{:type :string,
                         :cardinality :one,
                         :ident :account/name}],
                 :age [{:type :long,
                        :cardinality :one,
                        :ident :account/age}],
                 :sex [{:type :enum,
                        :cardinality :one,
                        :enum {:ns :account.sex, :values #{:m :f}},
                        :ident :account/sex}]}
                [:account]
                {}
                {:normalise normalise-loop
                 :normalise-single normalise-single
                 :normalise-attr normalise-attr}
                {:schema (schema/schema examples/account-name-age-sex)})
=> {:name "Chris", :age 10}

normalise-nil ^

[subdata _ nsv interim datasource]
Added 3.0

base function for treating nil values

v 3.0
(defn normalise-nil
  ([subdata _ nsv interim datasource]
   (h/error (str "NORMALISE_NIL: " nsv " is not in the schema.")
            {:id :no-schema :nsv nsv :key-path (:key-path interim)
             :ref-path (:ref-path interim)})))
link
(normalise-nil nil [:user :password] {} {} nil)
=> (throws)

normalise-single ^

[subdata [attr] nsv interim fns datasource]
Added 3.0

verifies and constructs a ref value

v 3.0
(defn normalise-single
  ([subdata [attr] nsv interim fns datasource]
   (if (= (:type attr) :ref)
     (cond (coll/hash-map? subdata)
           (let [nnsv ((wrap/wrap path/path-split) (-> attr :ref :ns))]
             ((:normalise fns)
              (coll/tree-nestify:all subdata)
              (get-in datasource (concat [:schema :tree] nnsv))
              nnsv interim fns datasource))

           :else
           (h/error (str "NORMALISE_SINGLE: In " nsv "," subdata " should be either a hashmaps or ids")
                    {:id :wrong-input :nsv nsv :key-path (:key-path interim)}))
     subdata)))
link
(normalise-single {:value "world"}
                  [{:type :ref,
                    :ident :link/next
                    :cardinality :one,
                    :ref {:ns :link,
                          :rval :prev,
                          :type :forward,
                          :key :link/next,
                          :val :next,
                          :rkey :link/_next,
                          :rident :link/prev}}]

                  [:link :next]
                  {}
                  {:normalise-attr normalise-attr
                   :normalise normalise-loop
                   :normalise-single normalise-single}
                  {:schema (schema/schema examples/link-value-next)})
=> {:value "world"}

normalise-wrap ^

[fns wrappers]
Added 3.0

helper function for normalise-wrappers

v 3.0
(defn normalise-wrap
  ([fns wrappers]
   (reduce-kv (fn [out k f]
                (let [nf (if-let [wrapvec (get wrappers k)]
                           (reduce (fn [f wrapper] (wrapper f)) f wrapvec)
                           f)]
                  (assoc out k nf)))
              {} fns)))
link
(resolve 'normalise-wrap) => var?

normalise-wrapper-fns ^

NONE
(def normalise-wrapper-fns)
link

normalise-wrappers ^

[datasource] [{:keys [pipeline options]} additions fns]
Added 3.0

adds function wrappers to the normalise functions

v 3.0
(defn normalise-wrappers
  ([datasource]
   (normalise-wrappers datasource {} normalise-wrapper-fns))
  ([{:keys [pipeline options]} additions fns]
   (->> {:normalise  [:plus
                      (if (:fill-assoc pipeline)     :fill-assoc)
                      (if (:fill-empty pipeline)     :fill-empty)
                      (if (:pre-transform pipeline)  :pre-transform)
                      (if (:pre-mask pipeline)       :pre-mask)
                      (if (:pre-require pipeline)    :pre-require)
                      (if (:post-require pipeline)   :post-require)
                      (if (:post-mask pipeline)      :post-mask)
                      (if (:post-transform pipeline) :post-transform)
                      :ref-path
                      :alias]
         :normalise-nil        [(if (:ignore pipeline) :ignore)]
         :normalise-branch     [(if (:allow pipeline)  :allow-branch)
                                :alias
                                :key-path]
         :normalise-attr       [(if (:allow pipeline) :allow-attr)
                                :key-path]

         :normalise-expression []

         :normalise-single     [:enum
                                :keyword
                                (if (:use-type-check options) :type-check)
                                (if (:convert pipeline) :convert)
                                (if (:validate pipeline) :validate)]}
        (reduce-kv (fn [out k v]
                     (assoc out k (->> (concat (get-in additions [k :pre])
                                               v
                                               (get-in additions [k :post]))
                                       (keep identity)
                                       (map fns))))
                   {}))))
link
(resolve 'normalise-wrappers) => var?

submaps ^

[m options subk]
Added 3.0

creates a submap based upon a lookup subkey

v 3.0
(defn submaps
  ([m options subk]
   (reduce (fn [out option]
             (let [sv (get-in m [option subk])]
               (assoc out option sv)))
           (apply dissoc m (seq options))
           options)))
link
(submaps {:allow  {:account :check}
          :ignore {:account :check}} #{:allow :ignore} :account)
=> {:allow :check, :ignore :check}

tree-directives ^

NONE
(def tree-directives
  #{:pre-require       ;;
    :pre-mask          ;;
    :pre-transform     ;;
    :fill-assoc        ;;
    :fill-empty        ;;
    :ignore            ;;
    :allow             ;;
    :expression        ;;
    :validate          ;;
    :convert           ;;
    :post-transform    ;;
    :post-mask         ;;
    :post-require      ;;
    })
link

wrap-key-path ^

[f]
Added 3.0

used for tracing the keys through `normalise`

v 3.0
(defn wrap-key-path
  ([f]
   (fn [tdata tsch nsv interim fns datasource]
     (f tdata tsch nsv (update-in interim [:key-path] (fnil #(conj % (last nsv)) [])) fns datasource))))
link
(normalise {:account {:orders {:+ {:account {:WRONG "Chris"}}}}}
           {:schema (schema/schema examples/account-orders-items-image)}
           {:normalise [wrap-plus]
            :normalise-branch [wrap-key-path]
            :normalise-attr [wrap-key-path]})

=>  (throws-info {:key-path [:account :orders :+ :account]})

wrap-plus ^

[f]
Added 3.0

allows additional attributes (besides the link :ns) to be added to the entity

v 3.0
(defn wrap-plus
  ([f]
   (fn [tdata tsch nsv interim fns datasource]
     (let [output (f (dissoc tdata :+) tsch nsv interim fns datasource)
           pinterim  (submaps interim tree-directives :+)]
       (if-let [tplus (:+ tdata)]
         (let [pinterim (update-in pinterim [:key-path] conj :+)]
           (assoc output :+
                  ((:normalise fns) tplus (-> datasource :schema :tree) [] pinterim fns datasource)))
         output)))))
link
(normalise {:account {:orders {:+ {:account {:user "Chris"}}}}}
           {:schema (schema/schema examples/account-orders-items-image)}
           {:normalise [wrap-plus]})
=> {:account {:orders {:+ {:account {:user "Chris"}}}}}
(normalise {:account {:orders {:+ {:account {:user "Chris"}}}}}
           {:schema (schema/schema examples/account-orders-items-image)}
           {})
=> (throws)

wrap-ref-path ^

[f]
Added 3.0

used for tracing the entities through `normalise`

v 3.0
(defn wrap-ref-path
  ([f]
   (fn [tdata tsch nsv interim fns datasource]
     (f tdata tsch nsv (update-in interim [:ref-path] (fnil #(conj % tdata) [])) fns datasource))))
link
(normalise {:account {:orders {:+ {:account {:WRONG "Chris"}}}}}
           {:schema (schema/schema examples/account-orders-items-image)}
           {:normalise [wrap-ref-path wrap-plus]})

=> (throws-info {:ref-path
                 [{:account {:orders {:+ {:account {:WRONG "Chris"}}}}}
                  {:account {:WRONG "Chris"}}]})

4    std.lib.transform: A Comprehensive Summary

The std.lib.transform module provides a powerful and extensible framework for transforming and normalizing data based on a schema. It's designed to process complex, nested data structures, particularly those used in database modeling or API definitions. The core of the module is the normalise function, which orchestrates a pipeline of transformations defined by various sub-namespaces. This module is crucial for ensuring data quality and consistency within the foundation-base project.

The module is organized into several sub-namespaces:

4.1    std.lib.transform.allow

This namespace provides transformation wrappers related to allowing or disallowing certain data based on schema definitions.

  • wrap-branch-model-allow [f]: A wrapper that filters branches (sub-maps) based on schema rules.n* wrap-attr-model-allow [f]: A wrapper that filters attributes (values) based on schema rules.

4.2    std.lib.transform.apply

This namespace likely contains functions for applying transformations. (No direct apply.clj found, but std.lib.transform.apply is referenced in std.lib.transform.clj).

4.3    std.lib.transform.base.alias

This namespace handles aliasing of keys during transformation.

  • wrap-alias [f]: A wrapper that applies key aliasing based on schema definitions.

4.4    std.lib.transform.base.enum

This namespace provides transformations related to enum types.

  • wrap-single-enum [f]: A wrapper that handles single enum values, likely for validation or coercion.

4.5    std.lib.transform.base.keyword

This namespace provides transformations related to keywords.

  • wrap-single-keyword [f]: A wrapper that handles single keyword values, likely for validation or coercion.

4.6    std.lib.transform.base.type-check

This namespace provides transformations related to type checking.

  • wrap-single-type-check [f]: A wrapper that performs type checking on single values.

4.7    std.lib.transform.convert

This namespace provides transformations for converting data types.

  • wrap-single-model-convert [f]: A wrapper that converts single data values based on schema type definitions.

4.8    std.lib.transform.fill-assoc

This namespace provides transformations for associating default values.

  • wrap-model-fill-assoc [f]: A wrapper that fills in associated default values based on schema definitions.

4.9    std.lib.transform.fill-empty

This namespace provides transformations for filling in empty values.

  • wrap-model-fill-empty [f]: A wrapper that fills in empty values based on schema definitions.

4.10    std.lib.transform.ignore

This namespace provides transformations for ignoring certain data.

  • wrap-nil-model-ignore [f]: A wrapper that ignores nil values based on schema rules.

4.11    std.lib.transform.mask

This namespace provides transformations for masking data.

  • wrap-model-pre-mask [f]: A wrapper that applies masking to data before other transformations.n* wrap-model-post-mask [f]: A wrapper that applies masking to data after other transformations.

4.12    std.lib.transform.require

This namespace provides transformations related to required fields.

  • wrap-model-pre-require [f]: A wrapper that checks for required fields before other transformations.n* wrap-model-post-require [f]: A wrapper that checks for required fields after other transformations.

4.13    std.lib.transform.validate

This namespace provides transformations for validating data.

  • wrap-single-model-validate [f]: A wrapper that validates single data values based on schema rules.

4.14    std.lib.transform (Facade Namespace)

This namespace acts as a facade, orchestrating the transformation pipeline and providing the main normalise function. It also defines common transformation directives and helper functions.

  • tree-directives: A set of keywords representing the different stages or types of transformations that can be applied (e.g., :pre-require, :fill-assoc, :validate, :convert).n submaps [m options subk]: Extracts sub-maps from a map m based on a set of options and a subk (subkey).n wrap-plus [f]: A wrapper that handles additional attributes (denoted by :+) in the data, allowing for recursive normalization of these attributes.n wrap-ref-path [f]: A wrapper used for tracing the reference path during normalization, useful for debugging or error reporting.n wrap-key-path [f]: A wrapper used for tracing the key path during normalization.n normalise-loop [tdata tsch nsv interim fns datasource]: The core recursive loop for the normalise function, iterating through data and applying transformations based on the schema.n normalise-nil [subdata _ nsv interim datasource]: Handles cases where a sub-schema is not found, typically throwing an error.n normalise-attr [subdata [attr] nsv interim fns datasource]: Handles the normalization of individual attributes, including collections of attributes.n normalise-single [subdata [attr] nsv interim fns datasource]: Handles the normalization of single attribute values, including reference resolution.n normalise-expression [subdata [attr] nsv interim datasource]: Normalizes expressions within the data.n normalise-wrap [fns wrappers]: Applies a series of wrappers to a set of normalization functions.n normalise-wrapper-fns: A map linking wrapper keywords (e.g., :plus, :fill-assoc, :alias) to their corresponding wrapper functions.n normalise-wrappers [datasource & [additions fns]]: Constructs a map of wrapped normalization functions based on the pipeline configuration and available wrappers.n normalise-base [tdata datasource wrappers]: The base function that applies the wrapped normalization functions to the data.n normalise [data datasource & [wrappers]]: The main function for transforming and normalizing data. It takes data, a datasource (containing the schema and pipeline configuration), and optional wrappers. It orchestrates the entire transformation pipeline, including pre- and post-processing steps.

Overall Importance:

The std.lib.transform module is a critical component for data management and processing within the foundation-base project. Its key contributions include:

  • Schema-Driven Data Transformation: Provides a declarative and extensible way to transform data based on a defined schema, ensuring data consistency and adherence to rules.n Robust Data Validation: Integrates various validation steps (required fields, type checks, custom checks) into the transformation pipeline.n Automated Data Coercion and Defaulting: Handles automatic type conversion, filling in default values, and associating related data.n Flexible Transformation Pipeline: Allows for a highly customizable sequence of transformations, including pre- and post-processing hooks.n Error Reporting and Tracing: Provides mechanisms for reporting errors during transformation, including tracing the path of data and references.n* Modularity and Extensibility: The use of wrappers and sub-namespaces allows for easy extension with new transformation types and rules.

By offering these comprehensive data transformation capabilities, std.lib.transform significantly enhances the foundation-base project's ability to manage and process diverse data models accurately and consistently, which is vital for its multi-language development ecosystem.

Related

jvm.artifactdocs

artifact coordinates and paths

jvm.classloaderdocs

classpath and classloaders

jvm.depsdocs

JVM dependency resolution

jvm.monitordocs

JVM metrics

jvm.namespacedocs

namespace management

jvm.protocoldocs

JVM extension protocols

jvm.reflectdocs

Java reflection

jvm.requiredocs

dependency-aware requiring