1    Introduction

1.1    Overview

std.lib.schema is part of the standard foundation library set. This page collects the public API reference for the namespace.

2    Walkthrough

2.1    Scope expansion

Schemas tag columns with scopes such as :-/id, :-/data, and :-/info. The wildcard scopes : are expanded by expand-scopes into concrete scope keywords.

expand wildcard scopes

(schema/expand-scopes :*/data)
=> #{:-/info :-/id :-/data :-/key}

(schema/expand-scopes :*/info)
=> #{:-/info :-/id :-/key}

2.2    Ordering and defaults

order-keys sorts a list of columns by their declared :order. get-defaults collects SQL defaults such as auto-generated UUIDs and boolean flags.

order keys according to schema

(let [tsch {:cache  [{:order 3}]
            :status [{:order 1}]
            :id     [{:order 0}]
            :name   [{:order 2}]}]
  (schema/order-keys tsch [:cache :status :id :WRONG]))
=> '(:id :status :name :cache :WRONG)

collect default values

(let [tsch {:__deleted__ [{:sql {:default false} :order 8}]
            :id          [{:sql {:default '(uuid)} :order 0}]}]
  (schema/get-defaults tsch))
=> '{:__deleted__ false :id (uuid)}

2.3    Validation checks

Validate incoming data against a schema: check-valid-columns rejects unknown columns, check-missing-columns ensures required columns are present, and check-fn-columns runs custom :check predicates.

check column validity

(let [tsch {:id     [{:required true :order 0}]
            :status [{:required true :order 1}]
            :name   [{:required true :order 2}]}]
  (schema/check-valid-columns tsch [:id :status])
  => [true]

  (schema/check-valid-columns tsch [:id :WRONG])
  => (contains-in [false {:not-allowed #{:WRONG}}]))

check required columns

(let [tsch {:id     [{:required true :order 0}]
            :status [{:required true :order 1}]
            :name   [{:required true :order 2}]}]
  (schema/check-missing-columns tsch [:status] :required)
  => [false {:missing #{:id :name}
             :required #{:id :status :name}}]

  (schema/check-missing-columns tsch [:status :name :id] :required)
  => [true])

run per-column check functions

(let [tsch {:status [{:check keyword? :order 1}]}]
  (schema/check-fn-columns tsch {:status 'a})
  => {})

2.4    Returning columns

get-returning builds the column list for a RETURNING clause. It accepts explicit columns and wildcard scopes, and returns ordered attribute entries.

collect returning columns

(let [tsch {:__deleted__ [{:scope :-/hidden :order 8}]
            :id          [{:scope :-/id :order 0}]
            :status      [{:scope :-/info :order 1}]
            :name        [{:scope :-/data :order 2}]
            :cache       [{:scope :-/ref :order 3}]}]
  (->> (schema/get-returning tsch [:*/data :cache])
       (map first)))
=> '(:id :status :name :cache)

2.5    End-to-end: validate input and compute a return set

A typical insert flow validates the requested columns, checks required fields, and then expands a scope-based return list.

validate data and build a RETURNING clause

(let [tsch   {:id          [{:scope :-/id    :required true :order 0}]
              :status      [{:scope :-/info  :required true :order 1}]
              :name        [{:scope :-/data  :required true :order 2}]
              :time-created [{:scope :-/system :order 3}]
              :__deleted__  [{:scope :-/hidden :sql {:default false} :order 4}]}
      input  [:id :status :name]]
  (schema/check-valid-columns tsch input)
  => [true]

  (schema/check-missing-columns tsch input :required)
  => [true]

  (->> (schema/get-returning tsch [:*/data :*/id])
       (map first))
  => '(:id :status :name))

3    API

check-fn-columns ^

[tsch data]
Added 4.0

perform a check using the `:check` key

v 4.0
(defn check-fn-columns
  ([tsch data]
   (let [checks (coll/keep-vals (fn [[{:keys [check]}]]
                               check)
                             tsch)]
     (->> data
          (keep (fn [[k v]]
                  (if-let [check (get checks k)]
                    (if (not (h/suppress (check v)))
                      [k {:key k
                          :value v
                          :check check
                          :message "Check failed for input" }]))))
          (into {})))))
link
(schema/check-fn-columns -tsch- {:status 'a})
=> {}

check-missing-columns ^

[tsch columns required-fn]
Added 4.0

check if columns are missing

v 4.0
(defn check-missing-columns
  ([tsch columns required-fn]
   (let [required (set (keep (fn [[k [attrs]]]
                               (if (required-fn attrs) k))
                             tsch))
         diff (clojure.set/difference required
                                      (set columns))]
     (if (empty? diff)
       [true]
       [false {:missing diff
               :required required}]))))
link
(schema/check-missing-columns -tsch-
                       [:status]
                       :required)
=> [false {:missing #{:name :cache},
           :required #{:name :cache :status}}]

(schema/check-missing-columns -tsch-
                       [:status :name :cache]
                       :required)
=> [true]

check-scope ^

[scope]
Added 4.0

check if a scope is valid

v 4.0
(defn check-scope
  [scope]
  (or (get (:*/everything +scope+) scope)
      (h/error "Scope not Valid" {:value scope
                                  :allowed (:*/everything +scope+)})))
link
(base/check-scope :-/info)
=> :-/info

(base/check-scope :-/WRONG)
=> (throws)

check-valid-columns ^

[tsch columns]
Added 4.0

check if columns are valid

v 4.0
(defn check-valid-columns
  ([tsch columns]
   (let [diff (clojure.set/difference (set (filter keyword? columns))
                                      (set (keys tsch)))]
     (if (empty? diff)
       [true]
       [false {:not-allowed diff
               :valid (set (keys tsch))}]))))
link
(schema/check-valid-columns -tsch- [:id :status])
=> [true]

(schema/check-valid-columns -tsch- [:id :WRONG])
=> (contains-in [false {:not-allowed #{:WRONG}}])

expand-scopes ^

[k]
Added 4.0

expand `*` keys into `-` keys

v 4.0
(defn expand-scopes
  [k]
  (get base/+scope+ k))
link
(schema/expand-scopes :*/data)
=> #{:-/info :-/id :-/data :-/key}

(schema/expand-scopes :*/info)
=> #{:-/info :-/id :-/key}

get-defaults ^

[tsch]
Added 4.0

get defaults in the schema

v 4.0
(defn get-defaults
  ([tsch]
   (coll/keep-vals (fn [[{:keys [sql]}]] (:default sql))
                   (comp not nil?)
                   tsch)))
link
(schema/get-defaults -tsch-)
=> '{:__deleted__ false, :id (postgres.core/uuid-generate-v4)}

get-returning ^

[tsch returning]
Added 4.0

collects the returning ids and columns

v 4.0
(defn get-returning
  ([tsch returning]
   (let [scope-ks (filter (fn [x] (and (keyword? x) (namespace x))) returning)
         col-ks   (set (filter (fn [x] (and (keyword? x) (nil? (namespace x)))) returning))
         [_ err]  (check-valid-columns tsch col-ks)
         _ (if err (h/error "Not valid." (assoc err :data returning)))
         union-fn (fn union-fn [s]
                    (apply clojure.set/union
                           (map (fn [k]
                                  (if (= "*" (namespace k))
                                    (union-fn (expand-scopes k))
                                    #{k}))
                                s))) 
         all-fn (union-fn scope-ks)]
     (->> (seq tsch)
          (sort-by (fn [[k [{:keys [order]}]]]
                     [(or order Integer/MAX_VALUE) k]))
          (filter  (fn [[k [{:keys [scope]}]]]
                     (or (all-fn scope)
                         (col-ks k))))))))
link
(->> (schema/get-returning -tsch-
                    [:*/data :cache])
     (map first))
=> '(:id :status :name :cache :time-created :time-updated)

linked-primary ^

[tsch k schema]
Added 4.0

gets the linked primary column

v 4.0
(defn linked-primary
  ([tsch k schema]
   (let [link  (-> tsch k first :ref :link)
         rid   (keyword (name (:id link)))
         rsch  (get (:tree schema) rid)
         [k [attrs]] (first (filter (fn [[k [{:keys [primary] :as m}]]]
                                (if primary [k m]))
                              rsch))]
     (if (nil? attrs)
       (h/error "Column not found" {:link link
                                    :key k
                                    :tree rsch})
       attrs))))
link
(schema/linked-primary -tsch-
                       :cache
                       +schema+)
=> '{:type :uuid, :cardinality :one, :primary true,
     :web {:example "AUD"},
     :sql {:default (postgres.core/uuid-generate-v4)}, :scope :-/id, :order 0, :ident :TaskCache/id}

order-keys ^

[tsch ks]
Added 4.0

order keys given schema

v 4.0
(defn order-keys
  ([tsch ks]
   (sort-by (fn [k] (or (-> tsch k (first) :order)
                        Integer/MAX_VALUE))
            ks)))
link
(schema/order-keys -tsch-
                   [:cache
                    :status
                    :id
                    :WRONG])
=> '(:id :status :cache :WRONG)

schema ^

[x] [x defaults]
Added 3.0

creates an extended schema for use by spirit

v 3.0
(defn schema
  ([x]
   (schema x (base/all-auto-defaults base/base-meta)))
  ([x defaults]
   (cond (vector? x)
         (let [schema (-> (vec->map x)
                          (schema-map defaults))]
           (assoc schema :vec x))

         (map? x)
         (schema-map x defaults))))
link
(-> (schema [:account [:name  {}
                       :email {:type :ref :ref {:ns :email}}]])
    :flat
    simplify)
=> {:email {:accounts :&account<*>}
    :account {:email :&email
              :name :string}}

schema? ^

[obj]
Added 3.0

checks if object is a schema

v 3.0
(defn schema?
  ([obj]
   (instance? Schema obj)))
link
(schema? (schema {:user/email [{}]}))
=> true

with:ref-fn ^

[[ref-fn] & body]
Added 4.0

passes a function for use in `reverse-ref-attr` method to add additional params to schema

v 4.0
(defmacro ^{:style/indent 1}
  with:ref-fn
  [[ref-fn] & body]
  `(binding [*ref-fn* ~ref-fn]
     ~@body))
link
(resolve 'with:ref-fn) => var?

4    std.lib.schema: A Comprehensive Summary

The std.lib.schema module provides a powerful and flexible framework for defining, manipulating, and validating data schemas. It is designed to handle complex, nested data structures, particularly those used in database modeling or API definitions. The module offers functionalities for schema creation, flattening, nesting, reference handling, and various assertion checks, making it a central component for data integrity and consistency within the foundation-base project.

The module is organized into several sub-namespaces:

4.1    std.lib.schema.base

This namespace defines fundamental concepts and utilities for schema processing, including scope definitions, default attribute values, and type checking.

  • +scope-brief+: A map defining hierarchical scopes (e.g., :*/min, :*/info, :*/data) for schema attributes, allowing for grouping and filtering of columns based on their relevance.n expand-scopes [m & [expanded]]: Expands globbed scope keywords (e.g., :*/data) into their constituent concrete scope keywords (e.g., :-/info, :-/id, :-/data).n +scope+: A pre-expanded version of +scope-brief+, providing a complete mapping of globbed scopes to their concrete keywords.n check-scope [scope]: Validates if a given scope keyword is a valid, known scope.n base-meta: A map defining metadata for base schema attributes (e.g., :ident, :type, :cardinality), including their required status, default values, and validation checks.n attr-add-ident [[k [attr :as v]]]: Adds the key k as the :ident attribute to a schema property.n attr-add-defaults [[k [attr :as v]] dfts]: Adds default values to schema properties based on a list of default attribute definitions.n defaults [](#k prop): Extracts default and auto-generation properties from a schema attribute definition.n all-auto-defaults [& [meta]]: Collects all attributes that have an :auto default value from base-meta or a custom meta.n all-defaults [& [meta]]: Collects all attributes that have a :default value from base-meta or a custom meta.n type-checks [t k]: A multimethod for retrieving type-checking functions based on a category t and type k (e.g., :default :string returns string?).

4.2    std.lib.schema.impl

This namespace provides the core implementation for creating and manipulating Schema objects, including flattening, nesting, and creating lookup structures.

  • simplify [flat]: A helper function to simplify a flattened schema for easier display, converting complex attribute maps into concise keyword representations.n Schema Record: The central record for a schema, holding:n flat: A flattened representation of the schema (dot-separated keys).n tree: A nested (tree-like) representation of the schema.n lu: A lookup map for reverse references.n vec: An optional vector representation of the schema.n It implements Object/toString for informative display.n create-lookup [fschm]: Creates a lookup map from a flattened schema, primarily used for resolving reverse references.n create-flat-schema [m & [defaults]]: Creates a flattened schema from an input map m, applying default attribute values. It handles adding :ident and default attributes, and processes reference attributes.n vec->map [v]: Converts a vector-based schema definition into a map-based representation.n schema-map [m & [defaults]]: Creates a Schema object from a map-based schema definition.n schema [x & [defaults]]: The main function for creating a Schema object. It can accept either a map or a vector representation of the schema.n schema? [obj]: A predicate to check if an object is a Schema instance.

4.3    std.lib.schema.ref

This namespace focuses on handling reference attributes within a schema, including creating forward and reverse references and managing their properties.

  • *ref-fn*: A dynamic var that can be bound to a function for adding additional parameters to reverse reference attributes.n with:ref-fn [[ref-fn] & body]: A macro to bind *ref-fn* for a block of code.n keyword-reverse [k]: Reverses a keyword by adding or removing an underscore prefix (e.g., :a/b -> :a/_b, :a/_b -> :a/b).n keyword-reversed? [k]: Checks if a keyword is in its "reversed" form (starts with an underscore).n is-reversible? [attr]: Determines if a reference attribute is eligible for automatic reverse reference generation.n determine-rval [entry]: Determines the :rval (reverse value) for a reference attribute, handling pluralization and naming conventions.n forward-ref-attr [](#attr): Creates the :ref schema attribute for a forward reference, populating its properties (e.g., :type, :key, :val, :rkey, :rident).n reverse-ref-attr [](#attr): Creates the reverse :ref schema attribute for a backward reference, defining its properties and linking it back to the forward reference.n forward-ref-attr-fn [entry]: A helper function for forward-ref-attr.n attr-ns-pair [](#attr): Constructs a [:ns :ident-root] pair for a schema attribute.n mark-multiple [nsgroups & [output]]: Marks groups of namespace/ident pairs that have multiple entries, used in reference processing.n* ref-attrs [fschm]: Creates both forward and reverse reference attributes for a flattened schema, automatically generating reverse references where applicable.

4.4    std.lib.schema (Facade Namespace)

This namespace acts as a facade, re-exporting key functions from its sub-namespaces and providing additional utilities for schema introspection and validation.

  • h/intern-in: Interns impl/schema, impl/schema?, base/check-scope, and ref/with:ref-fn into this namespace.n expand-scopes [k]: Expands globbed scope keywords (e.g., :*/data) into their constituent concrete scope keywords using base/+scope+.n linked-primary [tsch k schema]: Retrieves the primary key attribute of a linked table within the schema.n order-keys [tsch ks]: Orders a list of keys based on their :order attribute defined in the schema.n get-defaults [tsch]: Collects default values from the sql attributes of schema columns.n check-valid-columns [tsch columns]: Checks if a given set of columns are valid (exist) within the schema.n check-missing-columns [tsch columns required-fn]: Checks if any required columns (as determined by required-fn) are missing from a given set of columns.n check-fn-columns [tsch data]: Performs validation checks on data using :check functions defined in the schema attributes.n get-returning [tsch returning]: Collects a list of columns to be returned, expanding scope keywords (e.g., :*/data) and validating column names.

Overall Importance:

The std.lib.schema module is a critical component for managing data definitions and ensuring data integrity within the foundation-base project. Its key contributions include:

  • Centralized Schema Definition: Provides a unified way to define and manage complex data schemas, including relationships and validation rules.n Data Transformation: Offers tools for flattening and nesting schema representations, facilitating data processing and API interactions.n Automated Reference Handling: Simplifies the creation and management of forward and reverse references between tables/entities, crucial for relational data.n Robust Validation: Provides mechanisms for checking column validity, identifying missing required columns, and performing custom data validation.n Extensibility: The modular design allows for easy extension with new attribute types, validation rules, and processing logic.n* Code Generation Support: The structured schema representation is ideal for generating database schemas, API documentation, or client-side data models.

By offering these comprehensive schema management capabilities, std.lib.schema significantly enhances the foundation-base project's ability to handle diverse data models accurately and consistently, which is vital for its multi-language development ecosystem.

Related

jvm.artifactdocs

artifact coordinates and paths

jvm.classloaderdocs

classpath and classloaders

jvm.depsdocs

JVM dependency resolution

jvm.monitordocs

JVM metrics

jvm.namespacedocs

namespace management

jvm.protocoldocs

JVM extension protocols

jvm.reflectdocs

Java reflection

jvm.requiredocs

dependency-aware requiring