std.task

process and bulk task execution

std.task is a higher-level standard library family in foundation-base. This page explains when to use it, how it fits internally, and where to find the API surface.

1    Motivation

Use this layer when application or tooling code needs the behavior described by the page title without reaching directly into implementation namespaces. The top-level namespace is the starting point; subnamespaces expose more focused building blocks.

2    How to use it

Require the top-level namespace for common workflows, then move to subnamespaces when you need a lower-level primitive. Existing tests under test/std/task and test/std/task_test.clj are the best executable examples for edge cases.

create a task

(task? (task :namespace "list-interns" ns-interns))
=> true

define a task with deftask

(macroexpand-1
 '(deftask -list-aliases-
    {:template :namespace
     :main clojure.core/ns-aliases
     :item {:post (comp vec sort keys)}
     :doc "returns all aliases"}))
=> '(def -list-aliases-
      (std.task/task
       :namespace "-list-aliases-"
       {:template :namespace,
        :main clojure.core/ns-aliases,
        :item {:post (comp vec sort keys)},
        :doc "returns all aliases"}))

parse CLI args

(process-ns-args [":only" "foo"])
=> {:ns 'foo}

(process-ns-args [":verbose" ":timeout" "100"])
=> {:verbose true :timeout 100}

3    Internal usage

This library family is used across source, tests, generated examples, and docs tooling. During detailed documentation passes, collect concrete usage with code.manage/find-usages and code.manage/locate-code, then keep only high-signal examples in the page narrative.

4    API

->Task ^

[type name main construct arglists item result summary]

NONE
(impl/defimpl Task [type name main construct arglists item result summary]
  :invoke process/invoke
  :string task-string
  :final true)
link

Task ^

NONE
(impl/defimpl Task [type name main construct arglists item result summary]
  :invoke process/invoke
  :string task-string
  :final true)
link

collapse-only ^

[args]
Added 4.1

collapses multiple :only namespace args into a vector literal

v 4.1
(defn collapse-only
  [args]
  (loop [result []
         [k & more] args]
    (if-not k
      result
      (if (= :only (read-cli-arg k))
        (let [vals (take-while (complement keyword-cli-arg?) more)]
          (if (seq vals)
            (recur (conj result k (pr-str (mapv read-cli-arg vals)))
                   (drop (count vals) more))
            (recur (conj result k) more)))
        (recur (conj result k) more)))))
link
(collapse-only [":only" "foo"])
=> [":only" "[foo]"]

(collapse-only [":only" "foo" "bar" "baz"])
=> [":only" "[foo bar baz]"]

(collapse-only [":only" "foo" "bar" ":timeout" "100"])
=> [":only" "[foo bar]" ":timeout" "100"]

(collapse-only [":only"])
=> [":only"]

(collapse-only [":only" ":verbose"])
=> [":only" ":verbose"]

(process-ns-args (collapse-only [":only" "foo"]))
=> {:ns '[foo]}

(process-ns-args (collapse-only [":only" "foo" "bar" "baz"]))
=> {:ns '[foo bar baz]}

(process-ns-args (collapse-only [":only" "foo" "bar" ":timeout" "100"]))
=> {:ns '[foo bar] :timeout 100}

deftask ^

[name config & body]
Added 3.0

defines a top level task

v 3.0
(defmacro deftask
  ([name config & body]
   (invoke-intern-task :task name config body)))
link
(macroexpand-1
 '(deftask -list-aliases-
    {:template :namespace
     :main clojure.core/ns-aliases
     :item {:post (comp vec sort keys)}
     :doc  "returns all aliases"}))
=> '(def -list-aliases-
      (std.task/task
       :namespace "-list-aliases-"
       {:template :namespace,
        :main clojure.core/ns-aliases,
        :item {:post (comp vec sort keys)},
        :doc "returns all aliases"}))

invoke-intern-task ^

[name config] [_ name config _]
Added 3.0

creates a form defining a task

v 3.0
(invoke/definvoke invoke-intern-task
  [:method {:multi protocol.invoke/-invoke-intern
            :val :task}]
  ([name config]
   (invoke-intern-task :task name config nil))
  ([_ name config _]
   (let [template (:template config)
         body `(task ~template ~(str name) ~config)
         arglists (or (:arglists config)
                      (-> (task-defaults template) :arglists))
         name (with-meta name
                (merge (meta name)
                       (cond-> config
                         arglists (assoc :arglists (list 'quote arglists)))))]
     (list 'def name body))))
link
(invoke-intern-task '-task- '{:template :namespace
                              :main {:fn clojure.core/ns-aliases}})
=> '(def -task-
      (std.task/task :namespace "-task-"
                     {:template :namespace, :main {:fn clojure.core/ns-aliases}}))

keyword-cli-arg? ^

[s]
Added 4.1

checks whether a CLI string reads as a keyword

v 4.1
(defn keyword-cli-arg?
  [s]
  (keyword? (read-cli-arg s)))
link
(keyword-cli-arg? ":foo")
=> true

(keyword-cli-arg? ":foo/bar")
=> true

(keyword-cli-arg? "foo")
=> false

(keyword-cli-arg? "1")
=> false

(keyword-cli-arg? "[1 2]")
=> false

map->Task ^

[m__7997__auto__]
Added 3.0

constructs a invokable Task object

v 3.0
(impl/defimpl Task [type name main construct arglists item result summary]
  :invoke process/invoke
  :string task-string
  :final true)
link
(-> (map->Task {:type :namespace
                :name "list-interns"
                :main {:fn clojure.core/ns-interns}})
    (task?))
=> true

parse-cli-value ^

[k raw]

NONE
(defn- parse-cli-value
  [k raw]
  (if (= k :files)
    (let [raw (str/trim raw)]
      (cond
        (empty? raw)
        []
        
        (re-find #"^(?:[[({]|#{|")" raw)
        (try (read-string raw)
             (catch Throwable _
               raw))
        
        (re-find #"s+" raw)
        (vec (remove empty?
                     (str/split raw #"s+")))
        
        :else
        raw))
    (try (read-string raw)
         (catch Throwable _
           raw))))
link

process-ns-args ^

[args]
Added 4.0

processes arguments for tasks

v 4.0
(defn process-ns-args
  [args]
  (loop [m     {}
         [k v :as args]  args]
     (if (not k)
       m
       (let [k (try (read-string k)
                    (catch Throwable _
                      k))
             v (when v
                 (parse-cli-value k v))]
         (cond (not (keyword? k))
               (recur m (rest args))


              (or (keyword? v)
                  (= 1 (count args)))
              (recur (assoc m k true)
                     (rest args))
              
              :else
              (recur (case k
                       :only    (assoc m :ns v)
                       :with    (assoc m :ns [v])
                       (assoc m k v))
                     (rest (rest args))))))))
link
(process-ns-args [":only" "foo"])
=> {:ns 'foo}

(process-ns-args [":verbose"])
=> {:verbose true}

(process-ns-args [":verbose" ":other"])
=> {:verbose true :other true}

(process-ns-args [":timeout" "100"])
=> {:timeout 100}

(process-ns-args [":files" "test/code/project_test.clj test/std/task_test.clj"])
=> {:files ["test/code/project_test.clj"
            "test/std/task_test.clj"]}

(process-ns-args [":files" "["test/code/project_test.clj" "test/std/task_test.clj"]"])
=> {:files ["test/code/project_test.clj"
            "test/std/task_test.clj"]}

read-cli-arg ^

[s]
Added 4.1

reads a CLI string as an EDN value, falling back to raw string

v 4.1
(defn read-cli-arg
  [s]
  (try (read-string s)
       (catch Throwable _
         s)))
link
(read-cli-arg "1")
=> 1

(read-cli-arg ":foo")
=> :foo

(read-cli-arg "[1 2 3]")
=> [1 2 3]

(read-cli-arg "hello")
=> 'hello

(read-cli-arg "hello world")
=> 'hello

single-function-print ^

[params]
Added 3.0

if not `:bulk`, then print function output

v 3.0
(defn single-function-print
  ([params]
   (if (and (not (:bulk params))
            (-> params :print :function nil?))
     (assoc-in params [:print :function] true)
     params)))
link
(single-function-print {})
=> {:print {:function true}}

task ^

[m] [type name arg]
Added 3.0

creates a task

v 3.0
(defn task
  ([m]
   (map->Task m))
  ([type name arg]
   (let [[params main] (if (map? arg)
                         [arg (-> arg :main :fn)]
                         [{} arg])
         defaults     (task-defaults type)
         params       (collection/merge-nested defaults params)
         count        (or (-> params :main :argcount) 4)
         [main args?] (process/main-function main count)]
     (task (collection/merge-nested defaults
                           params
                           {:main {:fn main
                                   :args? args?}
                            :name name
                            :type type})))))
link
(task? (task :namespace "list-interns" ns-interns))
=> true

(task? (task :namespace
      "list-interns"
      {:main {:fn clojure.core/ns-interns}}))
=> true

task-defaults ^

Added 3.0

creates default settings for task groups

v 3.0
(defmulti task-defaults
  identity)
link
(task-defaults :default)
=> {:main {:arglists '([] [entry])}}

task-info ^

[task]
Added 3.0

displays the task-body

v 3.0
(defn task-info
  ([^Task task]
   {:fn (symbol (.name task))}))
link
(-> (task :namespace "list-interns" ns-interns)
    (task-info))
=> '{:fn list-interns}

task-status ^

[task]
Added 3.0

displays the task-status

v 3.0
(defn task-status
  ([^Task task]
   (.type task)))
link
(-> (task :namespace "list-interns" ns-interns)
    (task-status))
=> :namespace

task-string ^

[task]

NONE
(defn- task-string
  ([task]
   (str "#task" (task-status task) " " (task-info task))))
link

task? ^

[x]
Added 3.0

check if object is a task

v 3.0
(defn task?
  ([x]
   (instance? Task x)))
link
(-> (task :namespace "list-interns" ns-interns)
    (task?))
=> true

*interrupt* ^

NONE
(def ^:dynamic *interrupt* false)
link

invoke ^

[task & args]
Added 4.0

executes the task, given functions and parameters

v 4.0
(defn invoke
  ([task & args]
   (let [idx (collection/index-at #{:args} args)
         _    (if (and (neg? idx) (-> task :main :args?))
                (throw (ex-info "Require `:args` keyword to specify additional arguments"
                                {:input args})))
         [task-args func-args] (if (neg? idx)
                                 [args []]
                                 [(take idx args) (drop (inc idx) args)])
         [input params lookup env] (apply task-inputs task task-args)
         f  (-> (-> task :main :fn)
                (wrap-execute task)
                (wrap-input task))
         params (collection/merge-nested (:params task) params)
         result (apply f input params lookup env func-args)]
     (alter-var-root #'*interrupt*
                     (fn [_] false))
     result)))
link
(keys (invoke (task/task :default "ns-interns"
                         {:construct {:env (fn [_] {})
                                      :lookup (fn [_ _] '{std.task []})}
                          :main {:fn (fn [ns _ _ _]
                                       (ns-interns ns))}})
              'std.task.process-test))
=> (contains '[process-test-fn] :in-any-order :gaps-ok)

main-function ^

[func count]
Added 4.0

creates a main function to be used for execution

v 4.0
(defn main-function
  ([func count]
   (let [fcounts (fn/arg-count func)
         fcount  (if-not (empty? fcounts)
                   (apply min fcounts)
                   4)
         args?   (> fcount count)
         main    (cond (= count 4) (fn [input params lookup env & args]
                                     (try
                                       (apply func input params lookup env args)
                                       (catch clojure.lang.ArityException e
                                         ;; Fallback for functions detected as 4-arity but actually 3-arity
                                         (if (= (.-actual e) 4)
                                           (apply func input params env args)
                                           (throw e)))))
                       (= count 3) (fn [input params _ env & args]
                                     (apply func input params env args))
                       (= count 2) (fn [input params _ _ & args]
                                     (apply func input params args))
                       (= count 1) (fn [input _ _ _ & args]
                                     (apply func input args))
                       :else (throw (ex-info "`count` is a value between 1 to 4" {:count count})))]
     [main args?])))
link
(main-function ns-aliases 1)
=> (contains [fn/vargs? false])

select-filter ^

[selector id]
Added 4.0

matches given a range of filters

v 4.0
(defn select-filter
  [selector id]
  (cond (or (fn?  selector)
            (var? selector))
        (f/suppress (selector id))

        (or (string? selector)
            (symbol? selector)
            (keyword? selector))
        (.startsWith (str id) (str selector))

        (f/regexp? selector)
        (boolean (re-find selector (str id)))
        
        (set? selector) (selector id)

        (collection/form? selector)  (every? #(select-filter % id)
                                    selector)
        
        
        (vector? selector) (some #(select-filter % id)
                                 selector)

        :else
        (throw (ex-info "Selector not valid" {:selector selector}))))
link
(select-filter #"ello" 'hello)
=> true
(select-filter #"^ello" 'hello)
=> false

select-inputs ^

[task lookup env selector]
Added 4.0

selects inputs based on matches

v 4.0
(defn select-inputs
  ([task lookup env selector]
   (let [list-fn    (or (-> task :item :list)
                        (throw (ex-info "No `:list` function defined" {:key [:item :list]})))]
     (cond (= selector :all)
           (list-fn lookup env)
           
           :else
           (->> (list-fn lookup env)
                (filter #(select-filter selector %)))))))
link
(select-inputs {:item {:list (fn [_ _] ['code.test 'spirit.common])}}
               {}
               {}
               ['code])
=> ['code.test]

task-inputs ^

[task] [task input] [task input params] [task input params env] [task input params lookup env]
Added 4.0

constructs inputs to the task given a set of parameters

v 4.0
(defn task-inputs
  ([task]
   (let [input-fn (or (-> task :construct :input) (constantly nil))]
     (task-inputs task (input-fn task) task)))
  ([task input]
   (let [input-fn (or (-> task :construct :input) (constantly nil))
         [input params] (cond (map? input)
                              [(input-fn task) input]

                              :else [input {}])]
     (task-inputs task input params)))
  ([task input params]
   (let [env-fn (or (-> task :construct :env) (constantly {}))]
     (task-inputs task input params (env-fn (merge task params)))))
  ([task input params env]
   (let [lookup-fn (or (-> task :construct :lookup) (constantly {}))]
     (task-inputs task input params (lookup-fn task (merge env params)) env)))
  ([task input params lookup env]
   [input params lookup env]))
link
(task-inputs (task/task :default "ns-interns"
                        {:construct {:env (fn [_] {})
                                     :lookup (fn [_ _] '{std.task [a b c]})}
                         :main {:fn ns-interns}})
             'std.task)
=> '[std.task {} {std.task [a b c]} {}]

wrap-execute ^

[f task]
Added 3.0

enables execution of task with transformations

v 3.0
(defn wrap-execute
  ([f task]
   (fn [input params lookup env & args]
     (let [pre-fn    (or (-> task :item :pre) identity)
           post-fn   (or (-> task :item :post) identity)
           output-fn (or (-> task :item :output) identity)
           input  (pre-fn input)
           result (apply f input params lookup env args)
           result (post-fn result)]
       (if (:bulk params)
         [input (res/->result input result)]
         (output-fn result))))))
link
((wrap-execute process-test-fn +task+)
 1 {} {} {})
=> 3

wrap-input ^

[f task]
Added 3.0

enables execution of task with single or multiple inputs

v 3.0
(defn wrap-input
  ([f task]
   (fn [input params lookup env & args]
     (cond (= :list input)
           (let [list-fn  (or (-> task :item :list)
                              (throw (ex-info "No `:list` function defined" {:key [:item :list]})))]
             (list-fn lookup env))

           (or (keyword? input)
               (vector? input)
               (set? input)
               (collection/form? input))
           (let [inputs (select-inputs task lookup env input)]
             (apply bulk/bulk task f inputs params lookup env args))

           :else
           (apply f input params lookup env args)))))
link
((wrap-input process-test-fn +task+)
 1 {} {} {})
=> 2

(let [task (assoc-in +task+ [:item :list] (constantly [1 2 3]))
      f    (wrap-execute process-test-fn task)
      res  ((wrap-input f task) :all {} {} {})]
  (get res 2) => 3
  (get res 3) => 5
  (get res 4) => 7)

bulk ^

[task f inputs {:keys [print package title return], :as params} lookup env & args]
Added 3.0

process and output results for a group of inputs

v 3.0
(defn bulk
  ([task f inputs {:keys [print package title return] :as params} lookup env & args]
   (let [params   (assoc params :bulk true)
         _          (when (and (or (:function print) (:item print) (:result print) (:summary print))
                               title)
                      (print/print-title (if (fn? title)
                                           (title params env)
                                           title)))
         start     (time/time-ms)
         items     (bulk-items task f inputs params lookup env args)
         elapsed   (time/elapsed-ms start)
         warnings  (bulk-warnings params items)
         errors    (bulk-errors params items)
         results   (bulk-results task params items)
         summary   (bulk-summary task params items results warnings errors elapsed)]
     (bulk-package task
                   {:items items
                    :warnings warnings
                    :errors errors
                    :results results
                    :summary summary}
                   (or return :results)
                   package))))
link
(bulk +task+ bulk-test-fn [1 2 3] {:print {}} {} {} {})
=> map?

bulk-display ^

[index-len input-len]
Added 3.0

constructs bulk display options

v 3.0
(defn bulk-display
  ([index-len input-len]
   {:padding 1
    :spacing 1
    :columns [{:id :index :length index-len
               :color #{:blue}
               :align :right}
              {:id :input :length input-len}
              {:id :data  :length 60 :color #{:white}}
              {:id :time  :length 10 :color #{:bold}}]}))
link
(bulk-display 10 20)
=> map?

bulk-errors ^

[{:keys [print], :as params} items]
Added 3.0

outputs errors that have been processed

v 3.0
(defn bulk-errors
  ([{:keys [print] :as params} items]
   (let [errors (filter #(-> % second :status #{:critical :error}) items)]
     (when (and (:result print) (seq errors))
       (print/print "n")
       (print/print-subtitle (format "ERRORS (%s)" (count errors)))
       (print/print "n")
       (print/print-column errors :data #{:error}))
     errors)))
link
(bulk-errors {:print {}} [[1 {:status :error}]])
=> [[1 {:status :error}]]

bulk-items ^

[task f inputs {:keys [print parallel random], :as params} lookup env args]
Added 3.0

processes each item given a input

v 3.0
(defn bulk-items
  ([task f inputs {:keys [print parallel random] :as params} lookup env args]
   (when (:item print)
     (print/print "n")
     (print/print-subtitle (format "ITEMS (%s)" (count inputs))))
   (cond (empty? inputs) []

         :else
         (let [inputs     (if random (shuffle inputs) inputs)
               total      (count inputs)
               index-len  (let [digits (if (pos? total)
                                         (inc (long (Math/log10 total)))
                                         1)]
                            (+ 2 (* 2 digits)))
               input-len  (->> inputs (map (comp count str)) (apply max) (+ 2))
               display-fn (or (-> task :item :display) identity)
               display    (bulk-display index-len input-len)
               context    {:total  total
                           :idxs   (range total)
                           :display display
                           :display-fn display-fn}]
           (if (:item print) (print/print "n"))
           (if parallel
             (bulk-items-parallel f inputs context params lookup env args)
             (bulk-items-single f inputs context params lookup env args))))))
link
(let [results (bulk-items +task+ bulk-test-fn [1 2 3] {:print {}} {} {} {})]
  (and (= (count results) 3)
       (every? (fn [[k v]] (and (number? k) (= (:status v) :return))) results)))
=> true

bulk-items-parallel ^

[f inputs {:keys [idxs], :as context} params lookup env args]
Added 3.0

bulk operation processing in parallel

v 3.0
(defn bulk-items-parallel
  ([f inputs {:keys [idxs] :as context} params lookup env args]
   (->> (pmap (fn [idx input]
                (let [output  (volatile! nil)
                      out-str (env/with-out-str
                                (bulk-process-item f (assoc context
                                                            :idx idx
                                                            :input input
                                                            :output output)
                                                   params lookup env args))]
                  [@output out-str]))
              idxs
              inputs)
        (mapv (fn [[output out-str]]
                (print/print out-str)
                output)))))
link
(let [results (bulk-items-parallel bulk-test-fn [1 2 3]
                                   {:idxs (range 3) :display-fn identity}
                                   {:print {}} {} {} {})]
  (and (= (count results) 3)
       (every? (fn [[k v]] (and (number? k) (= (:status v) :return))) results)))
=> true

bulk-items-single ^

[f inputs {:keys [idxs], :as context} params lookup env args]
Added 3.0

bulk operation processing in single

v 3.0
(defn bulk-items-single
  ([f inputs {:keys [idxs] :as context} params lookup env args]
   (mapv (fn [idx input]
           (bulk-process-item f (assoc context
                                       :idx idx
                                       :input input)
                              params lookup env args))
         idxs
         inputs)))
link
(let [results (bulk-items-single bulk-test-fn [1 2 3]
                                 {:idxs (range 3) :display-fn identity}
                                 {:print {}} {} {} {})]
  (and (= (count results) 3)
       (every? (fn [[k v]] (and (number? k) (= (:status v) :return))) results)))
=> true

bulk-package ^

[task {:keys [items warnings errors results summary], :as bundle} return package]
Added 3.0

packages results for return

v 3.0
(defn bulk-package
  ([task {:keys [items warnings errors results summary] :as bundle} return package]
   (cond (= return :all)
         (bulk-package task bundle #{:items :warnings :errors :results :summary} package)

         (keyword? return)
         (first (vals (bulk-package task bundle #{return} package)))

         :else
         (let [items-fn    (or (-> task :item :output) identity)
               results-fn  (or (-> task :result :output) identity)]
           (reduce (fn [out kw]
                     (cond (#{:summary :warnings :errors} kw)
                           (assoc out kw (get bundle kw))

                           (= :items kw)
                           (cond->> (get bundle kw)
                             :then (map (fn [[key v]] [key (items-fn (:data v))]))
                             (not= package :vector) (into {})
                             :then (assoc out :items))

                           (= :results kw)
                           (cond->> (get bundle kw)
                             :then (map (fn [v] [(:key v) (results-fn (:data v))]))
                             (not= package :vector) (into {})
                             :then (assoc out :results))

                           :else out))
                   {}
                   return)))))
link
(bulk-package +task+ {:items [[1 {:data 1}]]
                      :results [{:key 1 :data 1}]
                      :warnings []
                      :errors []
                      :summary {}}
              :all :map)
=> (contains {:items {1 1} :results {1 1}})

bulk-process-item ^

[f {:keys [idx total input output display display-fn]} {:keys [print], :as params} lookup env args]
Added 3.0

bulk operation processing each item

v 3.0
(defn bulk-process-item
  ([f
    {:keys [idx total input output display display-fn]}
    {:keys [print] :as params} lookup env args]
   (let [start        (System/currentTimeMillis)
         [key result] (try (apply f input params lookup env args)
                           (catch Exception e
                             (print/println ">>" (.getMessage e))
                             (let [end   (System/currentTimeMillis)]
                               [input (res/result {:status :error
                                                   :time (- end start)
                                                   :data :errored})])))
         end    (System/currentTimeMillis)
         result (assoc result :time (- end start))
         {:keys [status data time]} result
         _  (if (:item print)
              (let [index (format "%s/%s" (inc idx) total)
                    item  (if (= status :return)
                            (display-fn data)
                            result)
                    time  (format "%.2fs" (/ time 1000.0))]
                (print/print-row [index key item time] display)))
         _ (if output (vreset! output [key result]))]
     [key result])))
link
(let [[k v] (bulk-process-item bulk-test-fn
                               {:idx 0 :total 1 :input 1 :output (volatile! nil)
                                :display-fn identity}
                               {:print {}}
                               {} {} {})]
  (and (= k 1)
       (= (:status v) :return)
       (= (:data v) 2)))
=> true

bulk-results ^

[task {:keys [print order-by], :as params} items]
Added 3.0

outputs results that have been processed

v 3.0
(defn bulk-results
  ([task {:keys [print order-by] :as params} items]
   (let [ignore-fn (-> task :result :ignore)

         remove-fn (fn [[key {:keys [data status] :as result}]]
                     (or (#{:error :warn :info :critical} status)
                         (and ignore-fn
                              (ignore-fn data))))
         results   (remove remove-fn items)
         _         (when (:result print)
                     (print/print "n")
                     (print/print-subtitle (format "RESULTS (%s)" (count results))))]
     (cond (empty? results)
           []

           :else
           (let [key-fns    (-> task :result :keys)
                 format-fns (-> task :result :format)
                 sort-by-fn (-> task :result :sort-by)
                 outputs  (mapv (fn [[key {:keys [id data] :as result}]]
                                  (->> key-fns
                                       (map (fn [[k f]] [k (f data)]))
                                       (into result)))
                                results)
                 outputs  (if order-by
                            (clojure.core/sort-by order-by outputs)
                            outputs)
                 columns  (-> task :result :columns)
                 display  {:padding 1
                           :spacing 1
                           :columns (prepare-columns columns outputs)}
                 row-keys (map :key columns)]
             (when (:result print)
               (print/print "n")
               (print/print-header row-keys display)
               (mapv (fn [output]
                       (let [row (mapv (fn [k]
                                         (let [data      (get output k)
                                               format-fn (get format-fns k)]
                                           (cond-> data format-fn (format-fn params))))
                                       row-keys)]
                         (print/print-row row display)))
                     outputs))
             outputs)))))
link
(bulk-results +task+ {:print {}} [[1 {:status :return :data 1}]])
=> (every-pred not-empty (partial every? map?))

bulk-summary ^

[task {:keys [print], :as params} items results warnings errors elapsed]
Added 4.1

prints deferred summary notices after the summary output

v 4.1
(defn bulk-summary
  ([task {:keys [print] :as params} items results warnings errors elapsed]
   (let [aggregate-fns (-> task :summary :aggregate)
         finalise-fn   (-> task :summary :finalise)
         cumulative    (apply + (map (comp :time second) items))
         summary (merge {:errors    (count errors)
                         :warnings  (count warnings)
                         :items     (count items)
                         :results   (count results)}
                        (->> aggregate-fns
                             (collection/map-vals (fn [[sel acc init]]
                                           (reduce (fn [out v]
                                                     (let [sv (sel v)]
                                                       (if (nil? sv)
                                                         out
                                                         (acc out sv))))
                                                   init
                                                   results)))))
         summary (if finalise-fn
                   (finalise-fn summary items results)
                   summary)
         display (->> summary
                      (remove (comp #(and (number? %)
                                          (zero? %))
                                    second))
                      (into {}))
          _       (when (:summary print)
                    (print/print "n")
                    (print/print-subtitle (format "SUMMARY %s"
                                                  (str (assoc display
                                                              :cumulative (time/format-ms cumulative)
                                                              :elapsed (time/format-ms elapsed)))))
                    (print/println))
          _       (doseq [line (-> summary meta :std.task/after-summary)]
                    (print/println line))]
      (assoc summary :cumulative cumulative :elapsed elapsed))))
link
(let [calls (atom [])
      task  {:summary {:finalise (fn [& _]
                                   (with-meta {:items 1
                                               :results 1}
                                     {:std.task/after-summary ["saved"]}))}}]
  (with-redefs [std.print/print (fn [& args]
                                  (swap! calls conj (vec (cons :print args))))
                std.print/print-subtitle (fn [& args]
                                           (swap! calls conj (vec (cons :subtitle args))))
                std.print/println (fn [& args]
                                    (swap! calls conj (vec (cons :println args))))]
    (bulk-summary task
                  {:print {:summary true}}
                  [[1 {:time 10 :status :return :data 1}]]
                  []
                  []
                  []
                  100)
    [(-> @calls last first)
     (-> @calls last second)]))
=> [:println "saved"]

bulk-warnings ^

[{:keys [print], :as params} items]
Added 3.0

outputs warnings that have been processed

v 3.0
(defn bulk-warnings
  ([{:keys [print] :as params} items]
   (let [warnings (filter #(-> % second :status (= :warn)) items)]
     (when (and (:result print) (seq warnings))
       (print/print "n")
       (print/print-subtitle (format "WARNINGS (%s)" (count warnings)))
       (print/print "n")
       (print/print-column warnings :data #{:warn}))
     warnings)))
link
(bulk-warnings {:print {}} [[1 {:status :warn}]])
=> [[1 {:status :warn}]]

prepare-columns ^

[columns outputs]
Added 4.0

prepares columns for printing

v 4.0
(defn prepare-columns
  ([columns outputs]
   (mapv (fn [{:keys [length key] :as column}]
           (let [id    key
                 length (cond (number? length)
                              length

                              :else
                              (->> outputs (map key) (map (comp count str)) (apply max) (+ 2)))]
             (assoc column :id key :length length)))
         columns)))
link
(prepare-columns [{:key :name}
                  {:key :data}]
                 [{:name "Chris"
                   :data "1"}
                  {:name "Bob"
                   :data "100"}])
=> [{:key :name, :id :name, :length 7}
    {:key :data, :id :data, :length 5}]

5    std.task Guide

std.task defines a protocol and structure for executing operations, managing their inputs/outputs, and formatting their results. It is the engine behind code.manage commands.

5.1    Core Concepts

  • Task: An invokable object with a defined lifecycle.n- Template: A set of defaults for a class of tasks (e.g., :namespace, :file).n- Process: The underlying execution logic (handling parallel/serial execution, argument parsing).

5.2    Usage

5.2.1    Defining a Task

(require '[std.task :as task])\n\n;; Simple task\n(task/deftask my-task\n  {:template :default\n   :main {:fn +}\n   :doc \"Adds numbers\"})\n\n(my-task 1 2) ;; => 3

5.2.2    Scenarios

5.2.2.1    1. Defining a Custom Task Template

If you have a set of tasks that share common behavior (e.g., they all operate on database records), you can define a template.

;; Define defaults for :db-task\n(defmethod task/task-defaults :db-task\n  [_]\n  {:main {:arglists '([id] [id options])}\n   :result {:columns [{:key :id :align :left}\n                      {:key :status :align :center}]}})\n\n;; Create a task using this template\n(task/deftask get-user\n  {:template :db-task\n   :main {:fn (fn [id] {:id id :status :active})}\n   :doc \"Fetches a user\"})

5.2.2.2    2. Parallel Processing

Tasks can be configured to run in parallel, which is useful for operations over many items (like files or namespaces).

(task/deftask process-files\n  {:template :default\n   :params {:parallel true\n            :title \"PROCESSING FILES\"}\n   :main {:fn (fn [file] (slurp file))} ;; Simplified example\n   })

When invoked with a collection, the task runner (via std.task.process) will utilize threads if :parallel is set.

5.2.2.3    3. Customizing Output/Result

You can control how the result is displayed, which is crucial for CLI tools.

(task/deftask analyze-data\n  {:template :default\n   :main {:fn (fn [x] {:score (rand-int 100) :name x})}\n\n   ;; Define output columns\n   :result {:keys {:count count} ;; Summary key\n            :columns [{:key :name :length 20 :align :left}\n                      {:key :score :length 10 :align :right :color #{:bold}}]}})\n\n;; Invocation might look like:\n;; (analyze-data [\"A\" \"B\"])\n;; => Prints table with Name and Score

5.2.2.4    4. Item Processing Configuration

The :item key allows transformation of individual inputs or outputs.

(task/deftask list-sorted\n  {:template :default\n   :main {:fn identity}\n   :item {:post (comp vec sort)} ;; Sort the output of the main function\n   })

5.2.2.5    5. Handling Command Line Arguments

If you are building a CLI entry point (like code.manage/-main), process-ns-args helps parse arguments.

(defn -main [& args]\n  (let [opts (task/process-ns-args args)]\n    ;; opts will look like {:only \"my.ns\" :verbose true}\n    ;; if called as: ... :only \"my.ns\" :verbose\n    ...))

6    std.task: A Comprehensive Summary

The std.task module in foundation-base provides a framework for defining, executing, and managing tasks, particularly focusing on bulk operations and processing items in a structured manner. It offers utilities for handling task inputs, executing functions with transformations, and presenting results, warnings, errors, and summaries in a user-friendly format.

The module is organized into two main sub-namespaces:

6.1    std.task.bulk

This namespace provides the core logic for performing bulk operations, processing items, and presenting the results. It's designed to handle collections of inputs, execute a given function for each, and then aggregate and display the outcomes.

  • bulk-display [index-len input-len]: Constructs display options for bulk output, defining column widths, colors, and alignment for index, input, data, and time.n bulk-process-item [f context params lookup env args]: Processes a single item within a bulk operation. It executes the function f with the item's input, captures exceptions, measures execution time, and optionally prints item-specific output.n bulk-items-parallel [f inputs context params lookup env args]: Processes a collection of inputs in parallel using pmap.n bulk-items-single [f inputs context params lookup env args]: Processes a collection of inputs sequentially.n bulk-items [task f inputs params lookup env args]: The main function for orchestrating item processing. It handles input shuffling (if random is true), calculates display lengths, and dispatches to either parallel or single-threaded processing based on parallel parameter. It also prints a subtitle for the items section.n bulk-warnings [params items]: Filters processed items for warnings (status :warn) and optionally prints them in a formatted column.n bulk-errors [params items]: Filters processed items for errors (status :critical or :error) and optionally prints them in a formatted column.n prepare-columns [columns outputs]: Prepares column definitions for printing, dynamically calculating column lengths based on output data if not explicitly provided.n bulk-results [task params items]: Filters and sorts processed items to present the final results. It applies ignore functions, formats output based on task definitions, and optionally prints results in a formatted table.n bulk-summary [task params items results warnings errors elapsed]: Generates and optionally prints a summary of the bulk operation, including counts of items, results, warnings, errors, and cumulative/elapsed times. It can also aggregate custom metrics defined in the task.n bulk-package [task bundle return package]: Packages the results of the bulk operation into a specified format (e.g., :all, :results, :summary, :items) and structure (e.g., :vector, :map).n* bulk [task f inputs params lookup env & args]: The top-level function for executing a bulk task. It orchestrates the entire process: printing titles, processing items, collecting warnings/errors/results, generating a summary, and packaging the final output.

6.2    std.task.process

This namespace provides utilities for constructing and invoking tasks, including handling function arity, input selection, and execution wrapping.

  • *interrupt*: A dynamic var (atom) used as a flag to signal task interruption.n main-function [func count]: Creates a wrapper function that adapts a given func to a standardized task execution signature (taking input, params, lookup, env, and args). It handles different arities of the original function.n select-filter [selector id]: A flexible filtering mechanism that matches an id against various selector types (functions, vars, strings, symbols, keywords, regex, sets, forms, vectors).n select-inputs [task lookup env selector]: Selects inputs for a task based on a selector. If selector is :all, it uses the task's :item :list function to get all possible inputs. Otherwise, it filters the list of inputs using select-filter.n wrap-execute [f task]: Wraps a task's execution function f with pre- and post-processing steps defined in the task's :item configuration. It also handles converting results to std.lib.result format for bulk operations.n wrap-input [f task]: Wraps a task's execution function f to handle different input types. If the input is a selector (keyword, vector, set, form), it triggers a bulk operation using bulk/bulk. If the input is :list, it returns the list of all possible inputs. Otherwise, it executes f with the single input.n task-inputs [task & args]: Constructs the complete set of inputs (input, params, lookup, env) for a task based on its configuration and provided arguments.n* invoke [task & args]: The primary function for executing a task. It prepares the task's inputs, wraps the main function with execution and input handling logic, and then executes it. It also manages the *interrupt* flag.

Overall Importance:

The std.task module is crucial for building automated workflows, command-line tools, and data processing pipelines within the foundation-base project. It provides:

  • Structured Task Definition: Allows tasks to be defined with clear configurations for input listing, execution logic, pre/post-processing, and result presentation.n Efficient Bulk Operations: Facilitates processing large collections of data or performing repetitive actions, with options for parallel execution and detailed output.n User-Friendly Output: Generates formatted tables for results, warnings, and errors, along with comprehensive summaries, enhancing the user experience for CLI tools.n Extensibility: The modular design with configurable functions and wrappers allows for easy customization and extension of task behavior.n Robust Error Handling: Integrates with std.lib.result to capture and report execution outcomes, including exceptions.

By offering a robust and flexible task management framework, std.task significantly contributes to the foundation-base project's ability to automate complex software engineering tasks and manage its multi-language development ecosystem effectively.

Related

jvm.artifactdocs

artifact coordinates and paths

jvm.classloaderdocs

classpath and classloaders

jvm.depsdocs

JVM dependency resolution

jvm.monitordocs

JVM metrics

jvm.namespacedocs

namespace management

jvm.protocoldocs

JVM extension protocols

jvm.reflectdocs

Java reflection

jvm.requiredocs

dependency-aware requiring