std.timeseries

journals, intervals, ranges, computation, and processing

std.timeseries is a higher-level standard library family in foundation-base. This page explains when to use it, how it fits internally, and where to find the API surface.

1    Motivation

Use this layer when application or tooling code needs the behavior described by the page title without reaching directly into implementation namespaces. The top-level namespace is the starting point; subnamespaces expose more focused building blocks.

2    How to use it

Require the top-level namespace for common workflows, then move to subnamespaces when you need a lower-level primitive. Existing tests under test/std/timeseries and test/std/timeseries_test.clj are the best executable examples for edge cases.

create a journal and add entries

(def sensor-journal
  (ts/journal {:meta {:time {:unit :ms :key :timestamp}
                      :entry {:flatten true}}}))

(-> sensor-journal
    (ts/add-entry {:temp 20.5 :humidity 50 :timestamp 1600000000000})
    (count))
=> 1

select entries from a journal

(let [j (-> (ts/journal {:meta {:time {:unit :ms :key :timestamp}}})                       (ts/add-entry {:value 1 :timestamp 0})
            (ts/add-entry {:value 2 :timestamp 1000})
            (ts/add-entry {:value 3 :timestamp 2000}))]
  (->> (ts/select j {:range [0 2000]})
       (map :value)))
=> [1 2 3]

3    Internal usage

This library family is used across source, tests, generated examples, and docs tooling. During detailed documentation passes, collect concrete usage with code.manage/find-usages and code.manage/locate-code, then keep only high-signal examples in the page narrative.

4    API

add-bulk ^

[{:keys [meta template], :as journal} entries]
Added 3.0

adds multiple entries to the journal

v 3.0
(defn add-bulk
  ([{:keys [meta template] :as journal} entries]
   (let [{:keys [key unit]} (:time meta)
         {:keys [flatten pre-flattened]} (:entry meta)
         entries (if (and flatten (not pre-flattened))
                   (let [{:keys [flat]} (or @template
                                            (create-template journal (first entries)))]
                     (map flat entries))
                   entries)
         entries (map #(add-time % key unit) entries)]
     (update-journal-bulk journal entries))))
link
;; Test for bulk + current over the limit
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 3
              :entries  [{:a 2 :s/time 2} {:a 3 :s/time 3}]})
    (add-bulk  [{:a 4 :s/time 4} {:a 5 :s/time 5}])
    ((juxt :entries journal-entries)))
[[{:a 5, :s/time 5}]
 [{:a 3, :s/time 3} {:a 4, :s/time 4} {:a 5, :s/time 5}]]

;; Test for bulk over the limit
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 3
              :entries  [{:a 2 :s/time 2}]})
    (add-bulk  [{:a 3 :s/time 3} {:a 4 :s/time 4} {:a 5 :s/time 5}])
    ((juxt :entries journal-entries)))
=> [() [{:a 3, :s/time 3} {:a 4, :s/time 4} {:a 5, :s/time 5}]]

;; Test for bulk + current under limit
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 3
              :entries  [{:a 2 :s/time 2}]})
    (add-bulk  [{:a 3 :s/time 3} {:a 4 :s/time 4}])
    ((juxt :entries journal-entries)))
=> [[{:a 2, :s/time 2} {:a 3, :s/time 3} {:a 4, :s/time 4}]
    [{:a 2, :s/time 2} {:a 3, :s/time 3} {:a 4, :s/time 4}]]

add-entry ^

[{:keys [meta template], :as journal} entry]
Added 3.0

adds an entry to the journal

v 3.0
(defn add-entry
  ([{:keys [meta template] :as journal} entry]
   (let [{:keys [key unit]} (:time meta)
         {:keys [flatten pre-flattened]} (:entry meta)
         entry (if (and flatten (not pre-flattened))
                 (let [{:keys [flat]} (or @template
                                          (create-template journal entry))]
                   (flat entry))
                 entry)
         entry (add-time entry key unit)]
     (update-journal-single journal entry))))
link
(-> (journal {:meta {:head {:range [0 3]}}})
    (add-entry {:start 1 :output {:value 1}})
    journal-info)
=> (contains-in {:id string?
                 :order :desc, :count 1,
                 :duration "0ms",
                 :template [:output.value :start],
                 :head [{:start 1, :output.value 1, :s/time string?}]})

create-template ^

[entry]
Added 3.0

creates a template for entry

v 3.0
(defn create-template
  ([entry]
   (let [tmpl    (raw-template entry)
         flat-fn (flat-fn tmpl)
         nest-fn (nest-fn tmpl)]
     {:raw tmpl :nest nest-fn :flat flat-fn})))
link
(create-template  {:data {:in  ""
                          :out "ABC"}})
=> (contains {:raw {:empty {:data.in "", :data.out ""}
                    :nest {:data {:in "{{data.in}}", :out "{{data.out}}"}}
                    :flat {:data.in "{{data.in}}", :data.out "{{data.out}}"}
                    :types {:data.in java.lang.String, :data.out java.lang.String}}
              :nest fn?
              :flat fn?})

derive ^

[{:keys [meta], :as journal} {:keys [range sample transform], :as params}]
Added 3.0

derives the journal given a select statement

v 3.0
(defn derive
  ([{:keys [meta] :as journal} {:keys [range sample transform]
                                :as params}]
   (let [{:keys [order]} (:time meta)
         subentries (select journal (dissoc params :series))
         subentries (case order
                      :asc (vec subentries)
                      :desc (apply list subentries))]
     (cond-> journal
       :then (assoc :entries subentries)
       transform (assoc-in [:meta :entry :flatten] true)))))
link
(class (derive (journal {:entries [{:a 1 :s/time 1} {:a 2 :s/time 2}]}) {}))
=> std.timeseries.journal.Journal

journal ^

[] [{:keys [id meta entries previous], :as m, :or {id (f/sid)}}]
Added 3.0

creates a new journal

v 3.0
(defn journal
  ([]
   (journal {}))
  ([{:keys [id meta entries previous] :as m
     :or {id (f/sid)}}]
   (let [{:keys [time] :as meta} (collection/merge-nested (:meta +defaults+) meta)
         entries-fn (case (:order time)
                      :desc  #(apply list %)
                      :asc #(apply vector %))]
     (-> m
         (assoc :id id
                :meta meta
                :template (volatile! nil)
                :entries (entries-fn entries)
                :previous (entries-fn previous))
         (types/ :strict)
         (map->Journal)))))
link
(class (journal {:meta {:time {:unit :s :format "HH:mm:ss"}
                        :entry {:flatten false}
                        :head {:range [0 3]}}}))
=> std.timeseries.journal.Journal

merge ^

[] [j0] [j0 j1]
Added 3.0

merges two journals of the same type together

v 3.0
(defn merge
  ([])
  ([j0] j0)
  ([j0 j1]
   (if-not (and (= (:meta j0)
                   (:meta j1))
                (= (:raw @(:template j0))
                   (:raw @(:template j1))))
     (throw (ex-info "Meta and Template requires to be the same"))
     (let [{:keys [order key]} (-> j0 :meta :time)
           comp-fn (common/order-comp order)
           entries (cond->> (merge-sorted [(:entries j0) (:entries j1)] key comp-fn)
                     (= order :asc) (vec)
                     (= order :desc) (apply list))]
       (assoc j0 :entries entries)))))
link
(let [j1 (journal {:entries [{:a 1 :s/time 1}]})
      j2 (journal {:entries [{:a 2 :s/time 2}]})]
  (class (merge j1 j2)))
=> std.timeseries.journal.Journal

process ^

[arr {:keys [time transform computes], :as opts}]
Added 3.0

processes time series

v 3.0
(defn process
  ([arr {:keys [time transform computes] :as opts}]
   (types/ opts :strict)
   (let [{:keys [key order]} time
         order  (or order
                    (if (< (key (first arr))
                           (key (last arr)))
                      :asc
                      :desc))
         time-opts (common/parse-time-expr (assoc time :order order))
         opts   (assoc opts :time time-opts)
         arr    (range/process-range arr opts)
         arr    (process-transform arr opts)
         output (process-compute arr opts)]
     output)))
link
(process (->> @(first (vals (:active @(:runtime (:collector @*instance*)))))
              :results
              (mapv (fn [{:keys [output] :as m}]
                      (merge (dissoc m :output) output))))

         {:time  {:key :start :unit :ns}
          :range   [0 :end]
          :sample  :100ms
          :transform {:interval :20ms
                      :sample 2
                      :time    {:aggregate :first}
                      :default {}
                      :custom  [{:keys [:bench.stats.lag]
                                 :sample 10
                                 :aggregate :max}
                                {:keys [:executor.type
                                        :executor.running
                                        :bench.call.duration.last]
                                 :aggregate :last}
                                {:keys [:bench.stats.time]
                                 :sample 10
                                 :aggregate :max}]}
          :compute {:lag-norm [:s/norm :bench.stats.lag]
                    :lag :bench.stats.lag}})

select ^

[journal] [{:keys [meta], :as journal} {:keys [range sample transform series template compute], :or {range :all}}]
Added 3.0

selects data ferom the journal

v 3.0
(defn select
  ([journal]
   (select journal {}))
  ([{:keys [meta] :as journal} {:keys [range sample transform series template compute]
                                :or {range :all}}]
   (let [entries  (journal-entries journal)
         template (get-template journal)
         transform (if transform
                     (cond-> transform
                       (-> meta :entry :flatten) (assoc :skip true)))
         entries (process/process entries
                                  (cond-> {:time (:time meta)
                                           :range range
                                           :sample sample
                                           :template template
                                           :compute compute}
                                    transform (assoc :transform transform)))]
     (cond-> entries
       series (select-series series)))))
link
(def -jnl- (->> [0 (journal {:meta {:time   {:key :start :order :desc}
                                    :head   {:range [0 3]}
                                    :hide   #{:meta :template :id}}})]
                (iterate (fn [[t journal]]
                           (let [t (+ t 1000000000 (rand-int 100000000))]
                             [t (add-entry journal {:start t
                                                    :output {:value (Math/sin
                                                                     (+ (/ t 100000000000)
                                                                        (rand)))}})])))
                (map second)
                (take 1000)
                (last)))

(select -jnl- {:range [:1m :5m] :sample 10})
=> (fn [entries]
     (= 10 (count entries)))

(first (select -jnl- {:range [:1m :5m] :sample 10}))
=> (contains {:start integer? :output.value number?})

update-meta ^

[journal meta]
Added 3.0

updates journal meta. used for display

v 3.0
(defn update-meta
  ([journal meta]
   (update journal :meta collection/merge-nested meta)))
link
(-> (journal {})
    (update-meta {:a 1})
    :meta)
=> (contains {:a 1})

+default+ ^

NONE
(def +default+)
link

cluster ^

[arr n f]
Added 3.0

clusters a space with an aggregate function

v 3.0
(defn cluster
  ([arr n f]
   (let [len   (count arr)
         squot (quot len n)
         squot (if (zero? squot) 1 squot)]
     (map f (partition squot arr)))))
link
(cluster (range 1 200) 10  math/mean)
=> [10 29 48 67 86 105 124 143 162 181]

create-template ^

[entry]
Added 3.0

creates a template for entry

v 3.0
(defn create-template
  ([entry]
   (let [tmpl    (raw-template entry)
         flat-fn (flat-fn tmpl)
         nest-fn (nest-fn tmpl)]
     {:raw tmpl :nest nest-fn :flat flat-fn})))
link
(create-template  {:data {:in  ""
                          :out "ABC"}})
=> (contains {:raw {:empty {:data.in "", :data.out ""}
                    :nest {:data {:in "{{data.in}}", :out "{{data.out}}"}}
                    :flat {:data.in "{{data.in}}", :data.out "{{data.out}}"}
                    :types {:data.in java.lang.String, :data.out java.lang.String}}
              :nest fn?
              :flat fn?})

duration ^

[arr {:keys [key unit order]}]
Added 3.0

calculates the duration from array and options

v 3.0
(defn duration
  ([arr {:keys [key unit order]}]
   (let [t0 (key (first arr))
         t1 (key (last arr))]
     (to-ms (math/abs (- t0 t1)) unit))))
link
(duration [1 2 3 4] {:key identity :unit :m})
=> 180000

flat-fn ^

[template]
Added 3.0

creates a nested to flat transform

v 3.0
(defn flat-fn
  ([template]
   (collection/transform-fn template [:nest :flat])))
link
(def transform-fn (flat-fn (raw-template {:a {:b {:c 1}}})))

(transform-fn {:a {:b {:c 10}}})
=> {:a.b.c 10}

from-ms ^

[ms to]
Added 3.0

converts ms time into a given unit

v 3.0
(defn from-ms
  (^long [ms to]
   (case to
     :ns (* ms 1000000)
     :us (* ms 1000)
     :ms ms
     :s  (quot ms 1000)
     :m  (quot ms (* 60 1000))
     :h  (quot ms (* 60 60 1000))
     :d  (quot ms (* 24 60 60 1000)))))
link
(from-ms 100 :ns)
=> 100000000

from-ns ^

[ns to]
Added 3.0

converts a time from ns

v 3.0
(defn from-ns
  (^long [ns to]
   (case to
     :ns ns
     :us (quot ns 1000)
     (from-ms (to-ms ns :ns) to))))
link
(from-ns 1000000 :ms)
=> 1

linspace ^

[arr n]
Added 3.0

takes the linear space of n samples

v 3.0
(defn linspace
  ([arr ^long n]
   (let [len (count arr)]
     (if (<= len n)
       arr
       (case n
         1 [(last arr)]
         2 [(first arr) (last arr)]
         (let [len  (dec len)
               spc  (quot len (dec n))]
           (-> (take-nth spc arr)
               (butlast)
               (vec)
               (conj (last arr)))))))))
link
(linspace (range 100) 10)
=> [0 11 22 33 44 55 66 77 88 99]

make-empty ^

[entry]
Added 3.0

creates an empty entry given structure

v 3.0
(defn make-empty
  ([entry]
   (collection/map-vals (fn [val]
                 (cond (number? val) 0
                       (string? val) ""
                       :else nil))
               entry)))
link
(make-empty {:a 1 :b "hello" :c :world})
=> {:a 0, :b "", :c nil}

nest-fn ^

[template]
Added 3.0

creates a nested to flat transform

v 3.0
(defn nest-fn
  ([template]
   (collection/transform-fn template [:flat :nest])))
link
(def transform-fn (nest-fn (raw-template {:a {:b {:c 1}}})))

(transform-fn {:a.b.c 10})
=> {:a {:b {:c 10}}}

order-comp ^

NONE
(def order-comp    (order-fn [< >]))
link

order-comp-eq ^

NONE
(def order-comp-eq (order-fn [<= >=]))
link

order-flip ^

[order]
Added 3.0

returns the opposite of the order

v 3.0
(defn order-flip
  ([order]
   (case order :asc :desc :desc :asc)))
link
(order-flip :asc)
=> :desc

order-fn ^

[[asc desc]]
Added 3.0

creates a function that returns or flips the input

v 3.0
(defn order-fn
  ([[asc desc]]
   (fn order-function
     ([order]
      (order-function order false))
     ([order flip]
      (cond-> order
        flip  (order-flip)
        :then (case :asc asc :desc desc))))))
link
(mapv (order-fn [:up :down])
      [:asc :desc]
      [false true])
=> [:up :up]

order-fns ^

[order] [order flip]
Added 3.0

returns commonly used functions

v 3.0
(defn order-fns
  ([order]
   (order-fns order false))
  ([order flip]
   {:comp-fn    (order-comp order flip)
    :comp-eq-fn (order-comp-eq order flip)
    :op-fn (order-op order flip)}))
link
(order-fns :asc)
=> {:comp-fn <, :comp-eq-fn <=, :op-fn +}

order-op ^

NONE
(def order-op      (order-fn [+ -]))
link

parse-sample-expr ^

[sample time-opts]
Added 3.0

parses a sample expression

v 3.0
(defn parse-sample-expr
  ([sample time-opts]
   (let [opts (cond (map? sample) sample

                    (vector? sample)
                    (if (or (keyword? (first sample))
                            (string? (first sample)))
                      {:size [0 sample] :strategy :range}
                      (let [[size strategy m] sample]
                        (merge {:size size :strategy (or strategy :linear)}
                               m)))

                    (integer? sample)
                    (parse-sample-expr [sample] time-opts)

                    (or (keyword? sample)
                        (string? sample))
                    {:size sample :strategy :interval :default {:aggregate :first}})
         parser (sampling-parser (:strategy opts))]
     (if parser
       (parser opts time-opts)
       opts))))
link
(parse-sample-expr [10] {})
=> {:size 10, :strategy :linear}

parse-time ^

[s] [s unit]
Added 3.0

parses a string or keyword time representation

v 3.0
(defn parse-time
  ([s]
   (parse-time s :ms))
  ([s unit]
   (let [s (core/strn s)]
     (from-ms (or (if-let [t (time/parse-ns s)]
                    (/ t 1000000))
                  (time/parse-ms s))
              unit))))
link
(parse-time :0.1m :us)
=> 6000000

(parse-time :0.5ms :us)
=> 500

parse-time-expr ^

[m]
Added 3.0

parses a time expression

v 3.0
(defn parse-time-expr
  ([m]
   (let [{:keys [unit interval] :as m} (merge (:time +default+) m)
         interval (cond (number? interval)
                        interval

                        (or (string? interval)
                            (keyword? interval))
                        (parse-time interval unit))]
     (assoc m :interval interval))))
link
(parse-time-expr {:interval "0.5ms"})
=> {:key identity, :unit :ms, :order :asc, :interval 0}

process-sample ^

[arr {:keys [size strategy every], :as m} time-opts]
Added 3.0

process sample given sampling function

v 3.0
(defn process-sample
  ([arr {:keys [size strategy every] :as m} time-opts]
   (if (nil? m)
     arr
     (let [{:keys [order key sort]} time-opts
           comp-fn (order-comp order)
           samples (case strategy
                     :linear (linspace arr size)
                     :random (take size (repeatedly #(rand-nth arr)))
                     :start  (take size (take-nth (or every 1) arr))
                     :end    (take size (take-nth (or every 1) (reverse arr)))
                     (let [f (sampling-fn strategy)]
                       (f arr m time-opts)))]
       (if (false? sort)
         samples
         (sort-by key comp-fn samples))))))
link
(process-sample (range 100)
                (parse-sample-expr [10 :random] {})
                {:key identity :order :asc})
;; (4 20 39 51 51 60 70 72 89 96)
=> coll?

;; extended sampling function from range
(process-sample (range 100)
                (parse-sample-expr [[0 :10ms] :range] {:key identity :order :asc :unit :ms})
                {:key identity :order :asc :unit :ms})
=> [0 1 2 3 4 5 6 7 8 9 10]

raw-template ^

[entry]
Added 3.0

creates a template for nest/flat mapping

v 3.0
(defn raw-template
  ([entry]
   (binding [core/*sep* "."]
     (let [dflat  (collection/tree-flatten entry)
           dtypes (collection/map-vals class dflat)
           tflat  (collection/map-entries (fn [[k v]]
                                   [k (str "{{" (core/strn k) "}}")]) dflat)
           tnest  (collection/tree-nestify tflat)]
       {:nest tnest :flat tflat :types dtypes :empty (make-empty dflat)}))))
link
(raw-template {:a {:b 1 :c "2"}})
=> {:nest {:a {:b "{{a.b}}",
               :c "{{a.c}}"}},
    :flat {:a.b "{{a.b}}",
           :a.c "{{a.c}}"},
    :types {:a.b java.lang.Long,
            :a.c java.lang.String}
    :empty {:a.b 0, :a.c ""}}

sampling-fn ^

Added 3.0

extensible sampling function

v 3.0
(defmulti sampling-fn
  identity)
link
(sampling-fn :unknown)
=> (throws)

sampling-parser ^

Added 3.0

extensible parser function

v 3.0
(defmulti sampling-parser
  identity)
link
(sampling-parser :unknown)
=> nil

to-ms ^

[ms from]
Added 3.0

converts a time to ms time

v 3.0
(defn to-ms
  (^long [ms from]
   (case from
     :ns (quot ms 1000000)
     :us (quot ms 1000)
     :ms ms
     :s  (* ms 1000)
     :m  (* ms 60 1000)
     :h  (* ms 60 60 1000)
     :d  (* ms 24 60 60 1000))))
link
(to-ms 10000 :ns)
=> 0

(to-ms 10000 :s)
=> 10000000

+aggregation-forms+ ^

NONE
(def +aggregation-forms+
  (collection/map-entries (fn [[k _]]
                   [(keyword "s" (name k)) (list k `+aggregations+)])
                 +aggregations+))
link

+aggregations+ ^

NONE
(def +aggregations+
  (collection/map-vals wrap-not-nil
              {:first    first
               :last     last
               :middle   middle-fn
               :mean     (comp float math/mean)
               :sum      #(apply + %)
               :max      max-fn
               :min      min-fn
               :range    range-fn
               :stddev   math/stdev
               :mode     math/mode
               :median   math/median
               :skew     math/skew
               :variance math/variance
               :random   rand-nth}))
link

+templates+ ^

NONE
(def +templates+)
link

apply-template ^

[[k & rest]]
Added 3.0

applies the template

v 3.0
(defn apply-template
  ([[k & rest]]
   (apply (get +templates+ k) rest)))
link
(= (apply-template [:s/norm :bench.stats.lag])
   (list `/ :bench.stats.lag [:s/max :bench.stats.lag]))
=> true

compile ^

[m]
Added 3.0

complies a map of expressions

v 3.0
(defn compile
  ([m]
   (collection/map-vals (comp eval compile-single) m)))
link
(compile '{:diff (- [:s/norm :bench.stats.time]
                    [:s/norm :bench.stats.lag])})
=> map?

compile-aggregates ^

[aggregates]
Added 3.0

compiles the aggregates

v 3.0
(defn compile-aggregates
  ([aggregates]
   (mapcat (fn [[sym [k form]]]
             [sym `(~(get +aggregation-forms+ k)
                    (filter identity ~(compile-form form)))])
           aggregates)))
link
(compile-aggregates '{v1 [:s/max :bench.stats.lag]})
=> seq?

compile-form ^

[form]
Added 3.0

compiles the entire form

v 3.0
(defn compile-form
  ([form]
   (let [aggregates  (volatile! {})
         single-form (doall (walk/prewalk (fn [x]
                                            (cond (and (keyword? x)
                                                       (nil? (namespace x)))
                                                  (compile-keyword x)

                                                  (and (vector? x)
                                                       (keyword? (first x))
                                                       (= "s" (namespace (first x))))
                                                  (do (cond (template? x)
                                                            (apply-template x)

                                                            :else
                                                            (let [sym (gensym)]
                                                              (vswap! aggregates assoc sym x)
                                                              sym)))

                                                  :else x))
                                          form))
         map-form   `(map (fn [~'output] ~single-form) ~'outputs)
         aggregate-forms (compile-aggregates @aggregates)]
     (if (empty? aggregate-forms)
       map-form
       `(let [~@aggregate-forms] ~map-form)))))
link
(compile-form '(- [:s/norm :bench.stats.time] [:s/norm :bench.stats.lag]))
=> seq?

compile-keyword ^

[k]
Added 3.0

compiles a single keyword

v 3.0
(defn compile-keyword
  ([k]
   `(get ~'output (keyword ~(name k)))))
link
(compile-keyword :bench.stats.lag)
=> `(get ~'output (keyword "bench.stats.lag"))

compile-single ^

[compute]
Added 3.0

complise a single fn form

v 3.0
(defn compile-single
  ([compute]
   `(fn [~'outputs] ~(compile-form compute))))
link
(eval (compile-single '(- [:s/norm :bench.stats.time] [:s/norm :bench.stats.lag])))
=> fn?

compute ^

[arr exprs]
Added 3.0

computes additional values given array

v 3.0
(defn compute
  ([arr exprs]
   (let [compute-fns   (compile exprs)
         compute-keys  (key-order exprs)]
     (reduce (fn [arr k]
               (let [f  (get compute-fns k)
                     result (f arr)]
                 (->> (map (fn [e] [k e]) result)
                      (map conj arr))))
             arr
             compute-keys))))
link
(-> (compute [{:start 10 :output.value 1}
              {:start 11 :output.value 4}
              {:start 12 :output.value 1}
              {:start 13 :output.value 6}
              {:start 14 :output.value 10.1}]
             {:t            '(* :start 100000000)
              :t0           [:s/adj :t]
              :t1           [:s/inv :t0]
              :output.norm  [:s/norm :output.value]})
    first)
=> {:start 10, :output.value 1, :t 1000000000, :t0 0, :t1 400000000, :output.norm 0.09900990099009901}

key-order ^

[m]
Added 3.0

v 3.0
(defn key-order
  ([m]
   (-> m
       (key-references)
       (sort/topological-sort))))
link
(key-order {:output.norm  [:s/norm :output.value]
            :t1           [:s/inv :t0]
            :t0           [:s/adj :t]
            :t            '(* :start 100000000)})
=> [:t :t0 :t1 :output.norm]

key-references ^

[m]
Added 3.0

finds all references for the expr map

v 3.0
(defn key-references
  ([m]
   (let [ks (set (keys m))]
     (collection/map-vals (fn [form]
                   (let [store (volatile! #{})]
                     (walk/postwalk (fn [x]
                                      (if (and (keyword? x)
                                               (ks x))
                                        (vswap! store conj x))
                                      x)
                                    form)
                     @store))
                 m))))
link
(key-references {:t            '(* :start 100000000)
                 :t0           [:s/adj :t]
                 :t1           [:s/inv :t0]
                 :output.norm  [:s/norm :output.value]})
=> {:t #{}, :t0 #{:t}, :t1 #{:t0}, :output.norm #{}}

max-fn ^

[arr]
Added 3.0

max function accepting array

v 3.0
(defn max-fn
  ([arr]
   (if-not (empty? arr)
     (apply max arr))))
link
(max-fn [1 2 3])
=> 3

middle-fn ^

[arr]
Added 3.0

middling function

v 3.0
(defn middle-fn
  ([arr]
   (let [len (count arr)]
     (nth arr (long (/ len 2))))))
link
(middle-fn [1 2 3])
=> 2

min-fn ^

[arr]
Added 3.0

min function accepting array

v 3.0
(defn min-fn
  ([arr]
   (if-not (empty? arr)
     (apply min arr))))
link
(min-fn [1 2 3])
=> 1

range-fn ^

[arr]
Added 3.0

range function accepting array

v 3.0
(defn range-fn
  ([arr]
   (if-not (empty? arr)
     (- (apply max arr) (apply min arr)))))
link
(range-fn [1 2 3])
=> 2

template? ^

[[k & rest]]
Added 3.0

checks if vector is a template

v 3.0
(defn template?
  ([[k & rest]]
   (contains? +templates+ k)))
link
(template? [:s/norm 1])
=> true

wrap-not-nil ^

[f]
Added 3.0

ensures no null values

v 3.0
(defn wrap-not-nil
  ([f]
   (fn [& args]
     (let [args (filter (complement nil?) args)]
       (apply f args)))))
link
((wrap-not-nil +) 1 nil)
=> 1

+defaults+ ^

NONE
(def +defaults+)
link

->Journal ^

[id meta template entries limit previous]

NONE
(impl/defimpl Journal [id meta template entries limit previous]
  :string  journal-string
  :invoke  journal-invoke
  :final   true

  clojure.lang.IDeref
  (deref [journal] (:entries journal)))
link

Journal ^

NONE
(impl/defimpl Journal [id meta template entries limit previous]
  :string  journal-string
  :invoke  journal-invoke
  :final   true

  clojure.lang.IDeref
  (deref [journal] (:entries journal)))
link

add-bulk ^

[{:keys [meta template], :as journal} entries]
Added 3.0

adds multiple entries to the journal

v 3.0
(defn add-bulk
  ([{:keys [meta template] :as journal} entries]
   (let [{:keys [key unit]} (:time meta)
         {:keys [flatten pre-flattened]} (:entry meta)
         entries (if (and flatten (not pre-flattened))
                   (let [{:keys [flat]} (or @template
                                            (create-template journal (first entries)))]
                     (map flat entries))
                   entries)
         entries (map #(add-time % key unit) entries)]
     (update-journal-bulk journal entries))))
link
;; Test for bulk + current over the limit
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 3
              :entries  [{:a 2 :s/time 2} {:a 3 :s/time 3}]})
    (add-bulk  [{:a 4 :s/time 4} {:a 5 :s/time 5}])
    ((juxt :entries journal-entries)))
[[{:a 5, :s/time 5}]
 [{:a 3, :s/time 3} {:a 4, :s/time 4} {:a 5, :s/time 5}]]

;; Test for bulk over the limit
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 3
              :entries  [{:a 2 :s/time 2}]})
    (add-bulk  [{:a 3 :s/time 3} {:a 4 :s/time 4} {:a 5 :s/time 5}])
    ((juxt :entries journal-entries)))
=> [() [{:a 3, :s/time 3} {:a 4, :s/time 4} {:a 5, :s/time 5}]]

;; Test for bulk + current under limit
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 3
              :entries  [{:a 2 :s/time 2}]})
    (add-bulk  [{:a 3 :s/time 3} {:a 4 :s/time 4}])
    ((juxt :entries journal-entries)))
=> [[{:a 2, :s/time 2} {:a 3, :s/time 3} {:a 4, :s/time 4}]
    [{:a 2, :s/time 2} {:a 3, :s/time 3} {:a 4, :s/time 4}]]

add-entry ^

[{:keys [meta template], :as journal} entry]
Added 3.0

adds an entry to the journal

v 3.0
(defn add-entry
  ([{:keys [meta template] :as journal} entry]
   (let [{:keys [key unit]} (:time meta)
         {:keys [flatten pre-flattened]} (:entry meta)
         entry (if (and flatten (not pre-flattened))
                 (let [{:keys [flat]} (or @template
                                          (create-template journal entry))]
                   (flat entry))
                 entry)
         entry (add-time entry key unit)]
     (update-journal-single journal entry))))
link
(-> (journal {:meta {:head {:range [0 3]}}})
    (add-entry {:start 1 :output {:value 1}})
    journal-info)
=> (contains-in {:id string?
                 :order :desc, :count 1,
                 :duration "0ms",
                 :template [:output.value :start],
                 :head [{:start 1, :output.value 1, :s/time string?}]})

add-time ^

[entry key unit]
Added 3.0

adds time to entry if if doesn't exist

v 3.0
(defn add-time
  ([entry key unit]
   (if-not (get entry key)
     (let [t (common/from-ns (time/time-ns) unit)]
       (assoc entry key t))
     entry)))
link
(add-time {} :t :s)
=> (contains {:t integer?})

create-template ^

[{:keys [template]} entry]
Added 3.0

creates a template a puts in the cache

v 3.0
(defn create-template
  ([{:keys [template]} entry]
   (let [tmpl (common/create-template entry)]
     (vreset! template tmpl))))
link
(let [j (journal {})]
  (create-template j {:a 1})
  @(:template j))
=> map?

derive ^

[{:keys [meta], :as journal} {:keys [range sample transform], :as params}]
Added 3.0

derives the journal given a select statement

v 3.0
(defn derive
  ([{:keys [meta] :as journal} {:keys [range sample transform]
                                :as params}]
   (let [{:keys [order]} (:time meta)
         subentries (select journal (dissoc params :series))
         subentries (case order
                      :asc (vec subentries)
                      :desc (apply list subentries))]
     (cond-> journal
       :then (assoc :entries subentries)
       transform (assoc-in [:meta :entry :flatten] true)))))
link
(class (derive (journal {:entries [{:a 1 :s/time 1} {:a 2 :s/time 2}]}) {}))
=> std.timeseries.journal.Journal

entries-seq ^

[{:keys [limit entries previous]}]
Added 3.0

gets entries in time order

v 3.0
(defn entries-seq
  ([{:keys [limit entries previous]}]
   (if limit
     (concat entries (take (- limit (count entries)) previous))
     entries)))
link
(-> (journal {:limit 2
              :entries  [{:a 2 :s/time 2}]
              :previous [{:a 1 :s/time 1} {:a 0 :s/time 0}]})
    (entries-seq))
=> [{:a 2, :s/time 2} {:a 1, :s/time 1}]

entries-vec ^

[{:keys [limit entries previous]}]
Added 3.0

gets entries in time order

v 3.0
(defn entries-vec
  ([{:keys [limit entries previous]}]
   (if limit
     (concat (if (seq previous)
               (subvec previous (count entries)))
             entries)
     entries)))
link
(-> (journal {:meta {:time {:order :asc}}
              :limit 2
              :entries  [{:a 2 :s/time 2}]
              :previous [{:a 0 :s/time 0} {:a 1 :s/time 1}]})
    (entries-vec))
=> [{:a 1, :s/time 1} {:a 2, :s/time 2}]

entry-display ^

[entry {:keys [meta]}]
Added 3.0

displays entry, formatting time

v 3.0
(defn entry-display
  ([entry {:keys [meta]}]
   (-> entry
       (update (:key (:time meta)) #(format-time % (:time meta))))))
link
(entry-display {:data {:in  ""
                       :out "ABC"}
                :time 100000}
               {:meta {:time {:key :time :unit :s :format "HH:mm:ss"}}})
=> (contains {:data {:in "", :out "ABC"}, :time #(.endsWith ^String % ":46:40")})

format-time ^

[val {:keys [unit format]}]
Added 3.0

output the time according to format and time unit

v 3.0
(defn format-time
  ([val {:keys [unit format]}]
   (.format (SimpleDateFormat. format)
            (Date. (common/to-ms val unit)))))
link
(format-time 10000 {:unit :s :format "HH:mm:ss"})
=> #(.endsWith ^String % ":46:40")

get-template ^

[{:keys [entries template], :as journal}]
Added 3.0

gets existing or creates a new template

v 3.0
(defn get-template
  ([{:keys [entries template] :as journal}]
   (or @template
       (when (seq entries)
         (create-template journal (first entries))))))
link
(get-template (journal {:entries [{:a {:b 1}
                                   :s/time 0}]}))
=> map?

journal ^

[] [{:keys [id meta entries previous], :as m, :or {id (f/sid)}}]
Added 3.0

creates a new journal

v 3.0
(defn journal
  ([]
   (journal {}))
  ([{:keys [id meta entries previous] :as m
     :or {id (f/sid)}}]
   (let [{:keys [time] :as meta} (collection/merge-nested (:meta +defaults+) meta)
         entries-fn (case (:order time)
                      :desc  #(apply list %)
                      :asc #(apply vector %))]
     (-> m
         (assoc :id id
                :meta meta
                :template (volatile! nil)
                :entries (entries-fn entries)
                :previous (entries-fn previous))
         (types/ :strict)
         (map->Journal)))))
link
(class (journal {:meta {:time {:unit :s :format "HH:mm:ss"}
                        :entry {:flatten false}
                        :head {:range [0 3]}}}))
=> std.timeseries.journal.Journal

journal-entries ^

[journal]
Added 3.0

gets entries from the journal

v 3.0
(defn journal-entries
  ([journal]
   (case (-> journal :meta :time :order)
     :asc (entries-vec journal)
     :desc (entries-seq journal))))
link
(-> (journal {:meta {:time {:order :asc}
                     :entry {:flatten true :pre-flattened true}}
              :limit 2
              :entries  [{:a 2 :s/time 2}]
              :previous [{:a 0 :s/time 0} {:a 1 :s/time 1}]})
    (add-entry  {:a 3 :s/time 3})
    (add-entry  {:a 4 :s/time 4})
    ((juxt :entries journal-entries)))
=> [[{:a 4, :s/time 4}]
    [{:a 3, :s/time 3} {:a 4, :s/time 4}]]

journal-info ^

[{:keys [id meta], :as journal}]
Added 3.0

returns info for the journal

v 3.0
(defn journal-info
  ([{:keys [id meta] :as journal}]
   (let [{:keys [hide time head]} meta
         entries (journal-entries journal)
         head (process/process entries (clojure.core/merge {:time time} head))]
     (cond-> {:id id :meta meta :order (:order time) :count (count entries)}
       (and (not (:duration hide))
            (not (empty? entries))) (assoc :duration (format/t:ms (common/duration entries time)))
       (not (:template hide)) (assoc :template (template-keys (get-template journal)))
       (not (:head hide))     (assoc :head (mapv #(entry-display % journal) head))
       :then   (#(apply dissoc % hide))))))
link
(-> (journal {:meta {:time   {:key :start :order :desc}
                     :head   {:range [0 3]}
                     :hide   #{:meta :template :id}}})
    journal-info)
=> {:order :desc, :count 0, :head []}

journal-invoke ^

[journal] [journal arg] [journal k v & more]
Added 3.0

invoke function for the journal

v 3.0
(defn journal-invoke
  ([journal]
   (journal-info (update-meta journal {:hide #{}})))
  ([journal arg]
   (cond (map? arg)
         (select journal arg)

         (= arg :info)
         (journal-info journal)

         :else
         (select journal {:series arg})))
  ([journal k v & more]
   (let [args (apply list k v more)
         args (if (odd? (count args))
                (cons :series args)
                args)]
     (select journal (apply hash-map args)))))
link
(journal-invoke (journal {}) :info)
=> (contains {:order :desc, :count 0, :head []})

journal-string ^

[journal]

NONE
(defn- journal-string
  ([journal]
   (str "#jnl " (journal-info journal))))
link

map->Journal ^

[m__7997__auto__]
Added 3.0

defines a journal object

v 3.0
(impl/defimpl Journal [id meta template entries limit previous]
  :string  journal-string
  :invoke  journal-invoke
  :final   true

  clojure.lang.IDeref
  (deref [journal] (:entries journal)))
link
(class (map->Journal {}))
=> std.timeseries.journal.Journal

merge ^

[] [j0] [j0 j1]
Added 3.0

merges two journals of the same type together

v 3.0
(defn merge
  ([])
  ([j0] j0)
  ([j0 j1]
   (if-not (and (= (:meta j0)
                   (:meta j1))
                (= (:raw @(:template j0))
                   (:raw @(:template j1))))
     (throw (ex-info "Meta and Template requires to be the same"))
     (let [{:keys [order key]} (-> j0 :meta :time)
           comp-fn (common/order-comp order)
           entries (cond->> (merge-sorted [(:entries j0) (:entries j1)] key comp-fn)
                     (= order :asc) (vec)
                     (= order :desc) (apply list))]
       (assoc j0 :entries entries)))))
link
(let [j1 (journal {:entries [{:a 1 :s/time 1}]})
      j2 (journal {:entries [{:a 2 :s/time 2}]})]
  (class (merge j1 j2)))
=> std.timeseries.journal.Journal

merge-sorted ^

[coll] [coll key-fn] [coll key-fn comp-fn]
Added 3.0

merges a series of arrays together

v 3.0
(defn merge-sorted
  ([coll]
   (merge-sorted coll identity))
  ([coll key-fn]
   (merge-sorted coll identity <))
  ([coll key-fn comp-fn]
   (->> coll
        (filter seq)
        (collection/unfold (fn [s]
                    (if (seq s)
                      (let [[[mf & mn] r]
                            (reduce (fn [[m r] x]
                                      (if (comp-fn (key-fn (first x)) (key-fn (first m)))
                                        [x (cons m r)]
                                        [m (cons x r)]))
                                    [(first s) ()]
                                    (rest s))]
                        (list mf (if mn (cons mn r) r)))))))))
link
(merge-sorted [[1 3 4 6 9 10 15]
               [2 3 6 7 8 9 10]
               [1 3 6 7 8 9 10]
               [1 2 3 4 8 9 10]])
=> [1 1 1 2 2 3 3 3 3 4 4 6 6 6 7 7 8 8 8 9 9 9 9 10 10 10 10 15]

(-> (map reverse [[1 3 4 6 9 10 15]
                  [2 3 6 7 8 9 10]
                  [1 3 6 7 8 9 10]
                  [1 2 3 4 8 9 10]])
    (merge-sorted identity >))
=> [15 10 10 10 10 9 9 9 9 8 8 8 7 7
    6 6 6 4 4 3 3 3 3 2 2 1 1 1]

select ^

[journal] [{:keys [meta], :as journal} {:keys [range sample transform series template compute], :or {range :all}}]
Added 3.0

selects data ferom the journal

v 3.0
(defn select
  ([journal]
   (select journal {}))
  ([{:keys [meta] :as journal} {:keys [range sample transform series template compute]
                                :or {range :all}}]
   (let [entries  (journal-entries journal)
         template (get-template journal)
         transform (if transform
                     (cond-> transform
                       (-> meta :entry :flatten) (assoc :skip true)))
         entries (process/process entries
                                  (cond-> {:time (:time meta)
                                           :range range
                                           :sample sample
                                           :template template
                                           :compute compute}
                                    transform (assoc :transform transform)))]
     (cond-> entries
       series (select-series series)))))
link
(def -jnl- (->> [0 (journal {:meta {:time   {:key :start :order :desc}
                                    :head   {:range [0 3]}
                                    :hide   #{:meta :template :id}}})]
                (iterate (fn [[t journal]]
                           (let [t (+ t 1000000000 (rand-int 100000000))]
                             [t (add-entry journal {:start t
                                                    :output {:value (Math/sin
                                                                     (+ (/ t 100000000000)
                                                                        (rand)))}})])))
                (map second)
                (take 1000)
                (last)))

(select -jnl- {:range [:1m :5m] :sample 10})
=> (fn [entries]
     (= 10 (count entries)))

(first (select -jnl- {:range [:1m :5m] :sample 10}))
=> (contains {:start integer? :output.value number?})

select-series ^

[entries series]
Added 3.0

select data from the series

v 3.0
(defn select-series
  ([entries series]
   (cond (keyword? series)
         (mapv series entries)

         (vector? series)
         (mapv #(select-series entries %) series)

         (list? series)
         (apply mapv vector
                (select-series entries (vec series)))

         (map? series)
         (let [{:s/keys [meta]} series
               series (dissoc series :s/meta)]
           (clojure.core/merge meta (collection/map-vals #(select-series entries %) series))))))
link
(let [j (-> (journal {:meta {:time {:key :time}}})
            (add-bulk [{:a 1 :time 1} {:a 2 :time 2}]))]
  (select-series (journal-entries j) '(:a)))
=> '((2) (1))

template-keys ^

[template]
Added 3.0

get keys for the template

v 3.0
(defn template-keys
  ([template]
   (if-let [flat (:flat (:raw template))]
     (sort (keys flat)))))
link
(template-keys (common/create-template {:data {:in  ""
                                               :out "ABC"}}))
=> [:data.in :data.out]

update-journal-bulk ^

[{:keys [limit entries], :as journal} new-entries]
Added 3.0

adds multiple entries to the journal

v 3.0
(defn update-journal-bulk
  ([{:keys [limit entries] :as journal} new-entries]
   (let [len (count entries)
         f (case (-> journal :meta :time :order)
             :asc vector :desc list)]
     (cond (or (nil? limit)
               (<= (+ len (count new-entries)) limit))
           (update journal :entries #(apply conj % new-entries))

           (<= limit (count new-entries))
           (assoc journal
                  :entries  ()
                  :previous (->> new-entries
                                 (drop (- (count new-entries) limit))
                                 (into (f))))

           :else
           (let [split (- limit len)
                 previous (apply conj entries (take split new-entries))
                 current (apply f (drop split new-entries))]
             (assoc journal :entries current :previous previous))))))
link
(-> (journal {})
    (update-journal-bulk [{:a 1} {:a 2}])
    :entries)
=> coll?

update-journal-single ^

[{:keys [limit entries], :as journal} entry]
Added 3.0

adds a single entry to the journal

v 3.0
(defn update-journal-single
  ([{:keys [limit entries] :as journal} entry]
   (if (or (nil? limit)
           (< (count entries) limit))
     (update journal :entries conj entry)
     (let [f (case (-> journal :meta :time :order)
               :asc vector :desc list)]
       (assoc journal :entries (f entry) :previous entries)))))
link
(-> (journal {})
    (update-journal-single {:a 1})
    :entries)
=> coll?

update-meta ^

[journal meta]
Added 3.0

updates journal meta. used for display

v 3.0
(defn update-meta
  ([journal meta]
   (update journal :meta collection/merge-nested meta)))
link
(-> (journal {})
    (update-meta {:a 1})
    :meta)
=> (contains {:a 1})

parse-range-expr ^

[range time-opts]
Added 3.0

parsing a range expression

v 3.0
(defn parse-range-expr
  ([range time-opts]
   (cond (map? range) range

         (= :all range) (parse-range-expr [0 -1] time-opts)

         :else
         (let [[start type? end?] range
               [start type end] (cond (nil? end?)
                                      [start :to type?]

                                      :else
                                      [start type? end?])
               start (parse-range-unit start time-opts)
               end   (parse-range-unit end time-opts)
               type  (cond (and (= (first end) :absolute)
                                (neg? (second start)))
                           :end

                           (= (first end) :absolute)
                           :to

                           :else type)]
           {:type  type
            :start start
            :end   end}))))
link
(parse-range-expr [:20s 0.8] {:unit :ms})
=> {:type :to, :start [:time 20000], :end [:ratio 0.8]}

(parse-range-expr [0 :for :1m] {:unit :ms})
=> {:type :for, :start [:array 0], :end [:time 60000]}

(parse-range-expr [:-2m (Date. 0)] {:unit :ms})
=> {:type :end, :start [:time -120000], :end [:absolute 0]}

parse-range-unit ^

[x {:keys [unit]}]
Added 3.0

categorising the unit range

v 3.0
(defn parse-range-unit
  ([x {:keys [unit]}]
   (cond (= :start x)
         [:array 0]

         (= :end x)
         [:array -1]

         (= :all x)
         [:array -1]

         (inst? x)
         [:absolute (common/from-ms (inst-ms x) unit)]

         (or (keyword? x)
             (string? x))
         [:time (common/parse-time x unit)]

         (integer? x)
         [:array x]

         (number? x)
         [:ratio x]

         :else
         (throw (ex-info "Invalid range expression" {:input x})))))
link
(parse-range-unit :start {})
=> [:array 0]

(parse-range-unit :end {})
=> [:array -1]

(parse-range-unit :1m {:unit :ms})
=> [:time 60000]

(parse-range-unit :-1m {:unit :ms})
=> [:time -60000]

(parse-range-unit 0.44 {})
=> [:ratio 0.44]

process-filter ^

[arr filter-opts time-opts]
Added 3.0

processes items given a filter

v 3.0
(defn process-filter
  ([arr filter-opts time-opts]
   (cond (nil? filter-opts)
         arr

         (fn? filter-opts)
         (filter filter-opts arr)

         (map? filter-opts)
         (filter (fn [m]
                   (->> filter-opts
                        (map (fn [[k v]]
                               (let [v0 (get m k)]
                                 (if (fn? v)
                                   (v v0)
                                   (= v v0)))))
                        (every? true?)))
                 arr)

         :else (throw (ex-info "Invalid filter" {:input filter-opts})))))
link
(process-filter  [1 2 3 4 5 6 7 8]
                 even?
                 {})
=> [2 4 6 8]

(process-filter  [{:id 1 :val 1}
                  {:id 2 :val 2}
                  {:id 3 :val 3}
                  {:id 4 :val 1}
                  {:id 5 :val 4}
                  {:id 6 :val 1}]
                 {:val 1}
                 {})
=> [{:id 1, :val 1} {:id 4, :val 1} {:id 6, :val 1}]

process-range ^

[arr {:keys [range filter sample], time-opts :time}]
Added 3.0

range stage in the process pipeline

v 3.0
(defn process-range
  ([arr {:keys [range filter sample] time-opts :time}]
   (let [range-opts  (parse-range-expr range time-opts)
         sample-opts (common/parse-sample-expr sample time-opts)]
     (-> (select-range arr range-opts time-opts)
         (process-filter filter time-opts)
         (common/process-sample sample-opts time-opts)))))
link
(process-range (range 10000)
               {:time    {:key identity :unit :s :order :asc}
                :range   [:1m :5m]
                :sample  [10 :linear]})
=> [60 86 112 138 164 190 216 242 268 300]

(process-range (range 10000)
               {:time    {:key identity :unit :ms :order :asc}
                :range   [:-1s (Date. 7000)]
                :sample  [10 :linear]})
=> [6000 6111 6222 6333 6444 6555 6666 6777 6888 7000]

range-end-for ^

[arr {:keys [length]} [tag val] {:keys [order key]}]
Added 3.0

ends the array range given :for option

v 3.0
(defn range-end-for
  ([arr {:keys [length]} [tag val] {:keys [order key]}]
   (case tag
     :array (take val arr)
     :ratio  (cond (= val 1) arr
                   :else (let [num (math/ceil (* val length))]
                           (take num arr)))
     (let [{:keys [comp-eq-fn op-fn]} (common/order-fns order)
           t (case tag
               :absolute val
               :time (-> (key (first arr))
                         (op-fn val)))]
       (take-while #(comp-eq-fn (key %) t) arr)))))
link
(range-end-for [2 3 4] {} [:array 2] {:order :asc :key identity})
=> [2 3]

range-end-to ^

[arr {:keys [length dropped start]} [tag val] {:keys [order key]}]
Added 3.0

ends the array range given :to option

v 3.0
(defn range-end-to
  ([arr {:keys [length dropped start]} [tag val] {:keys [order key]}]
   (let [len (count arr)]
     (case tag
       :array (take (- val dropped) arr)
       :ratio  (cond (= val 1) arr
                     :else (let [num (- (math/ceil (* val length)) dropped)]
                             (take num arr)))
       (let [{:keys [comp-eq-fn op-fn]} (common/order-fns order)
             t (case tag
                 :absolute val
                 :time  (op-fn start val))]
         (take-while #(comp-eq-fn (key %) t) arr))))))
link
(range-end-to [2 3 4] {:dropped 2} [:array 4] {:order :asc :key identity})
=> [2 3]

range-op ^

[type [tag val] arr {:keys [key order]}]
Added 3.0

standardises units for negative inputs

v 3.0
(defn range-op
  ([type [tag val] arr {:keys [key order]}]
   (cond (not (neg? val))
         [tag val]

         (= :for type)
         (throw (ex-info "Value cannot be negative in :for expressions"))

         :else
         (case tag
           :array [tag (let [len (count arr)
                             idx (+ len val)]
                         (if-not (pos? idx) 1 idx))]
           :ratio [tag (+ 1 val)]
           :time  [tag (let [t0 (key (first arr))
                             t1 (key (last arr))
                             op-fn (common/order-op order)
                             t' (op-fn t1 val)]
                         (case order
                           :asc  (- t' t0)
                           :desc (- t0 t')))]
           [tag val]))))
link

range-start ^

[arr {:keys [length]} [tag val] {:keys [order key]}]
Added 3.0

chooses the start of the array

v 3.0
(defn range-start
  ([arr {:keys [length]} [tag val] {:keys [order key]}]
   (case tag
     :array  [val (drop val arr)]
     :ratio  (let [num (long (* val length))]
               [num (drop num arr)])
     (let [{:keys [comp-fn op-fn]} (common/order-fns order)
           t (case tag
               :absolute val
               :time (-> (key (first arr))
                         (op-fn val)))
           counter (f/counter -1)
           out (doall (drop-while (fn [v]
                                    (f/inc! counter)
                                    (comp-fn (key v) t)) arr))]
       [@counter out]))))
link
(range-start [1 2 3 4] {} [:array 2] {:order :asc :key identity})
=> [2 [3 4]]

(range-start [-2 -1 0 1 2] {} [:time 2] {:order :asc :key identity})
=> [2 [0 1 2]]

(range-start [-2 -1 0 1 2] {} [:absolute 0] {:order :asc :key identity})
=> [2 [0 1 2]]

range-wrap ^

[f start]
Added 3.0

function wrapper for range-start and range-end functions

v 3.0
(defn range-wrap
  ([f start]
   (fn [arr params [tag val] time-opts]
     (cond (or (empty? arr)
               (nil? tag))
           (cond->> arr
             start (vector 0))

           :else
           (f arr params [tag val] time-opts)))))
link
((range-wrap range-start true) [1 2 3] {} [:array 1] {:order :asc :key identity})
=> [1 [2 3]]

select-range ^

[arr {:keys [type], :as range-opts} time-opts]
Added 3.0

selects the range

v 3.0
(defn select-range
  ([arr {:keys [type] :as range-opts} time-opts]
   (let [select-fn (if (= type :end)
                     select-range-end
                     select-range-standard)]
     (select-fn arr range-opts time-opts))))
link
(select-range [1 2 3 4 5]
              {:type :for :start [:array 1] :end [:array 3]}
              {:order :asc :key identity})
=> [2 3 4]

(select-range [1 2 3 4 5]
              {:type :for :start [:time 1] :end [:time 3]}
              {:order :asc :key identity})
=> [2 3 4 5]

(select-range [1 2 3 4 5 6 7 8]
              {:type :end :start [:array -3] :end [:absolute 7]}
              {:order :asc :key identity})
=> [5 6 7]

(select-range [1 2 3 4 5 6 7 8]
              {:type :end :start [:time -3] :end [:absolute 7]}
              {:order :asc :key identity})
=> [4 5 6 7]

select-range-end ^

[arr {:keys [start end], :as m} {:keys [key], :as time-opts}]
Added 3.0

helper for range select when a date value is at the end

v 3.0
(defn select-range-end
  ([arr {:keys [start end] :as m} {:keys [key] :as time-opts}]
   (let [start-fn (range-wrap range-start true)
         end-fn (range-wrap range-end-to false)
         arr    (end-fn arr {} end time-opts)
         length (count arr)
         start  (range-op :to start arr time-opts)
         params {:length length :start (if-not (empty? arr)
                                         (key (first arr)))}
         [dropped arr] (start-fn arr params start time-opts)]
     (vec arr))))
link

select-range-standard ^

[arr {:keys [type start end], :as m} {:keys [key], :as time-opts}]
Added 3.0

helper for standard range select

v 3.0
(defn select-range-standard
  ([arr {:keys [type start end] :as m} {:keys [key] :as time-opts}]
   (let [length (count arr)
         start  (range-op :to start arr time-opts)
         end    (range-op type end arr time-opts)
         start-fn   (range-wrap range-start true)
         end-fn     (range-wrap (case type
                                  :to  range-end-to
                                  :for range-end-for)
                                false)
         params {:length length :start (if-not (empty? arr)
                                         (key (first arr)))}]
     (let [[dropped arr] (start-fn arr params start time-opts)
           arr (end-fn arr (assoc params :dropped dropped) end time-opts)]
       (vec arr)))))
link

5    std.timeseries Guide

std.timeseries is a library for handling time-series data using a "Journal" abstraction. It supports ingestion, storage, retrieval, downsampling, and aggregation of time-ordered records.

5.1    Core Concepts

  • Journal: The primary data structure. It manages a sorted list of entries and associated metadata.n- Entry: A map representing a data point. Must have a time key (default :s/time).n- Processing: The engine for selecting, aggregating, and transforming data ranges.

5.2    Usage

5.2.1    Scenarios

5.2.1.1    1. Real-time Metric Collection

Scenario: Ingesting sensor data.

Create a journal configured for high-frequency data (e.g., milliseconds).

(require '[std.timeseries :as ts])\n\n(def sensor-journal\n  (ts/journal {:meta {:time {:unit :ms :key :timestamp}\n                      :entry {:flatten true} ;; Save space by flattening nested maps\n                      }}))\n\n(defn on-sensor-read [data]\n  ;; data: {:temp 20.5 :humidity 50 :timestamp 1600000000000}\n  (alter-var-root #'sensor-journal ts/add-entry data))

5.2.1.2    2. Downsampling for Visualization

Scenario: Fetching a 24-hour chart with 100 data points.

You have raw data every second, but you only want 100 points for a graph.

(ts/select sensor-journal\n           {:range [:24h :now] ;; Or specific timestamps\n            :sample 100        ;; Target count\n            :transform {:default {:aggregate :mean} ;; Average values in each bucket\n                        :temp {:aggregate :max}}    ;; But show max temp\n            })

5.2.1.3    3. Merging Disparate Sources

Scenario: Combining logs from two servers.

You have two journals with potentially overlapping or interleaved time periods.

(def combined-journal (ts/merge journal-a journal-b))\n\n;; The merge respects time ordering.\n;; Pre-requisite: Journals must share the same metadata structure.

5.2.1.4    4. Handling Irregular Intervals

Scenario: Data arrives sporadically, but you need a regular 1-second interval output.

Use derive to create a normalized view.

(ts/derive raw-journal\n           {:range :all\n            :transform {:interval :1s  ;; Force 1s buckets\n                        :default {:aggregate :last ;; Use last known value\n                                  :fill :previous} ;; Fill gaps with previous value\n                        }})

5.2.1.5    5. Complex Window Analysis

Scenario: Moving average.

While std.timeseries focuses on storage and retrieval, you can compute derived series during selection.

(ts/select my-journal\n           {:range :1h\n            :compute {:moving-avg (fn [entries] ...)} ;; Custom computation\n            })

5.2.1.6    6. Efficient Storage with Templates

Scenario: Storing repetitive map structures.

If every entry looks like {:a 1 :b {:c 2}}, the journal can learn a "template" to store them as flat vectors internally, saving memory.

(def j (ts/journal {:meta {:entry {:flatten true}}}))\n;; The first entry added determines the template structure.\n(ts/add-entry j {:a 1 :b 2})\n;; Internally stored as something like [1 2] + template reference.

6    std.timeseries: A Comprehensive Summary

The std.timeseries module in foundation-base provides a robust framework for managing, processing, and analyzing time-series data. It offers functionalities for data aggregation, sampling, transformation, and journaling, making it suitable for applications requiring historical data tracking, performance monitoring, or event logging. The module is designed to be flexible, allowing for custom processing pipelines and integration with various data representations.

The module is organized into several sub-namespaces:

6.1    std.timeseries.common

This namespace provides fundamental utilities and helper functions for time-series operations, including data manipulation, template creation, and order handling.

  • linspace [arr n]: Generates n linearly spaced samples from an array.n cluster [arr n f]: Clusters an array into n groups and applies an aggregate function f to each cluster.n make-empty [entry]: Creates an empty entry (map) with default zero/empty values based on the structure of a sample entry.n raw-template [entry]: Creates a template for nested-to-flat and flat-to-nested map transformations, including type information and empty values.n flat-fn [template]: Creates a function to flatten a nested map into a dot-separated key map based on a template.n nest-fn [template]: Creates a function to nest a flat map into a hierarchical map based on a template.n create-template [entry]: Creates a comprehensive template for an entry, including raw template, flat, and nest transformation functions.n order-flip [order]: Flips the order (:asc to :desc, or vice-versa).n order-fn [](#asc desc): Creates a function that returns an order-specific value or flips it.n order-comp, order-comp-eq, order-op: Pre-defined order functions for comparison and operations.n order-fns [order & [flip]]: Returns a map of commonly used order-specific functions (:comp-fn, :comp-eq-fn, :op-fn).n +default+: A map of default options for time-series processing, including time key, unit, order, interval, and sample.n from-ms [ms to]: Converts milliseconds to a specified time unit (ns, us, ms, s, m, h, d).n to-ms [ms from]: Converts a time value from a specified unit to milliseconds.n from-ns [ns to]: Converts nanoseconds to a specified time unit.n duration [arr opts]: Calculates the duration between the first and last elements in an array based on time options.n parse-time [s & [unit]]: Parses a string or keyword time representation (e.g., ":0.1m") into milliseconds or a specified unit.n parse-time-expr [m]: Parses a time expression map, converting interval strings/keywords to numeric values.n sampling-fn: A multimethod for extensible sampling functions.n sampling-parser: A multimethod for extensible sampling parser functions.n parse-sample-expr [sample time-opts]: Parses a sample expression, converting various input formats into a standardized sample options map.n* process-sample [arr m time-opts]: Processes an array by applying a sampling function based on the provided sample options.

6.2    std.timeseries.journal

This namespace defines the Journal record and associated functions for managing a time-series journal, which acts as a chronological log of entries. It provides functionalities for adding, selecting, deriving, and merging journal entries.

  • format-time [val opts]: Formats a time value according to a specified unit and format string.n +defaults+: Default options for journal metadata, including time key, unit, order, entry flattening, head range, and select options.n template-keys [template]: Extracts sorted keys from a template's flat representation.n entry-display [entry meta]: Formats a journal entry for display, including time formatting.n create-template [journal entry]: Creates and caches a template for journal entries.n get-template [journal]: Retrieves an existing template or creates a new one if the journal has entries.n entries-seq [journal]: Returns journal entries as a sequence, respecting the journal's limit and order.n entries-vec [journal]: Returns journal entries as a vector, respecting the journal's limit and order.n journal-entries [journal]: Returns journal entries in the specified time order.n journal-info [journal]: Returns information about the journal, including count, order, duration, template keys, and head entries.n journal-invoke [journal & args]: The invoke function for the Journal record, allowing it to be called directly for info or selection.n Journal Deftype: The core record for a time-series journal, holding id, meta, template, entries, limit, and previous entries. It implements clojure.lang.IDeref and has a custom toString method.n journal [& [m]]: Creates a new Journal instance, merging default metadata and ensuring proper initialization.n add-time [entry key unit]: Adds a timestamp to an entry if it doesn't already exist.n update-journal-single [journal entry]: Adds a single entry to the journal, managing the limit and previous entries.n add-entry [journal entry]: Adds a single entry to the journal, handling flattening and timestamping.n update-journal-bulk [journal new-entries]: Adds multiple entries to the journal, managing the limit and previous entries.n add-bulk [journal entries]: Adds multiple entries to the journal, handling flattening and timestamping.n update-meta [journal meta]: Updates the journal's metadata, typically for display purposes.n select-series [entries series]: Selects data from a series of entries based on a keyword, vector, list, or map specification.n select [journal & [params]]: Selects and processes data from the journal based on range, sample, transform, series, and compute parameters.n derive [journal params]: Derives a new journal (or a modified version of the current one) by applying selection and transformation parameters.n merge-sorted [coll & [key-fn comp-fn]]: Merges multiple sorted collections into a single sorted collection.n* merge [& journals]: Merges two or more journals of the same type, combining their entries while maintaining order.

6.3    std.timeseries.process

This namespace provides the core processing pipeline for time-series data, including range selection, sampling, transformation (aggregation), and computation of indicators.

  • prep-merge [m time-opts]: Prepares merge functions and options for aggregation, including resolving aggregate functions and parsing sample expressions.n create-merge-fn [m time-opts]: Creates a merge function for aggregating data based on sample and aggregate options.n create-custom-fns [custom time-opts]: Creates a map of custom merge functions for specific keys, as defined in the custom transform options.n map-merge-fn [transform-opts time-opts]: Creates a merge function for map-based time-series data, handling default, time, and custom aggregations.n time-merge-fn [transform-opts time-opts]: Creates a merge function specifically for time-based aggregations.n parse-transform-expr [m type time-opts]: Parses a transform expression, converting interval strings/keywords to numeric values and creating the appropriate merge function.n transform-interval [arr transform type time-opts]: Transforms an array based on a specified interval, grouping and merging data points.n process-transform [arr opts]: Processes the transform stage of the time-series pipeline, applying interval-based transformations and flattening entries if configured.n common/sampling-parser extension: Extends :interval strategy for sampling.n common/sampling-fn extension: Extends :interval strategy for sampling.n process-compute [arr opts]: Processes the compute stage, applying indicators and computations defined in the compute options.n* process [arr opts]: The main function for processing time series. It orchestrates the entire pipeline: parsing time options, processing ranges, applying transformations, and computing indicators.

6.4    std.timeseries (Facade Namespace)

This namespace acts as a facade, re-exporting key functions from its sub-namespaces for convenience.

  • create-template (from common)n journal, add-bulk, add-entry, update-meta, derive, merge, select (from journal)n process (from process)

Overall Importance:

The std.timeseries module is a powerful and flexible tool for handling time-series data within the foundation-base project. Its key contributions include:

  • Unified Data Model: Provides a consistent way to represent and manipulate time-series data, regardless of the underlying Java time types.n Flexible Processing Pipeline: Supports a multi-stage processing pipeline (range, sample, transform, compute) that can be customized for various analytical needs.n Data Aggregation and Sampling: Offers robust mechanisms for aggregating data over intervals and sampling data points, crucial for reducing noise and extracting insights.n Journaling and Historical Tracking: The Journal component provides a structured way to log and retrieve time-series entries, enabling historical analysis and event tracking.n Extensibility: The extensive use of protocols and multimethods allows for easy extension with new time representations, sampling strategies, aggregation functions, and computation indicators.n* Interoperability: The module's ability to coerce between different time representations facilitates integration with various Java and Clojure libraries.

By providing a comprehensive and extensible framework for time-series data, std.timeseries significantly contributes to the foundation-base project's capabilities in areas such as performance monitoring, event analysis, and data visualization.

Related

jvm.artifactdocs

artifact coordinates and paths

jvm.classloaderdocs

classpath and classloaders

jvm.depsdocs

JVM dependency resolution

jvm.monitordocs

JVM metrics

jvm.namespacedocs

namespace management

jvm.protocoldocs

JVM extension protocols

jvm.reflectdocs

Java reflection

jvm.requiredocs

dependency-aware requiring