std.concurrent
concurrency primitives, queues, executors, requests, relays, and threads
Start with the facade for common operations, then use the focused pages for lifecycle guidance, walkthroughs, and complete API references.
1 Choose a primitive
Concurrency libraries
Each focused namespace has its own walkthrough and API page.
std.concurrent.atom
Atom-backed batch queues and completion-aware hubs.
std.concurrent.bus
Addressed request and response loops between registered threads.
std.concurrent.executor
Thread pools, submission, scheduling, metrics, and shutdown.
std.concurrent.pool
Reusable resources with acquisition, release, cleanup, and health.
std.concurrent.print
Serialized asynchronous printing from concurrent workers.
std.concurrent.queue
Blocking queues, deques, timeouts, draining, and bulk processing.
std.concurrent.relay
Interactive process and socket streams over a shared bus.
std.concurrent.request
Single, asynchronous, bulk, and transactional request orchestration.
std.concurrent.request-apply
Invokable request definitions with transforms and execution modes.
std.concurrent.request-command
Reusable command templates for request clients.
std.concurrent.thread
Thread construction, inspection, joining, and coordination.
2 Walkthrough
The layers are designed to compose. A blocking queue provides back-pressure, an executor runs anonymous tasks, a bus owns addressed handler loops, and a pool manages reusable resources. Request and relay abstractions build higher-level workflows on those primitives.
submit work to a pooled executor
(let [work (concurrent/queue:fixed 32)
executor (concurrent/executor:pool 2 4 1000 work)]
(try
@(concurrent/submit executor
(fn [] {:status :complete}))
=> {:status :complete}
(finally
(concurrent/exec:shutdown executor))))
use a blocking queue
(let [q (concurrent/queue:fixed 4)]
(concurrent/put q 1)
(concurrent/put q 2)
[(concurrent/take q)
(concurrent/take q)])
=> [1 2]
3 Facade API
The facade re-exports the most commonly used thread, queue, executor, pool, bus, request, and relay operations. The focused pages document the full source namespaces.
- *shared*
- +units+
- ->timeunit
- all-stacktraces
- aq:executor
- aq:new
- aq:process
- aq:submit
- bulk
- bulk:inputs
- bulk:map
- bulk:transact
- bus
- bus:all-ids
- bus:all-threads
- bus:close
- bus:close-all
- bus:create
- bus:deregister
- bus:get-count
- bus:get-id
- bus:get-queue
- bus:get-thread
- bus:has-id?
- bus:kill
- bus:kill-all
- bus:open
- bus:register
- bus:reset-counters
- bus:send
- bus:send-all
- bus:wait
- bus:with-temp
- bus?
- deque
- deque?
- drain
- exec:at-capacity?
- exec:await-termination
- exec:current-active
- exec:current-completed
- exec:current-size
- exec:current-submitted
- exec:get-queue
- exec:increase-capacity
- exec:keep-alive
- exec:pool-max
- exec:pool-size
- exec:queue
- exec:rejected-handler
- exec:shutdown
- exec:shutdown-now
- exec:shutdown?
- exec:terminated?
- exec:terminating?
- executor
- executor-string
- executor:cached
- executor:health
- executor:info
- executor:kill
- executor:pool
- executor:props
- executor:scheduled
- executor:share
- executor:shared
- executor:single
- executor:start
- executor:started?
- executor:stop
- executor:stopped?
- executor:submit
- executor:type
- executor:unshare
- hub-state
- hub:add-entries
- hub:executor
- hub:new
- hub:process
- hub:submit
- hub:wait
- peek
- peek-first
- peek-last
- pool
- pool:acquire
- pool:cleanup
- pool:create
- pool:dispose
- pool:dispose-over
- pool:dispose:mark
- pool:dispose:unmark
- pool:release
- pool:resources:busy
- pool:resources:idle
- pool:resources:thread
- pool:with-resource
- pool?
- pop
- process-bulk
- push
- put
- put-first
- put-last
- queue
- queue:fixed
- queue:limited
- queue?
- relay
- relay:bus
- relay:create
- remaining-capacity
- remove
- req
- req-fn
- req:applicative
- req:bulk
- req:call-bulk
- req:call-single
- req:command
- req:in
- req:opts
- req:opts-clean
- req:process-bulk
- req:process-single
- req:run
- req:transact
- schedule
- schedule:fixed-delay
- schedule:fixed-rate
- send
- stacktrace
- submit
- submit-notify
- take
- take-first
- take-last
- thread
- thread:active-count
- thread:alive?
- thread:all
- thread:all-ids
- thread:classloader
- thread:current
- thread:daemon?
- thread:dump
- thread:global-uncaught
- thread:has-access?
- thread:has-lock?
- thread:id
- thread:interrupt
- thread:interrupted?
- thread:join
- thread:notify
- thread:notify-all
- thread:run
- thread:sleep
- thread:spin
- thread:start
- thread:uncaught
- thread:wait-on
- thread:yield
- transact
- wrap-min-time
v 3.0
(defn ->timeunit
(^TimeUnit [obj]
(cond (keyword? obj)
(get +units+ obj)
:else
obj)))
link
(q/->timeunit :ms) => java.util.concurrent.TimeUnit/MILLISECONDS
v 3.0
(defn all-stacktraces
([]
(Thread/getAllStackTraces)))
link
(all-stacktraces) => #(instance? java.util.Map %)
aq:executor ^
[{:keys [target], :as opts}]
creates a executor that takes in an atom queue
v 3.0
(defn aq:executor
([{:keys [target] :as opts}]
(let [queue (aq:new)
executor (exe/executor:single 1)
submit (aq:submit executor queue opts)]
{:queue queue
:executor executor
:submit submit
:options opts})))
link
(let [-res- (atom []) -aq- (aq:executor {:handler (fn [_ items] (swap! -res- concat items)) :interval 10 :max-batch 10}) -exe- (:executor -aq-)] (try ((:submit -aq-) 1 2 3) (Thread/sleep 100) @-res- (finally (executor/exec:shutdown -exe-)))) => [1 2 3]
v 3.0
(defn aq:process
([f aq max-batch]
(loop []
(let [queue (at/swap-return! aq
(fn [queue] [queue []]))]
(->> (partition-all max-batch queue)
(mapv f))))))
link
(let [+state+ (atom [])] (aq:process (fn [elems] (swap! +state+ conj elems)) (atom [1 2 3 4 5]) 3) @+state+) => [[1 2 3] [4 5]]
aq:submit ^
[executor queue {:keys [target handler max-batch interval], :as opts}]
submission function for one or multiple entries to aq
v 3.0
(defn aq:submit
([executor queue {:keys [target handler max-batch interval] :as opts}]
(let [bulk-fn (fn []
(aq:process (fn [items]
(-> (handler target items)
(env/explode)))
queue
max-batch))]
(fn [& entries]
(do (apply swap! queue conj entries)
(exe/submit-notify executor
bulk-fn
interval)
nil)))))
link
(let [-out- (atom []) -exe- (executor/executor:single) -q- (atom []) -sub- (aq:submit -exe- -q- {:handler (fn [_ items] (swap! -out- concat items)) :interval 10 :max-batch 10})] (try (-sub- 1 2 3) (Thread/sleep 100) @-out- (finally (executor/exec:shutdown -exe-)))) => [1 2 3]
v 3.0
(defn bulk
([client thunk]
(bulk client thunk {}))
([client thunk opts]
(let [{:keys [async catch chain post] :as opts} (req:opts-init opts)
[bulk current] (bulk-collect client thunk chain)
received (f/on:all current
(fn [& arr]
(reduce h/call (apply vector arr) post)))
received (cond-> received
catch (f/on:exception catch))
final (f/future:chain received chain)
_ (cond bulk
(bulk-process client bulk opts)
:else
(swap! (get *current* client) conj final))]
(req:return final (boolean (or (not bulk)
(get *transact* client)
async))))))
link
(bulk |client| (fn [] (req-fn |client| {:type :eval, :form '(+ 1 2 3)}) (req-fn |client| {:type :eval, :form '(+ 4 5 6)}))) => [6 15] (bulk |client| (fn [] (req-fn |client| {:type :eval, :form '(+ 1 2 3)}) (req-fn |client| {:type :eval, :form '(+ 4 5 6)})) {:chain [#(apply + %)]}) => 21 (bulk |client| (fn [] (req-fn |client| {:type :eval, :form 1} {:chain [inc]}) (req-fn |client| {:type :eval, :form 2} {:chain [inc]}))) => [2 3] (bulk |client| (fn [] (bulk |client| (fn [] (req-fn |client| {:type :eval, :form 1} {:chain [inc]}) (req-fn |client| {:type :eval, :form 2} {:chain [inc]}))) (req-fn |client| {:type :eval, :form 3} {:chain [inc]}) (req-fn |client| {:type :eval, :form 4} {:chain [inc]}))) => [[2 3] 4 5] (bulk |client| (fn [] (bulk |client| (fn [] (req-fn |client| {:type :eval, :form 1} {:chain [inc]}) (req-fn |client| {:type :eval, :form 2} {:chain [inc]})) {:chain [#(apply + %)]}) (req-fn |client| {:type :eval, :form 3} {:chain [inc]}) (req-fn |client| {:type :eval, :form 4} {:chain [inc]})) {:chain [#(apply + %)]}) => 14
v 3.0
(defn bulk:inputs
([client thunk]
(let [context (bulk-context)]
(binding [*bulk* (assoc *bulk* client context)
*current* (assoc *current* client (atom []))]
(thunk)
(:inputs @(:cache context))))))
link
(bulk:inputs |client| (fn [] (req:unit |client| {:type :eval :form '(+ 1 2 3)}) (req:unit |client| {:type :eval :form '(+ 4 5 6)}))) => [{:type :eval, :form '(+ 1 2 3)} {:type :eval, :form '(+ 4 5 6)}]
v 3.0
(defn bulk:map
([client f inputs]
(bulk:map client f inputs {}))
([client f inputs opts]
(bulk client
(fn []
(mapv (partial f client) inputs))
opts)))
link
(bulk:map |client| req-fn [{:type :eval :form 1} {:type :eval :form 2}]) => [1 2]
bulk:transact ^
[client thunk] [client thunk opts]
creates a transaction within a bulk context
v 3.0
(defn bulk:transact
([client thunk]
(bulk:transact client thunk {}))
([client thunk opts]
(let [{:keys [async chain pre post] :as opts} (req:opts-init opts)
state (volatile! nil)
_ (reduce h/call nil pre)]
(bulk client
(fn []
(let [res (transact client thunk)]
(vreset! state res)))
(req:opts-clean opts))
(-> @state
(f/on:all (fn [& arr]
(reduce h/call (apply vector arr) post)))
(f/future:chain chain)
(req:return (boolean (or (get *bulk* client)
(get *transact* client)
async)))))))
link
(bulk:transact |client| (fn [] (req |client| {:type :eval :form 1} {:chain [inc]}) (req |client| {:type :eval :form 1} {:chain [#(* 2 %)]}))) => [2 2]
v 3.0
(defn bus:all-ids
([{:keys [state] :as bus}]
(let [{:keys [threads]} @state]
(set (keys threads)))))
link
(bus:with-temp bus (bus:all-ids bus)) => (contains #{string?})
v 3.0
(defn bus:all-threads
([{:keys [state] :as bus}]
(let [{:keys [threads]} @state]
(collection/map-vals :thread threads))))
link
(bus:with-temp bus (= (first (vals (bus:all-threads bus))) (Thread/currentThread))) => true
v 3.0
(defn bus:close
([bus id]
(bus:send bus id {:op :exit})))
link
(bus:with-temp bus (let [{:keys [stopped id]} @(bus:open bus (fn [m] (update m :value inc)))] (Thread/sleep 10) (bus:close bus id) [@stopped (bus:get-count bus)])) => (contains-in [{:exit :normal, :id string? :unprocessed empty? :start number? :end number?} 1])
v 3.0
(defn bus:close-all
([bus]
(collection/map-entries (fn [id]
[id (bus:close bus id)])
(bus:all-ids bus))))
link
(bus:with-temp bus (let [_ @(bus:open bus (fn [m] (update m :value inc))) _ @(bus:open bus (fn [m] (update m :value inc))) _ @(bus:open bus (fn [m] (update m :value inc))) _ @(bus:open bus (fn [m] (update m :value inc)))] (Thread/sleep 10) (bus:close-all bus) (Thread/sleep 100) (bus:get-count bus))) => 1
v 3.0
(defn bus:create
([] (bus:create nil))
([m]
(map->Bus (merge {:state (atom {})
:results (atom {})
:output (queue/queue)
:counters {:sent (atom 0)
:received (atom 0)}}
m))))
link
(bus:create) => bus?
v 3.0
(defn bus:deregister
([bus]
(bus:deregister bus (bus:get-id bus)))
([{:keys [state] :as bus} id]
(swap! state (fn [m]
(let [{:keys [thread]} (get-in m [:threads id])]
(-> m
(update :threads dissoc id)
(update :lookup dissoc thread)))))))
link
(bus:with-temp bus (bus:register bus "foo" (Thread.)) (bus:deregister bus "foo") (bus:has-id? bus "foo")) => false
v 3.0
(defn bus:get-count
([{:keys [state] :as bus}]
(-> state deref :threads count)))
link
(bus:with-temp bus (bus:get-count bus)) => 1
v 3.0
(defn bus:get-id
([bus]
(bus:get-id bus (thread/thread:current)))
([{:keys [state]} thread]
(let [{:keys [lookup]} @state]
(get lookup thread))))
link
(bus:with-temp bus (bus:get-id bus)) => string?
bus:get-queue ^
[bus] [{:keys [state], :as bus} obj]
gets the message queue associated with the thread
v 3.0
(defn bus:get-queue
([bus]
(bus:get-queue bus (thread/thread:current)))
([{:keys [state] :as bus} obj]
(let [{:keys [threads lookup]} @state]
(:queue (if (instance? Thread obj)
(get threads (get lookup obj))
(get threads obj))))))
link
(bus:with-temp bus (vec (bus:get-queue bus))) => []
v 3.0
(defn bus:get-thread
([{:keys [state]} id]
(let [{:keys [threads]} @state]
(:thread (get threads id)))))
link
(bus:with-temp bus (->> (bus:get-id bus) (bus:get-thread bus))) => (cc/thread:current)
v 3.0
(defn bus:has-id?
([{:keys [state]} id]
(let [{:keys [threads]} @state]
(boolean (get threads id)))))
link
(bus:with-temp bus (bus:has-id? bus (bus:get-id bus))) => true
v 3.0
(defn bus:kill
([bus id]
(let [thread (bus:get-thread bus id)]
(if (and thread (not= thread (thread/thread:current)))
(doto thread
(thread/thread:interrupt))))))
link
(bus:with-temp bus (let [{:keys [id stopped]} @(bus:open bus (fn [m] (update m :value inc)))] (Thread/sleep 10) (bus:kill bus id) [(deref stopped 1000 :timeout) (bus:get-count bus)])) => (contains-in [{:exit :interrupted :id string? :unprocessed empty? :start number? :end number?} 1])
v 3.0
(defn bus:kill-all
([bus]
(collection/map-entries (fn [id]
[id (bus:kill bus id)])
(bus:all-ids bus))))
link
(bus:with-temp bus (let [_ @(bus:open bus (fn [m] (update m :value inc))) _ @(bus:open bus (fn [m] (update m :value inc))) _ @(bus:open bus (fn [m] (update m :value inc))) _ @(bus:open bus (fn [m] (update m :value inc)))] (Thread/sleep 10) (bus:kill-all bus) (Thread/sleep 10) (bus:get-count bus))) => 1
bus:open ^
[bus handler] [bus handler {:keys [id], :as opts, :or {id (std.lib.foundation/sid)}}]
bus:opens a new handler loop given function
v 3.0
(defn bus:open
([bus handler]
(bus:open bus handler {}))
([bus handler {:keys [id] :as opts
:or {id (std.lib.foundation/sid)}}]
(if-not (component/started? bus) (std.lib.foundation/error "Bus not started" {:bus bus}))
(let [started (f/incomplete)
opts (assoc opts :id id :started started :stopped (f/incomplete))
_ (f/future
(run-handler bus handler opts))]
started)))
link
(bus:with-temp bus (let [{:keys [id]} (deref (bus:open bus (fn [m] (update m :value inc))) 1000 :timeout)] (Thread/sleep 100) (deref (bus:send bus id {:value 1}) 1000 :timeout))) => (contains {:value 2, :id string?}) (bus:with-temp bus (let [{:keys [id stopped]} (deref (bus:open bus (fn [m] (update m :value inc)) {:timeout 400}) 1000 :timeout)] [(deref stopped 1000 :timeout) (bus:get-count bus)])) => (contains-in [{:exit :timeout :id string? :start number? :end number?} 1])
bus:register ^
[bus] [bus id] [{:keys [state], :as bus} id thread]
registers a thread to the bus
v 3.0
(defn bus:register
([bus]
(bus:register bus (std.lib.foundation/sid)))
([bus id]
(bus:register bus id (thread/thread:current)))
([{:keys [state] :as bus} id thread]
(let [queue (queue/queue)]
(swap! state (fn [m]
(-> m
(assoc-in [:threads id]
{:id id :thread thread :queue queue})
(assoc-in [:lookup thread] id))))
queue)))
link
(bus:with-temp bus (bus:register bus "foo" (Thread.)) (bus:has-id? bus "foo")) => true
v 4.0
(defn bus:reset-counters
[bus]
(collection/map-vals #(atom/swap-return! % (fn [v] [v 0]))
(:counters bus)))
link
(bus:with-temp bus (bus:send bus (bus:get-id bus) {:op :test}) (bus:reset-counters bus) @(:received (:counters bus))) => 0
bus:send ^
[{:keys [results counters], :as bus} id msg]
sends a message to the given thread
v 3.0
(defn bus:send
([{:keys [results counters] :as bus} id msg]
(if-let [queue (bus:get-queue bus id)]
(let [msg-id (or (:id msg) (std.lib.foundation/sid))
return (f/incomplete)]
(swap! results assoc msg-id return)
(queue/put queue (assoc msg :id msg-id))
(swap! (:sent counters) inc)
return))))
link
(bus:with-temp bus (bus:send bus (bus:get-id bus) {:op :hello :message "world"}) (cc/take (bus:get-queue bus) 1000 :ms)) => (contains {:op :hello, :message "world", :id string?})
v 3.0
(defn bus:send-all
([bus msg]
(collection/map-entries (fn [id] [id (bus:send bus id msg)]) (bus:all-ids bus))))
link
(bus:with-temp bus (bus:send-all bus {:op :all}) (bus:wait bus {:timeout 1000})) => (contains {:op :all})
bus:wait ^
[bus] [bus {:keys [timeout timeunit], :or {timeunit :ms}}]
bus:waits on the message queue for message
v 3.0
(defn bus:wait
([bus]
(bus:wait bus {}))
([bus {:keys [timeout timeunit]
:or {timeunit :ms}}]
(queue/take (bus:get-queue bus) timeout timeunit)))
link
(bus:with-temp bus (bus:send bus (bus:get-id bus) {:op :hello :message "world"}) (bus:wait bus {:timeout 1000})) => (contains {:op :hello, :message "world", :id string?}) (bus:with-temp bus (bus:wait bus {:timeout 100}))
v 3.0
(defmacro bus:with-temp
([var & body]
`(let [~var (bus)
~'_ (bus:register ~var)]
(try ~@body
(finally (bus:deregister ~var)
(stop-bus ~var))))))
link
(bus:with-temp bus (bus? bus)) => true
v 3.0
(defn ^LinkedBlockingDeque deque
([]
(LinkedBlockingDeque.))
([& elements]
(LinkedBlockingDeque. ^java.util.Collection elements)))
link
(q/deque 1 2 3) => java.util.concurrent.LinkedBlockingDeque
v 3.0
(defn deque?
([obj]
(instance? BlockingDeque obj)))
link
(q/deque? (q/deque)) => true (q/deque? (q/queue)) => false
v 3.0
(defn drain
([^BlockingQueue queue]
(let [target (java.util.ArrayList. (count queue))]
(.drainTo queue target)
target))
([^BlockingQueue queue ^long max]
(let [target (java.util.ArrayList. max)]
(drain queue max target)
target))
([^BlockingQueue queue ^long max ^java.util.Collection target]
(.drainTo queue target max)
target))
link
((juxt #(q/drain % 2) #(into [] %)) (q/queue 1 2 3 4)) => [[1 2] [3 4]]
v 3.0
(defn exec:at-capacity?
([executor]
(= (exec:current-active executor)
(exec:pool-max executor))))
link
(let [exe (executor:single 1)] (try (try (submit exe (fn [] (Thread/sleep 1000))) (catch Throwable t)) (exec:at-capacity? exe) (finally (exec:shutdown-now exe)))) => boolean?
exec:await-termination ^
[service] [service ms]
await termination for executor service
v 3.0
(defn exec:await-termination
([^ExecutorService service]
(exec:await-termination service Long/MAX_VALUE))
([^ExecutorService service ^long ms]
(.awaitTermination service ms TimeUnit/MILLISECONDS)))
link
;; Shutdown (def -counter- (atom 0)) (-> (executor:single) (doto (submit (fn [] (Thread/sleep 100) (swap! -counter- inc)))) (doto (submit (fn [] (Thread/sleep 100) (swap! -counter- inc)))) (exec:shutdown) (doto (exec:await-termination))) @-counter- => 2 ;; Shutdown now (def -counter- (atom 0)) (-> (executor:single) (doto (submit (fn [] (Thread/sleep 100) (swap! -counter- inc)))) (doto (submit (fn [] (Thread/sleep 100) (swap! -counter- inc)))) (exec:shutdown-now) (doto (exec:await-termination))) @-counter- => 0
v 3.0
(defn exec:current-active
([^ThreadPoolExecutor service]
(.getActiveCount service)))
link
(let [exe (executor:single)] (try (exec:current-active exe) (finally (exec:shutdown exe)))) => number?
v 3.0
(defn exec:current-completed
([^ThreadPoolExecutor service]
(.getCompletedTaskCount service)))
link
(let [exe (executor:single)] (try (exec:current-completed exe) (finally (exec:shutdown exe)))) => number?
v 3.0
(defn exec:current-size
([^ThreadPoolExecutor service]
(.getPoolSize service)))
link
(let [exe (executor:single)] (try (exec:current-size exe) (finally (exec:shutdown exe)))) => number?
v 3.0
(defn exec:current-submitted
([^ThreadPoolExecutor service]
(.getTaskCount service)))
link
(let [exe (executor:single)] (try (exec:current-submitted exe) (finally (exec:shutdown exe)))) => number?
v 3.0
(defn ^BlockingQueue exec:get-queue
([service]
(if (instance? ThreadPoolExecutor service)
(.getQueue ^ThreadPoolExecutor service)
(throw (ex-info "Cannot access queue" {:service service})))))
link
(let [queue (q/queue) exe (executor:pool 10 10 1000 queue)] (try (= queue (exec:get-queue exe)) (finally (exec:shutdown exe)))) => true
exec:increase-capacity ^
[executor] [executor n]
increases the capacity of the executor
v 3.0
(defn exec:increase-capacity
([executor]
(exec:increase-capacity executor (* 2 (exec:pool-max executor))))
([executor n]
(let [max (exec:pool-max executor)]
(if (< max n) (exec:pool-max executor n))
(exec:pool-size executor n))))
link
(let [exe (executor:single)] (try (doto exe (exec:increase-capacity)) (exec:pool-size exe) (finally (exec:shutdown exe)))) => 2
v 3.0
(defn exec:keep-alive
([^ThreadPoolExecutor service]
(.getKeepAliveTime service TimeUnit/MILLISECONDS))
([^ThreadPoolExecutor service length]
(.setKeepAliveTime service length TimeUnit/MILLISECONDS)))
link
(let [exe (executor:single)] (try (doto exe (exec:keep-alive 1000)) (exec:keep-alive exe) (finally (exec:shutdown exe)))) => 1000
v 3.0
(defn exec:pool-max
([^ThreadPoolExecutor service]
(.getMaximumPoolSize service))
([^ThreadPoolExecutor service size]
(.setMaximumPoolSize service size)))
link
(let [exe (executor:single)] (try (doto exe (exec:pool-max 10)) (exec:pool-max exe) (finally (exec:shutdown exe)))) => 10
v 3.0
(defn exec:pool-size
([^ThreadPoolExecutor service]
(.getCorePoolSize service))
([^ThreadPoolExecutor service size]
(.setCorePoolSize service (int size))))
link
(let [exe (executor:single)] (try (doto exe (exec:pool-max 20) (exec:pool-size 10)) (exec:pool-size exe) (finally (exec:shutdown exe)))) => 10
v 3.0
(defn ^BlockingQueue exec:queue
([]
(exec:queue nil))
([arg]
(cond (q/queue? arg) arg
(nil? arg) (q/queue)
(integer? arg) (q/queue:fixed arg)
(map? arg)
(cond (integer? (:size arg))
(q/queue:fixed (:size arg))
:else
(q/queue))
:else (throw (ex-info "Invalid input" {:input arg})))))
link
(exec:queue) (exec:queue 1) (exec:queue {:size 1}) (exec:queue {}) => java.util.concurrent.LinkedBlockingQueue (exec:queue (q/queue)) => java.util.concurrent.LinkedBlockingQueue
v 3.0
(defn exec:rejected-handler
([^ThreadPoolExecutor service]
(.getRejectedExecutionHandler service))
([^ThreadPoolExecutor service f]
(.setRejectedExecutionHandler service
(reify RejectedExecutionHandler
(rejectedExecution [_ task executor]
(f task executor))))))
link
(def -counter- (atom 0)) (doto (executor:single 1) (exec:rejected-handler (fn [_ _] (swap! -counter- inc))) (f/-> (doseq [i (range 10)] (submit % (fn [] (Thread/sleep 1000))))) (exec:shutdown-now)) (<= 8 @-counter-) => true (let [exe (executor:single)] (try (exec:rejected-handler exe) (finally (exec:shutdown exe)))) => anything
v 3.0
(defn exec:shutdown
([^ExecutorService service]
(doto service
(track/untrack)
(.shutdown))))
link
(-> (executor:single) (doto (submit (fn [] (Thread/sleep 1000)))) (doto (submit (fn [] (Thread/sleep 1000)))) (doto (exec:shutdown))) => (all exec:shutdown? exec:terminating? (comp not exec:terminated?))
v 3.0
(defn exec:shutdown-now
([^ExecutorService service]
(doto service
(track/untrack)
(.shutdownNow))))
link
(-> (executor:single) (doto (submit (fn [] (Thread/sleep 1000)))) (doto (submit (fn [] (Thread/sleep 1000)))) (exec:shutdown-now) (exec:shutdown?)) => true
v 3.0
(defn exec:shutdown?
([^ExecutorService service]
(.isShutdown service)))
link
(-> (executor:single) (doto (exec:shutdown)) (exec:shutdown?)) => true
exec:terminated? ^
[service]
checks if executor is shutdown and all threads have finished
v 3.0
(defn exec:terminated?
([^ExecutorService service]
(.isTerminated service)))
link
(-> (executor:single) (doto (exec:shutdown)) (doto (exec:await-termination)) (exec:terminated?)) => true
v 3.0
(defn exec:terminating?
([^ThreadPoolExecutor service]
(.isTerminating service)))
link
(-> (executor:single) (doto (submit (fn [] (Thread/sleep 1000)))) (doto (exec:shutdown)) (exec:terminating?)) => true
v 3.0
(defmulti executor
:type)
link
(doto (executor {:type :pool :size 3 :max 3 :keep-alive 1000}) (exec:shutdown)) (doto (executor {:type :single :size 1}) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor
executor-string ^
NONE
(defn- executor-string
[executor]
(let [tag (if (instance? ScheduledThreadPoolExecutor executor)
"scheduled"
"raw")]
(str "#" tag ".executor" (executor:info executor [:type :counter :current]))))
link
v 3.0
(defn ^ExecutorService executor:cached
([]
(doto (Executors/newCachedThreadPool)
(track/track))))
link
(doto (executor:cached) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor
v 3.0
(defn executor:health
([executor]
(if (exec:shutdown? executor)
{:status :not-healthy}
{:status :ok})))
link
(let [exe (executor:single)] (try (executor:health exe) (finally (exec:shutdown exe)))) => {:status :ok} (-> (executor:single) (exec:shutdown) (executor:health)) => {:status :not-healthy}
v 3.0
(defn executor:info
([^ThreadPoolExecutor executor]
(executor:info executor #{:type :running :current :counter :options}))
([^ThreadPoolExecutor executor k]
(let [[return items] (cond (keyword? k)
[k #{k}]
:else
[identity (set k)])
queue (.getQueue executor)]
(cond-> {}
(:type items) (assoc :type (executor:type executor))
(:running items) (assoc :running (not (exec:shutdown? executor)))
(:current items) (assoc :current {:threads (exec:current-size executor)
:active (exec:current-active executor)
:queued (count queue)
:terminated (exec:terminated? executor)})
(:counter items) (assoc :counter {:submit (exec:current-submitted executor)
:complete (exec:current-completed executor)})
(:options items) (assoc :options {:pool {:size (exec:pool-size executor)
:max (exec:pool-max executor)
:keep-alive (exec:keep-alive executor)}
:queue {:remaining (q/remaining-capacity queue)
:total (count queue)}})
:then return))))
link
(let [exe (executor:single)] (try (executor:info exe) (finally (exec:shutdown exe)))) => {:type :single, :running true, :current {:threads 0, :active 0, :queued 0, :terminated false}, :counter {:submit 0, :complete 0}, :options {:pool {:size 0, :max 1, :keep-alive 0}, :queue {:remaining 2147483647, :total 0}}}
executor:pool ^
[size max keep-alive] [size max keep-alive size-or-queue]
constructs a pool executor
v 3.0
(defn ^ExecutorService executor:pool
([size max keep-alive]
(executor:pool size max keep-alive (q/queue)))
([^long size ^long max ^long keep-alive size-or-queue]
(doto (ThreadPoolExecutor. size max keep-alive
TimeUnit/MILLISECONDS (exec:queue size-or-queue))
(track/track))))
link
(doto (executor:pool 10 10 1000 {:size 10}) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor
v 3.0
(defn executor:props
([^ThreadPoolExecutor executor]
{:pool {:size {:get exec:pool-size
:set exec:pool-size}
:max {:get exec:pool-max
:set exec:pool-max}
:keep-alive {:get exec:keep-alive
:set exec:keep-alive}}
:rejected-handler {:get exec:rejected-handler
:set exec:rejected-handler}}))
link
(let [exe (executor:single)] (try (executor:props exe) (finally (exec:shutdown exe)))) => {:pool {:size {:get exec:pool-size :set exec:pool-size}, :max {:get exec:pool-max :set exec:pool-max}, :keep-alive {:get exec:keep-alive :set exec:keep-alive}} :rejected-handler {:get exec:rejected-handler :set exec:rejected-handler}}
v 3.0
(defn ^ScheduledThreadPoolExecutor executor:scheduled
([^long size]
(doto (ScheduledThreadPoolExecutor. size)
(track/track))))
link
(doto (executor:scheduled 10) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor
v 3.0
(defn executor:share
([id ^ExecutorService executor]
(swap! *shared* (fn [m]
(if (or (get @f/*pools* id)
(get m id))
(throw (ex-info "Already exists" {:id id}))
(assoc m id executor))))))
link
(let [exe (executor:single)] (try (executor:share :test exe) (get (executor:shared) :test) (finally (executor:unshare :test) (exec:shutdown exe)))) => anything (executor:unshare :test)
v 3.0
(defn executor:shared
([]
(merge @*shared* @f/*pools*)))
link
(-> (executor:shared) keys sort) => [:async :default :pooled]
v 3.0
(defn ^ExecutorService executor:single
([]
(executor:single (q/queue)))
([size-or-queue]
(doto (ThreadPoolExecutor. 0 1 0 TimeUnit/MILLISECONDS (exec:queue size-or-queue))
(track/track))))
link
;; any sized pool (doto (executor:single) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor ;; fixed pool (doto (executor:single {:size 10}) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor
v 3.0
(defn executor:type
([^ThreadPoolExecutor executor]
(cond (instance? ScheduledThreadPoolExecutor executor)
:scheduled
(= 1 (exec:pool-max executor))
:single
(and (= Integer/MAX_VALUE (exec:pool-max executor))
(let [queue (exec:get-queue executor)]
(and (zero? (q/remaining-capacity queue))
(zero? (count queue)))))
:cached
:else :pool)))
link
(let [exe (executor:single)] (try (executor:type exe) (finally (exec:shutdown exe)))) => :single
v 3.0
(defn executor:unshare
([id]
(atom/swap-return! *shared* (fn [m] [(get m id) (dissoc m id)]))))
link
(let [exe (executor:single)] (try (executor:share :test exe) (executor:unshare :test) (get (executor:shared) :test) (finally (executor:unshare :test) (exec:shutdown exe)))) => nil
v 4.0
(defn hub-state
[elems]
{:ticket (f/incomplete)
:queue elems})
link
(hub-state [1 2 3]) => (contains {:ticket f/future? :queue [1 2 3]})
v 3.0
(defn hub:add-entries
([hub entries]
(at/swap-return! hub
(fn [{:keys [ticket queue] :as m}]
(let [start (count queue)]
[[ticket start (count entries)] {:ticket ticket
:queue (into queue entries)}])))))
link
(hub:add-entries (hub:new) [1 2 3 4 5]) => (contains [f/future? 0 5])
hub:executor ^
[object {:keys [handler max-batch interval], :as opts}]
creates a hub based executor
v 3.0
(defn hub:executor
([object {:keys [handler max-batch interval]
:as opts}]
(let [queue (hub:new)
executor (exe/executor:single 1)
submit (hub:submit object executor queue opts)]
{:queue queue
:executor executor
:submit submit})))
link
(let [-exe- (hub:executor nil {:handler (fn [& args] args) :interval 50 :max-batch 1000}) -exec- (:executor -exe-)] (try (do (def -res- ((:submit -exe-) 1 2 3 4 5 6)) @(first -res-)) => '[(nil (1 2 3 4 5 6))] (finally (executor/exec:shutdown -exec-))))
v 3.0
(defn hub:new
([]
(hub:new []))
([elems]
(atom (hub-state elems))))
link
@(hub:new [1 2 3]) => (contains {:ticket f/future? :queue [1 2 3]})
v 3.0
(defn hub:process
([f hub max-batch]
(let [[ticket queue] (at/swap-return! hub
(fn [{:keys [ticket queue]}]
[[ticket queue] (hub-state [])]))]
(->> (partition-all max-batch queue)
(mapv f)
(f/future:force ticket)))))
link
(let [+state+ (atom [])] (hub:process (fn [elems] (swap! +state+ conj elems)) (hub:new [1 2 3 4 5]) 3) @+state+) => [[1 2 3] [4 5]]
hub:submit ^
[object executor hub {:keys [handler max-batch interval]}]
submission function for the hub
v 3.0
(defn hub:submit
([object executor hub {:keys [handler max-batch interval]}]
(let [bulk-fn (fn [] (hub:process (fn [items]
(handler object items))
hub
max-batch))]
(fn [& entries]
(let [[ticket start count] (hub:add-entries hub entries)
hit (exe/submit-notify executor
bulk-fn
interval)]
[ticket start count hit])))))
link
(let [-out- (atom []) -exe- (executor/executor:single) -hub- (hub:new) -sub- (hub:submit nil -exe- -hub- {:handler (fn [_ items] (swap! -out- concat items) items) :interval 10 :max-batch 10})] (try (let [[ticket] (-sub- 1 2 3)] @ticket) @-out- (finally (executor/exec:shutdown -exe-)))) => [1 2 3]
v 4.0
(defn hub:wait
[hub & [timeout]]
(let [{:keys [ticket queue]} @hub]
(if (not (zero? (count queue)))
(deref ticket (or timeout 1000) :timeout))))
link
(let [-hub- (hub:new [1 2 3])] (f/future (Thread/sleep 50) (swap! -hub- update :ticket f/future:force true)) (hub:wait -hub-)) => true
v 3.0
(defn peek
([^BlockingQueue queue]
(.peek queue)))
link
(let [queue (q/queue 1 2 3)] [(q/peek queue) (vec queue)]) => [1 [1 2 3]]
v 3.0
(defn peek-first
([^BlockingDeque queue]
(.peekFirst queue)))
link
(q/peek-first (q/deque 1 2 3)) => 1
v 3.0
(defn peek-last
([^BlockingDeque queue]
(.peekLast queue)))
link
(q/peek-last (q/deque 1 2 3)) => 3
pool ^
[{:keys [size max keep-alive poll resource], :as m}]
creates and starts the pool
v 3.0
(defn pool
([{:keys [size max keep-alive poll resource] :as m}]
(-> (pool:create m)
(pool:start))))
link
(component/with [p (pool {:size 2 :max 10 :keep-alive 10000 :poll 20000 :resource {:create (fn [] (rand)) :initial 0.3 :thread-local true}})] p => pool?)
v 3.0
(defn pool:acquire
([{:keys [state resource] :as pool}]
(let [{:keys [thread-local]} resource
resource-fn (fn [resource thread-id]
(-> resource
(update :used inc)
(assoc :update-time (time/time-ns)
:status :busy
:thread-id thread-id)))
update-fn (fn [m thread-id id resource]
(-> m
(update :busy assoc id
(resource-fn resource thread-id))
(update-in [:stats :acquire] inc)
(update-in [:lookup thread-id]
(fnil #(conj % id) #{}))))
acquire-fn (fn [{:keys [busy idle options lookup] :as m}]
(let [{:keys [max]} options
[id resource] (first idle)
thread-id (t/thread:id)
local-id (first (get lookup thread-id))]
(cond (and thread-local local-id)
[[:success local-id
(f/completed (:object (get busy local-id)))]
m]
id
[[:success id
(f/completed (:object resource))]
(-> (update-fn m thread-id id resource)
(update :idle dissoc id))]
(and max
(<= max
(+ (count busy)
(count idle))))
[[:reject nil (ex-info "Maximum limit reached" {:max max})]
(update-in m [:stats :reject] inc)]
:else
[[:success (std.lib.foundation/sid) (f/incomplete)]
(update-in m [:stats :create] inc)])))
[status id boxed] (atom/swap-return! state acquire-fn)
allocate-fn (fn [state id boxed]
(f/fulfil boxed
(fn []
(let [resource (pool-resource id pool)]
(swap! state (fn [m] (update-fn m (t/thread:id) id resource)))
(:object resource)))))
object (if (f/future? boxed)
(do (if (not (f/future:complete? boxed))
(allocate-fn state id boxed))
@boxed)
boxed)]
(case status
:success [id object]
:reject (throw object)))))
link
(component/with [|pool| (create-pool)] (pool:acquire |pool|)) => (contains [string? '])
v 3.0
(defn pool:cleanup
([{:keys [state] :as pool}]
(let [current (time/time-ns)
{:keys [stop]} (:resource pool)
{:keys [health] :or {health (constantly {:status :ok})}} (:resource pool)
{:keys [idle]} @state
dead-ids (keep (fn [[id resource]]
(if-not (-> (health resource)
(:status)
(= :ok))
id))
idle)
dispose (fn [m id] (dispose-fn m id stop))
_ (swap! state (fn [m] (reduce dispose m dead-ids)))
cleanup-fn (fn [{:keys [idle options] :as m}]
(let [{:keys [size keep-alive]} options
over (- (count idle) size)
[ids m] (cond (pos? over)
(let [ids (->> idle
(keep (fn [[id {:keys [update-time]}]]
(if (< (+ update-time keep-alive)
current)
id)))
(take over))]
[ids (reduce dispose m ids)])
:else
[[] m])]
[ids (update-in m [:stats :cleanup] inc)]))
ids (atom/swap-return! state cleanup-fn)]
ids)))
link
(component/with [|pool| (create-pool)] (def -ids- (->> (for [i (range 8)] (pool:acquire |pool|)) (mapv first))) (count (pool:resources:busy |pool|)) => 8 (doseq [id -ids-] (pool:release |pool| id)) (count (pool:resources:busy |pool|)) => 0 (pool:cleanup |pool|) (count (pool:resources:idle |pool|)) => 3)
v 3.0
(defn pool:create
([m]
(-> (assoc (apply dissoc m +opts+)
:state (atom {:stats {:create 0
:reject 0
:dispose 0
:acquire 0
:release 0
:history []
:cleanup 0
:total-time 0.0
:busy-time 0.0}
:busy {}
:idle {}
:lookup {}
:options (merge *defaults* (select-keys m +opts+))}))
(map->Pool))))
link
(let [pool (pool:create {:size 5 :max 8 :keep-alive 10000 :poll 20000 :resource {:create (fn [] ') :initial 0.3 :thread-local true}})] [(pool? pool) (f/atom? (:state pool)) (-> pool :resource :thread-local)]) => [true true true]
v 3.0
(defn pool:dispose
([{:keys [state resource] :as pool} id]
(swap! state dispose-fn id (:stop resource)) id))
link
(component/with [|pool| (create-pool)] (let [id (first (keys (pool:resources:idle |pool|)))] (pool:dispose |pool| id) (contains? (pool:resources:idle |pool|) id))) => false
pool:dispose-over ^
[{:keys [state resource], :as pool} id]
disposes if idle and busy are over size limit
v 3.0
(defn pool:dispose-over
([{:keys [state resource] :as pool} id]
(atom/swap-return! state (fn [{:keys [idle options] :as m}]
(if (get idle id)
[id (dispose-fn m id (:stop resource))]
[nil m])))))
link
(component/with [|pool| (create-pool)] (let [[id _] (pool:acquire |pool|)] (pool:release |pool| id) (pool:dispose-over |pool| id))) => string?
v 3.0
(defn pool:dispose:mark
([]
(vreset! *dispose* true)))
link
(component/with [|pool| (create-pool)] (dotimes [i 2] ((wrap-pool-resource (fn [pool] (pool:dispose:mark)) |pool|))) (count (pool:resources:idle |pool|)) => 0)
v 3.0
(defn pool:dispose:unmark
([]
(vreset! *dispose* nil)))
link
(component/with [|pool| (create-pool)] (dotimes [i 2] ((wrap-pool-resource (fn [pool] (pool:dispose:mark) (pool:dispose:unmark)) |pool|))) (count (pool:resources:idle |pool|)) => 2)
pool:release ^
[pool id] [{:keys [state], :as pool} id dispose?]
releases a resource back to the pool
v 3.0
(defn pool:release
([pool id]
(pool:release pool id false))
([{:keys [state] :as pool} id dispose?]
(let [resource-fn (fn [{:keys [update-time busy] :as resource}]
(let [current (time/time-ns)
diff (- current update-time)]
(assoc resource
:update-time current
:busy (+ busy diff)
:status :idle
:thread-id nil)))
update-fn (fn [m thread-id id resource]
(-> m
(update :idle assoc id (resource-fn resource))
(update :busy dissoc id)
(update-in [:stats :release] inc)
(update :lookup
(fn [lu]
(let [nentries (disj (get lu thread-id) id)]
(if (empty? nentries)
(dissoc lu thread-id)
(assoc lu thread-id nentries)))))))
release-fn (fn [{:keys [busy options] :as m}]
(let [{:keys [thread-id] :as resource} (get busy id)]
(cond (nil? resource)
[[:error (ex-info "Cannot find id" {:id id})]
m]
:else
[[:success id]
(update-fn m thread-id id resource)])))
[[status id] {:keys [idle busy options executor]}] (atom/swap-return! state release-fn true)
_ (if dispose? (pool:dispose pool id))
_ (if (and executor
(< (:size options)
(+ (count idle)
(count busy))))
(f/future:run (fn []
(pool:dispose-over pool id))
{:pool executor
:delay (:keep-alive options)}))]
(case status
:success id
:error nil))))
link
(component/with [|pool| (create-pool)] (let [[id _] (pool:acquire |pool|)] (pool:release |pool| id))) => string?
v 3.0
(defn pool:resources:busy
([{:keys [state] :as pool}]
(collection/map-vals :object (:busy @state))))
link
(component/with [|pool| (create-pool)] (pool:resources:busy |pool|)) => {} (component/with [|pool| (create-pool)] (-> (doto |pool| (pool:acquire) (pool:acquire)) (pool:resources:busy) count)) => 2
v 3.0
(defn pool:resources:idle
([{:keys [state] :as pool}]
(collection/map-vals :object (:idle @state))))
link
(component/with [|pool| (create-pool)] (count (pool:resources:idle |pool|))) => 2 (component/with [|pool| (create-pool)] (-> (doto |pool| (pool:acquire) (pool:acquire)) (pool:resources:idle))) => {}
pool:resources:thread ^
[pool] [{:keys [state], :as pool} thread]
returns acquired resources for a given thread
v 3.0
(defn pool:resources:thread
([pool]
(pool:resources:thread pool (t/thread:current)))
([{:keys [state] :as pool} thread]
(let [thread-id (t/thread:id thread)
{:keys [busy lookup]} @state
ids (get lookup thread-id)]
(collection/map-juxt [identity (comp :object #(get busy %))] ids))))
link
(component/with [|pool| (create-pool)] (-> (doto |pool| (pool:acquire) (pool:acquire)) (pool:resources:thread) count)) => 2
pool:with-resource ^
[[obj pool] & body]
takes an object from the pool, performs operation then returns it
v 3.0
(defmacro pool:with-resource
([[obj pool] & body]
`((wrap-pool-resource
(fn [obj#]
(let [~obj obj#]
~@body))
~pool))))
link
(component/with [|pool| (create-pool)] (pool:with-resource [obj |pool|] (str obj)) => "")
v 3.0
(defn pool?
([obj]
(instance? Pool obj)))
link
(component/with [|pool| (create-pool)] (pool? |pool|)) => true
v 3.0
(defn process-bulk
([f queue maximum]
(loop [total (count queue)]
(let [n (if (> total maximum)
maximum
total)
txs (drain queue n)
_ (when-not (empty? txs)
(f txs))]
(let [more (count queue)]
(if-not (zero? more)
(recur more)))))))
link
(def +state+ (atom [])) (q/process-bulk (fn [elems] (swap! +state+ conj elems)) (q/queue 1 2 3 4 5) 3) @+state+ => [[1 2 3] [4 5]]
v 3.0
(defn push
([^BlockingDeque queue element]
(.push queue element)))
link
(-> (doto (q/deque) (q/push 1)) (vec)) => [1]
put ^
[queue element] [queue element timeout] [queue element timeout timeunit]
puts an element at the back
v 3.0
(defn put
([^BlockingQueue queue element]
(put queue element nil))
([^BlockingQueue queue element timeout]
(put queue element timeout :ms))
([^BlockingQueue queue element timeout timeunit]
(if (nil? timeout)
(.put queue element)
(.offer queue element timeout (->timeunit timeunit)))))
link
(->> (doto (q/queue) (q/put 1)) (into [])) => [1]
put-first ^
[queue element] [queue element timeout] [queue element timeout timeunit]
puts at the front of the queue
v 3.0
(defn put-first
([^BlockingDeque queue element]
(put-first queue element nil))
([^BlockingDeque queue element timeout]
(put-first queue element timeout :ms))
([^BlockingDeque queue element timeout timeunit]
(if (nil? timeout)
(.putFirst queue element)
(.offerFirst queue element timeout (->timeunit timeunit)))))
link
(-> (doto (q/deque) (q/put-first 1)) (vec)) => [1]
put-last ^
[queue element] [queue element timeout] [queue element timeout timeunit]
puts at the back of the queue
v 3.0
(defn put-last
([^BlockingDeque queue element]
(put-last queue element nil))
([^BlockingDeque queue element timeout]
(put-last queue element timeout :ms))
([^BlockingDeque queue element timeout timeunit]
(if (nil? timeout)
(.putLast queue element)
(.offerLast queue element timeout (->timeunit timeunit)))))
link
(-> (doto (q/deque) (q/put-last 1)) (vec)) => [1]
v 3.0
(defn ^BlockingQueue queue
([]
(LinkedBlockingQueue.))
([& elements]
(LinkedBlockingQueue. ^java.util.Collection elements)))
link
(q/queue 1 2 3) => java.util.concurrent.LinkedBlockingQueue
v 3.0
(defn ^BlockingQueue queue:fixed
([size]
(ArrayBlockingQueue. size)))
link
(type (q/queue:fixed 10)) => java.util.concurrent.ArrayBlockingQueue
v 3.0
(defn ^LimitedQueue queue:limited
([size]
(LimitedQueue. size)))
link
(q/queue? (q/queue:limited 10)) => true
v 3.0
(defn queue?
([obj]
(instance? BlockingQueue obj)))
link
(q/queue? (q/queue)) => true (q/queue? []) => false
v 3.0
(defn relay
([{:keys [type] :as m}]
(-> (relay:create m)
(component/start))))
link
(relay {:type :process :args ["bash"]}) => map?
v 4.0
(defn relay:create
([{:keys [id type] :as m}]
(map->Relay (assoc m
:id (or id (f/sid))
:instance (atom nil)))))
link
(relay:create {:type :process :args ["bash"]}) => map?
v 3.0
(defn remaining-capacity
([^BlockingQueue queue]
(.remainingCapacity queue)))
link
(q/remaining-capacity (q/queue)) => 2147483647 (q/remaining-capacity (doto (q/queue:fixed 2) (q/put 1))) => 1
v 3.0
(defn remove
([^BlockingQueue queue obj]
(.remove queue obj)))
link
(let [q (q/deque 1 2 3)] (q/remove q 2) (into [] q)) => [1 3]
req ^
[client command] [client command opts]
execute command on single, bulk and transact calls
v 3.0
(defmacro req
([client command]
(let [context (-> (meta &form)
(assoc :ns (env/ns-sym)))]
`(req-fn ~client ~command {:context (quote ~context)})))
([client command opts]
(let [context (-> (meta &form)
(assoc :ns (env/ns-sym)))]
`(req-fn ~client ~command (assoc ~opts :context (quote ~context))))))
link
(req |client| {:type :eval :form 1}) => 1 (req |client| {:type :transact/start}) => :transact/start (req |client| {:type :eval :form 1}) => :transact/queued (req |client| {:type :eval :form 1}) => :transact/queued (req |client| {:type :transact/end}) => [1 1]
v 3.0
(defn req-fn
([client command]
(req-fn client command {}))
([client command opts]
(let [bulk (get *bulk* client)
transact (get *transact* client)
opts (req:opts-init opts)
opts (if transact
(transact-prep client transact command opts)
opts)]
(cond *inputs*
(swap! *inputs* conj command)
bulk
(req:unit client command opts)
:else
(req:single client command opts)))))
link
(binding [*inputs* (atom [])] (req-fn :client {:type :eval :form 1}) @*inputs*) => [{:type :eval :form 1}]
v 3.0
(defn req:applicative
([m]
(map->ReqApplicative m)))
link
(req:applicative {:type :single :client :demo}) => (contains {:type :single :client :demo})
v 3.0
(defmacro req:bulk
([[client opts] & body]
(let [context (-> (meta &form)
(assoc :ns (env/ns-sym)
:type :bulk))]
`(bulk ~client
(fn []
~@body)
(assoc ~opts :context (quote ~context))))))
link
(req:bulk [+] (req + [1 2 3]) (req + [1 2 3]) (req - [4 5 6])) => [6 6]
generated
;; generated at runtime, no single source form
link
generated
;; generated at runtime, no single source form
link
v 3.0
(defn req:command
([m]
(map->Command m)))
link
(req:command {:type :single :name :echo}) => (contains {:type :single :name :echo})
v 3.0
(defmacro req:in
([& body]
`(binding [*inputs* (atom [])]
((fn [] ~@body))
@*inputs*)))
link
(req:in (req nil [1]) (req nil [2]) (req nil [3])) => [[1] [2] [3]]
req:opts ^
[opts] [opts {:keys [pre post chain], :as m}]
clean opts for processing inputs
v 3.0
(defn req:opts
([opts]
(req:opts opts nil))
([opts {:keys [pre post chain] :as m}]
(cond-> opts
m (merge (dissoc m :chain :post :pre))
pre (update :pre #(vec (into % pre)))
post (update :post (comp vec (partial concat post)))
chain (update :chain (comp vec (partial concat chain))))))
link
(req:opts {} {}) => {}
v 3.0
(defn req:opts-clean
([opts]
(dissoc opts :measure :time :debug :pre :post
:chain :async :final :received :transact)))
link
(req:opts-clean {}) => {}
generated
;; generated at runtime, no single source form
link
generated
;; generated at runtime, no single source form
link
req:run ^
[command client args] [{:keys [options process], :as command} client args opts]
runs a command
v 3.0
(defn req:run
([command client args]
(req:run command client args {}))
([{:keys [options process] :as command} client args opts]
(let [input-fn (or (:input options) f/NIL)
output-fn (or (:output options) f/NIL)
input (format-input command args (merge (input-fn args) opts))
out-fn #(format-output command % (merge (output-fn args) opts))
opts (-> (merge (:default options) opts)
(r/req:opts process)
(r/req:opts {:post [out-fn]}))]
(run-request command client input opts))))
link
(with-redefs [run-request (fn [_ client input opts] [client input opts])] (let [[client input opts] (req:run {:options {:select [:id] :input (fn [args] {:id (:id args)}) :output (fn [_] {:id 2})} :format {:input (fn [input opts] [input opts]) :output (fn [output opts] [output opts])} :process {:chain [inc]}} :client {:id 1 :value 2} {:extra true})] [client input (map? opts) (= [inc] (:chain opts)) (= 1 (count (:post opts))) (fn? (first (:post opts)))])) => [:client [{:id 1, :value 2} {:id 1}] true true true true]
v 3.0
(defmacro req:transact
([[client opts] & body]
(let [context (-> (meta &form)
(assoc :ns (env/ns-sym)
:type :transact))]
`(bulk:transact ~client
(fn []
~@body)
(assoc ~opts :context (quote ~context))))))
link
(-> (req:transact [|client| {:debug true}] (req |client| {:type :eval :form 1} {:debug true}) (req |client| {:type :eval :form 2} {:debug true})) (env/with-out-str)) => string?
schedule ^
[service f interval] [service f interval {:keys [min]}]
schedules task for execution
v 3.0
(defn schedule
([^ScheduledThreadPoolExecutor service ^Callable f ^long interval]
(schedule service f interval nil))
([^ScheduledThreadPoolExecutor service ^Callable f ^long interval {:keys [min]}]
(let [^Callable f (cond-> f
min (wrap-min-time min))]
(.schedule service f interval TimeUnit/MILLISECONDS))))
link
(f/-> (doto (executor:scheduled 1) (schedule (fn [] (Thread/sleep 10)) 10) (schedule (fn [] (Thread/sleep 10)) 20) (schedule (fn [] (Thread/sleep 10)) 30) (schedule (fn [] (Thread/sleep 10)) 40)) (do (Thread/sleep 100) (exec:current-completed %))) => 4
schedule:fixed-delay ^
[service f delay] [service f delay {:keys [initial min]}]
schedules task at fixed delay
v 3.0
(defn schedule:fixed-delay
([^ScheduledThreadPoolExecutor service ^Runnable f ^long delay]
(schedule:fixed-delay service f delay nil))
([^ScheduledThreadPoolExecutor service ^Runnable f ^long delay {:keys [initial min]}]
(let [^Runnable f (cond-> f
min (wrap-min-time min))]
(.scheduleWithFixedDelay service f (or initial 0) delay TimeUnit/MILLISECONDS))))
link
(def -counter- (atom 0)) (f/-> (doto (executor:scheduled 1) (schedule:fixed-delay (fn [] (Thread/sleep 20)) 30)) (do (Thread/sleep 100) (exec:shutdown-now %) (exec:current-completed %))) => #(<= 1 %)
schedule:fixed-rate ^
[service f interval] [service f interval {:keys [initial min]}]
schedules task at a fixed rate
v 3.0
(defn schedule:fixed-rate
([^ScheduledThreadPoolExecutor service ^Runnable f ^long interval]
(schedule:fixed-rate service f interval nil))
([^ScheduledThreadPoolExecutor service ^Runnable f ^long interval {:keys [initial min]}]
(let [^Runnable f (cond-> f
min (wrap-min-time min))]
(.scheduleAtFixedRate service f (or initial 0) interval TimeUnit/MILLISECONDS))))
link
(def -counter- (atom 0)) (f/-> (doto (executor:scheduled 1) (schedule:fixed-rate (fn [] (Thread/sleep 20)) 30)) (do (Thread/sleep 100) (exec:shutdown-now %) (exec:current-completed %))) => #(<= 3 %)
v 3.0
(defn send
([{:keys [instance] :as relay} msg]
(let [{:keys [in out]} (or @instance
(f/error "Relay not started"
{:relay relay :msg msg}))
{:keys [op line]
:as msg} (if (string? msg)
{:op :read-some
:line msg}
msg)
op (or (transport/+read-alias+ op)
(+read-ops+ op)
(+control-ops+ op)
(f/error "Op not valid." {:input op
:available (set (concat +control-ops+
+read-ops+))
:alias transport/+read-alias+}))]
(case op
:kill (relay-stop relay)
:exit (do (.close ^OutputStream (:raw out))
(bus/bus:close (:bus in) (:id in)))
:flush (transport/send-write-flush out)
:raw (transport/send-write-raw out line)
:partial (transport/send-write-line out line)
(transport/send-command in out op line msg)))))
link
(let [r (relay {:type :process :args ["bash"]})] (try (-> @(send r "echo hello") :output) => "hellon" (finally (relay-stop r))))
v 3.0
(defn stacktrace
([]
(stacktrace (thread:current)))
([^Thread thread]
(.getStackTrace thread)))
link
(vec (stacktrace)) => vector?
submit ^
[service f] [service f {:keys [min max delay default], :as m}]
submits a task to an executor
v 3.0
(defn submit
([^ExecutorService service ^Callable f]
(submit service f nil))
([^ExecutorService service f {:keys [min max delay default] :as m}]
(let [^Callable f (cond-> f
min (wrap-min-time min (or delay 0)))
opts (cond-> {:pool service}
max (assoc :timeout max)
default (assoc :default default)
delay (assoc :delay delay))]
(f/future:run f opts))))
link
(let [exe (executor:single)] (try @(submit exe (fn []) {:min 100}) (finally (exec:shutdown exe)))) (let [exe (executor:single)] (try (try @(submit exe (fn [] (Thread/sleep 1000)) {:max 100}) (catch java.util.concurrent.ExecutionException e (if (instance? java.util.concurrent.TimeoutException (.getCause e)) :timeout (throw e)))) (finally (exec:shutdown exe)))) => :timeout
submit-notify ^
[service f] [service f {:keys [min max delay default], :as m}]
submits a task (generally to a fixed size queue)
v 3.0
(defn submit-notify
([^ExecutorService service ^Callable f]
(submit-notify service f nil))
([^ExecutorService service f {:keys [min max delay default] :as m}]
(if (pos? (q/remaining-capacity (exec:get-queue service)))
(h/suppress (submit service f m)))))
link
(doto (executor:single 1) (submit-notify (fn []) 1000) (submit-notify (fn []) 1000) (submit-notify (fn []) 1000) (exec:shutdown)) => java.util.concurrent.ThreadPoolExecutor
take ^
[queue] [queue timeout] [queue timeout timeunit]
takes an element from the queue
v 3.0
(defn take
([^BlockingQueue queue]
(take queue nil))
([^BlockingQueue queue timeout]
(take queue timeout :ms))
([^BlockingQueue queue timeout timeunit]
(if (nil? timeout)
(.take queue)
(.poll queue timeout (->timeunit timeunit)))))
link
((juxt q/take #(into [] %)) (q/queue 1 2 3)) => [1 [2 3]]
take-first ^
[queue] [queue timeout] [queue timeout timeunit]
takes from the front of the queue
v 3.0
(defn take-first
([^BlockingDeque queue]
(.takeFirst queue))
([^BlockingDeque queue timeout]
(take-first queue timeout :ms))
([^BlockingDeque queue timeout timeunit]
(if (nil? timeout)
(.takeFirst queue)
(.pollFirst queue timeout (->timeunit timeunit)))))
link
(q/take-first (q/deque 1 2 3)) => 1
take-last ^
[queue] [queue timeout] [queue timeout timeunit]
takes from the back of the queue
v 3.0
(defn take-last
([^BlockingDeque queue]
(.takeLast queue))
([^BlockingDeque queue timeout]
(take-last queue timeout :ms))
([^BlockingDeque queue timeout timeunit]
(if (nil? timeout)
(.takeLast queue)
(.pollLast queue timeout (->timeunit timeunit)))))
link
(q/take-last (q/deque 1 2 3)) => 3
thread ^
[{:keys [handler daemon priority classloader uncaught name start]}]
creates a new thread
v 3.0
(defn ^Thread thread
([{:keys [^Runnable handler ^bool daemon priority classloader uncaught name start]}]
(cond-> (Thread. handler)
name (doto (.setName name))
daemon (doto (.setDaemon daemon))
priority (doto (.setPriority priority))
classloader (doto (.setContextClassLoader classloader))
uncaught (doto (.setUncaughtExceptionHandler uncaught))
start (doto (.start)))))
link
(let [thread (thread {:handler (fn []) :name "hello-thread" :daemon true})] [(.getName thread) (thread:daemon? thread)]) => ["hello-thread" true]
v 3.0
(defn thread:alive?
([^Thread thread]
(.isAlive thread)))
link
(thread:alive? (thread:current)) => true
v 3.0
(defn thread:all-ids
([]
(set (map #(.getId ^Thread %) (thread:all)))))
link
(thread:all-ids) => set?
thread:classloader ^
[] [thread] [thread loader]
gets and sets the context classloader
v 3.0
(defn thread:classloader
([]
(thread:classloader (thread:current)))
([^Thread thread]
(.getContextClassLoader thread))
([^Thread thread loader]
(.setContextClassLoader thread loader)))
link
(let [thread (thread:current) original (thread:classloader thread) loader (.getContextClassLoader (Thread/currentThread))] (try (thread:classloader thread loader) (= loader (thread:classloader thread)) (finally (thread:classloader thread original)))) => true
v 3.0
(defn thread:daemon?
([]
(thread:daemon? (thread:current)))
([^Thread thread]
(.isDaemon thread)))
link
(thread:daemon? (thread:current)) => boolean?
v 3.0
(defn thread:global-uncaught
([]
(Thread/getDefaultUncaughtExceptionHandler))
([f]
(Thread/setDefaultUncaughtExceptionHandler f)))
link
(let [original (thread:global-uncaught) handler (reify Thread$UncaughtExceptionHandler (uncaughtException [_ _ _] nil))] (try (thread:global-uncaught handler) (= handler (thread:global-uncaught)) (finally (thread:global-uncaught original)))) => true
v 3.0
(defn thread:has-access?
([^Thread thread]
(try
(.checkAccess thread)
true
(catch SecurityException e
false))))
link
(thread:has-access? (thread:current)) => true
v 3.0
(defn thread:has-lock?
([^Object lock]
(Thread/holdsLock lock)))
link
(let [lock (Object.)] (locking lock (thread:has-lock? lock))) => true
v 3.0
(defn thread:id
([] (thread:id (thread:current)))
([^Thread thread]
(.getId thread)))
link
(thread:id) => number?
v 3.0
(defn thread:interrupt
([^Thread thread]
(.interrupt thread)))
link
(doto (thread {:handler (fn [] (f/suppress (Thread/sleep 100))) :start true}) (thread:interrupt)) => Thread
v 3.0
(defn thread:interrupted?
([^Thread thread]
(.isInterrupted thread)))
link
(thread:interrupted? (thread:current)) => false
thread:join ^
[] [thread] [thread millis] [thread millis nanos]
calls join on a thread
v 3.0
(defn thread:join
([]
(thread:join (thread:current)))
([^Thread thread]
(.join thread))
([^Thread thread ^long millis]
(.join thread millis))
([^Thread thread millis nanos]
(.join thread millis nanos)))
link
(thread:join (thread {:handler (fn []) :start true})) => nil
v 3.0
(defn thread:notify
([^Object lock]
(locking lock (.notify lock))))
link
(let [lock (Object.)] (thread:notify lock)) => nil
v 3.0
(defn thread:notify-all
([^Object lock]
(locking lock (.notifyAll lock))))
link
(let [lock (Object.)] (thread:notify-all lock)) => nil
v 3.0
(defn thread:run
([^Thread thread]
(.run thread)))
link
(-> (thread {:handler (fn [])}) (thread:run)) => nil
v 3.0
(defn thread:start
([^Thread thread]
(.start thread)))
link
(let [started (promise) t (thread {:handler (fn [] (deliver started true) (thread:sleep 50))})] (thread:start t) [(deref started 100 false) (do (thread:join t) true)]) => [true true]
v 3.0
(defn thread:uncaught
([]
(thread:uncaught (thread:current)))
([^Thread thread]
(.getUncaughtExceptionHandler thread))
([^Thread thread f]
(.setUncaughtExceptionHandler thread f)))
link
(let [handler (reify Thread$UncaughtExceptionHandler (uncaughtException [_ _ _] nil)) thread (thread {:handler (fn [])})] (thread:uncaught thread handler) (= handler (thread:uncaught thread))) => true
v 3.0
(defn thread:wait-on
([^Object lock]
(locking lock (.wait lock))))
link
(let [lock (Object.)] (future (thread:sleep 500) (thread:notify lock)) (thread:wait-on lock)) => nil
v 3.0
(defn transact
([client thunk]
(let [top (get *transact* client)
context (or top (transact-context))
start-cmd (transact-start client)
end-cmd (transact-end client)
_ (if (nil? top)
(req-fn client start-cmd {:async true}))
results (binding [*transact* (assoc *transact* client context)]
(thunk)
(:outputs @context))
final (-> @context :inputs :final)
output (if (nil? top)
(req-fn client end-cmd {:async true}))
_ (if output
(f/on:complete output
(fn [val err]
(if err
(f/future:force final :exception err)
(f/future:force final :value val)))))]
results)))
link
(->> (transact |client| (fn [] (req |client| {:type :eval :form 1} {:chain [#(* 8 %)]}))) (map deref)) => '(8) ;; ;; bulk in transact ;; (->> (transact |client| (fn [] (bulk |client| (fn [] (transact |client| (fn [] (req |client| {:type :eval :form 1} {}))) (req |client| {:type :eval :form 2} {}))) (bulk |client| (fn [] (req |client| {:type :eval :form 3} {}) (req |client| {:type :eval :form 4} {}))))) (map deref)) => '(1 2 3 4) ;; ;; Nesting bulk and transact ;; (bulk |client| (fn [] (req |client| {:type :eval :form 1} {:chain [#(* 8 %)]}) (transact |client| (fn [] (req |client| {:type :eval :form 1} {:chain [#(* 2 %)]}) (req |client| {:type :eval :form 1} {:chain [#(* 2 %)]}))) (transact |client| (fn [] (bulk |client| (fn [] (req |client| {:type :eval :form 2} {:chain [#(* 2 %)]}))))) (transact |client| (fn [] (req |client| {:type :eval :form 3} {:chain [#(* 2 %)]}))))) => [8 :transact/start 2 2 [1 1] :transact/start [4] [2] :transact/start 6 [3]]
v 3.0
(defn ^Callable wrap-min-time
([f total]
(wrap-min-time f total 0))
([f total delay]
(fn []
(let [start (System/currentTimeMillis)
_ (if (and delay (pos? delay)) (Thread/sleep (long delay)))
_ (f)
end (System/currentTimeMillis)
duration (- end start)]
(if (> total duration)
(Thread/sleep (long (- total duration))))))))
link
((wrap-min-time (fn []) 20 0)) => nil ((wrap-min-time (fn []) 100 10)) => nil
4 std.concurrent: A Comprehensive Summary (including submodules)
The std.concurrent module provides a comprehensive suite of utilities for managing concurrency in Clojure applications, building upon Java's java.util.concurrent package. It offers abstractions for thread management, message passing, task execution, and resource pooling, aiming to simplify the development of robust and scalable concurrent systems.
4.1 std.concurrent (Main Namespace)
This namespace acts as an aggregator, re-exporting key functions and macros from its sub-namespaces to provide a unified and convenient interface for concurrent programming.
Key Re-exported Functions:
- From
std.concurrent.thread: Allthread:*functions.n Fromstd.concurrent.queue: Allqueue:*functions,take,put,peek,drain,pop,push,remove.n Fromstd.concurrent.executor: Allexecutor:*andexec:*functions,submit,schedule.n Fromstd.concurrent.atom: Allaq:*andhub:*functions.n Fromstd.concurrent.bus: Allbus:*functions.n Fromstd.concurrent.pool: Allpool:*functions.n Fromstd.concurrent.request: Allreq:*functions,bulk,transact.n Fromstd.concurrent.request-apply:req:applicative.n Fromstd.concurrent.request-command:req:command,req:run.n* Fromstd.concurrent.relay:send,relay:create,relay,relay:bus.
4.2 std.concurrent.thread (Thread Management)
This sub-namespace provides low-level utilities for creating, inspecting, and manipulating java.lang.Thread objects.
Key Functions:
thread:current: Returns the current thread.nthread:id: Returns the ID of a thread.nthread:interrupt: Interrupts a thread.nthread:sleep: Pauses the current thread.nthread:spin: Hints to the processor that the current thread is busy-waiting.nthread:wait-on,thread:notify,thread:notify-all: Standard Java object wait/notify mechanisms.nthread:has-lock?: Checks if the current thread holds the monitor lock on an object.nthread:yield: Hints to the scheduler that the current thread is willing to yield its current use of a processor.nstacktrace,all-stacktraces: Retrieves stack trace information.nthread:all,thread:all-ids: Lists all active threads or their IDs.nthread:dump: Dumps the stack trace of the current thread.nthread:active-count: Returns the number of active threads in the current thread's thread group.nthread:alive?: Checks if a thread is alive.nthread:daemon?: Checks if a thread is a daemon thread.nthread:interrupted?: Checks if a thread has been interrupted.nthread:has-access?: Checks if a thread allows access to current.nthread:start,thread:run: Starts or runs a thread's execution.nthread:join: Waits for a thread to terminate.nthread:uncaught,thread:global-uncaught: Manages uncaught exception handlers.nthread:classloader: Manages the context class loader for a thread.nthread: A constructor function for creating and configuring newThreadinstances.
4.3 std.concurrent.queue (Blocking Queues and Deques)
This sub-namespace provides convenient wrappers and extensions for Java's java.util.concurrent.BlockingQueue and BlockingDeque interfaces, simplifying message passing and task buffering.
Key Functions:
->timeunit: Converts a keyword (e.g.,:ms,:s) to ajava.util.concurrent.TimeUnitenum.ndeque: Creates aLinkedBlockingDeque.nqueue: Creates aLinkedBlockingQueue.nqueue?,deque?: Predicates to check if an object is aBlockingQueueorBlockingDeque.nqueue:fixed: Creates anArrayBlockingQueuewith a fixed capacity.nqueue:limited: Creates ahara.lib.concurrent.LimitedQueue(a custom limited queue).ntake: Retrieves and removes the head of the queue, with optional timeouts.ndrain: Removes all available elements from the queue and adds them to a collection.nput: Inserts an element at the tail of the queue, with optional timeouts.npeek: Retrieves, but does not remove, the head of the queue.nremaining-capacity: Returns the remaining capacity of the queue.npeek-first,peek-last: Retrieves, but does not remove, the first or last element of a deque.nput-first,put-last: Inserts an element at the head or tail of a deque, with optional timeouts.ntake-first,take-last: Retrieves and removes the first or last element of a deque, with optional timeouts.npush: Inserts an element at the front of a deque.npop: Retrieves and removes the first element of a deque.nremove: Removes a specific element from the queue.n*process-bulk: Processes elements from a queue in batches.
4.4 std.concurrent.executor (Executor Services)
This sub-namespace provides functions for creating and managing various types of java.util.concurrent.ExecutorService instances, including single-threaded, pooled, cached, and scheduled executors. It integrates with std.lib.component for lifecycle management and std.lib.component.track for tracking.
Key Functions:
wrap-min-time: Wraps aCallableto ensure its execution takes at least a minimum amount of time, with an optional initial delay.nexec:queue: Creates aBlockingQueuesuitable for use with executors.nexecutor:single: Creates anExecutorServicethat uses a single worker thread.nexecutor:pool: Creates aThreadPoolExecutorwith a fixed or bounded pool size.nexecutor:cached: Creates a cached thread pool that reuses existing threads.nexec:shutdown,exec:shutdown-now: Initiates an orderly or immediate shutdown of an executor.nexec:get-queue: Retrieves the underlyingBlockingQueueof aThreadPoolExecutor.nsubmit: Submits aCallabletask for execution, returning aFuture. Supportsmin,max(timeout), anddelayoptions.nsubmit-notify: Submits a task only if the executor's queue has remaining capacity.nexecutor:scheduled: Creates aScheduledThreadPoolExecutor.nschedule: Schedules a task for one-time execution after a delay.nschedule:fixed-rate: Schedules a task to run at a fixed rate.nschedule:fixed-delay: Schedules a task to run with a fixed delay between executions.nexec:await-termination: Blocks until all tasks have completed execution after a shutdown request.nexec:shutdown?,exec:terminated?,exec:terminating?: Checks the shutdown status of an executor.nexec:current-size,exec:current-active,exec:current-submitted,exec:current-completed: Retrieves statistics about the executor's threads and tasks.nexec:pool-size,exec:pool-max,exec:keep-alive: Manages core pool size, maximum pool size, and thread keep-alive time.nexec:rejected-handler: Gets or sets the handler for tasks that cannot be executed.nexec:at-capacity?,exec:increase-capacity: Checks and adjusts executor capacity.nexecutor:type: Returns the type of executor (e.g.,:single,:pool,:scheduled,:cached).nexecutor:info: Returns detailed information about an executor.nexecutor:props: Returns component properties for an executor.nexecutor:health: Returns the health status of an executor.nexecutor:start,executor:stop,executor:kill,executor:started?,executor:stopped?: Component lifecycle functions.nexecutor(multimethod): A multimethod for creating executors based on a:typekeyword.n*executor:shared,executor:share,executor:unshare: Manages a registry of named, sharedExecutorServiceinstances.
4.5 std.concurrent.atom (Atom Queues and Hubs)
This sub-namespace provides specialized atom-based queueing mechanisms for batch processing and managing asynchronous task submissions.
Key Functions:
aq:new: Creates an atom holding a vector to act as a queue.naq:process: Processes elements from an atom queue in batches, applying a function to each batch.naq:submit: Returns a submission function that adds entries to an atom queue and triggers batch processing on an executor.naq:executor: Creates an executor specifically designed to work with an atom queue, handling batch submissions.nhub:new: Creates a "hub" atom, which is an atom containing a map with aticket(an incomplete future) and aqueue.nhub:process: Processes elements from a hub's queue in batches, fulfilling the hub's ticket when done.nhub:add-entries: Adds entries to a hub's queue.nhub:submit: Returns a submission function for a hub, which adds entries and triggers batch processing on an executor.nhub:executor: Creates an executor that uses a hub for batch processing.n*hub:wait: Waits for a hub's processing to complete.
4.6 std.concurrent.bus (Message Bus)
This sub-namespace implements a message bus system for inter-thread communication, allowing threads to register, send messages, and receive responses. It's built on std.concurrent.queue and std.concurrent.thread.
Key Functions:
bus:get-thread,bus:get-id: Retrieves thread objects or their registered IDs.nbus:has-id?,bus:all-ids,bus:all-threads,bus:get-count: Query bus state.nbus:get-queue: Retrieves the message queue for a specific thread.nbus:register,bus:deregister: Registers or deregisters threads with the bus.nbus:send: Sends a message to a specific thread's queue, returning aFuturefor the response.nbus:wait: Waits for a message to arrive in the current thread's queue.nhandler-thunk,run-handler: Internal functions for managing message handler loops in separate threads.nbus:send-all: Sends a message to all registered threads.nbus:open: Starts a new message handler loop in a separate thread, returning aFuturethat completes when the handler starts.nbus:close,bus:close-all: Sends an:exitmessage to gracefully stop handler loops.nbus:kill,bus:kill-all: Forcefully interrupts handler threads.nmain-thunk,main-loop: Internal functions for the bus's main message processing loop.nstarted?-bus,start-bus,stop-bus,info-bus: Component lifecycle and info functions for the bus.nBus(defimpl record): The concrete record type for the message bus.nbus:create,bus: Creates and starts a message bus instance.nbus?: Checks if an object is aBus.nbus:with-temp(macro): A macro for creating a temporary bus, registering the current thread, and ensuring proper cleanup.n*bus:reset-counters: Resets message sent/received counters.
4.7 std.concurrent.pool (Resource Pooling)
This sub-namespace provides a generic resource pooling mechanism for managing a collection of reusable resources (e.g., database connections, network sockets). It ensures efficient resource utilization and proper lifecycle management.
Key Functions:
resource-info,resource-string: Provides information and string representation for individual pooled resources.npool-resource: Creates aPoolResourcerecord, which tracks the state of a single resource.npool:acquire: Acquires a resource from the pool. If no idle resources are available and the pool is not at its maximum capacity, it attempts to create a new resource.npool:dispose: Disposes of an idle resource.npool:dispose-over: Disposes of resources if the idle/busy count exceeds a limit.npool:release: Releases a resource back to the pool.npool:cleanup: Periodically cleans up idle resources based onkeep-aliveand health checks.npool-handler: The handler function for the pool's cleanup thread.npool:started?,pool:stopped?,pool:start,pool:stop,pool:kill: Component lifecycle functions for the pool.npool:info: Returns detailed information about the pool's state (idle, busy, stats).npool:props: Returns component properties for the pool.npool:health: Returns the health status of the pool.npool:track-path: Returns the tracking path for the pool.nPool(defimpl record): The concrete record type for the resource pool.npool?: Checks if an object is aPool.npool:create,pool: Creates and starts a resource pool instance.npool:resources:thread,pool:resources:busy,pool:resources:idle: Retrieves resources associated with a thread, or all busy/idle resources.npool:dispose:mark,pool:dispose:unmark: Marks a resource for disposal or unmarks it.nwrap-pool-resource: Wraps a function to automatically acquire and release a resource from the pool.n*pool:with-resource(macro): A macro for using a resource from the pool within a lexical scope, ensuring it's released afterwards.
4.8 std.concurrent.print (Asynchronous Printing)
This sub-namespace provides an asynchronous printing mechanism, allowing print operations to be offloaded to a separate executor, preventing blocking of the main application thread.
Key Functions:
print-handler: The handler function for processing print items.nget-executor: Retrieves the print executor.nsubmit: Submits items for asynchronous printing.nprint,println,prn,pprint: Asynchronous versions of standard Clojure print functions.npprint-str: Pretty-prints an item to a string.nwith-system(macro): Bindsenv/*local*tofalseto force system-level printing.nwith-out-str(macro): Captures output from asynchronous print operations.
4.9 std.concurrent.relay (Process/Socket Communication Relay)
This sub-namespace provides a generic relay mechanism for communicating with external processes or network sockets. It uses a message bus (std.concurrent.bus) to manage asynchronous I/O and command sending.
Key Functions:
get-bus: Retrieves the common stream bus used by relays.nwith:bus(macro): Binds the default relay bus.nattach-read-passive: Attaches a passive reader to an input stream, processing messages asynchronously.nattach-interactive: Attaches an interactive reader to an input stream, using the bus for communication.nrelay-stream: Creates aRelayStreamrecord, representing an input or output stream with associated metadata.nrelay-stream?: Checks if an object is aRelayStream.nmake-socket-instance,make-process-instance,make-instance: Creates instances for socket or process communication, setting up input/output streams.nrelay-start,relay-stop: Component lifecycle functions for the relay.nRelay(defimpl record): The concrete record type for the relay.nrelay:create,relay: Creates and starts a relay instance.nsend: Sends commands (e.g.,:raw,:partial,:read-line) to the relay, which are then processed by the external process/socket.
4.10 std.concurrent.relay.transport (Relay Transport Layer)
This sub-namespace provides the low-level implementation for reading from and writing to input/output streams in the std.concurrent.relay module.
Key Functions:
bytes-output: Creates aByteArrayOutputStream.nmock-input-stream: Creates a mockInputStreamfor testing.nread-bytes-limit,read-bytes-line,read-bytes-some: Functions for reading bytes from anInputStreamwith various strategies (limited amount, line by line, until timeout).nop-count,op-clean,op-clean-some: Operations for querying stream status or cleaning its content.nop-read-all-bytes,op-read-all,op-read-some-bytes,op-read-some,op-read-line,op-read-limit: Operations for reading data from the stream in various formats and with limits/timeouts.nprocess-by-line,process-by-handler: Functions for processing stream content line by line or with a custom handler.nprocess-op: Dispatches to the appropriate stream operation based on the command.nsend-write-raw,send-write-flush,send-write-line: Functions for writing raw bytes, flushing, or writing a line with a newline to anOutputStream.nsend-command: Sends a command to the output stream and optionally waits for a response from the input stream.
4.11 std.concurrent.request (Request/Response Abstraction)
This sub-namespace provides a generic request/response abstraction for interacting with various "clients" (e.g., database connections, API endpoints). It supports single requests, bulk requests, and transactional requests, with options for pre/post-processing, chaining, and asynchronous execution.
Core Concepts:
IRequestProtocol: Defines methods forrequest-single,request-bulk,process-single,process-bulk.nIRequestTransactProtocol: Defines methods fortransact-start,transact-end,transact-combine.n Request Contexts: Manages*bulk*and*transact*dynamic variables to track bulk and transactional operations.n* Options (req:opts): A rich set of options for controlling request behavior, includingpre/posthooks,chainfunctions,asyncexecution,measure(timing), anddebug.
Key Functions:
bulk-context,transact-context: Creates contexts for bulk and transactional operations.nreq:opts-clean,req:opts,req:opts-init: Functions for managing and initializing request options.nopts:timer,opts:wrap-measure,measure:debug: Utilities for timing and debugging requests.nreq:return: Handles returning results, potentially dereferencing futures.nreq:single-prep,req:single-complete,req:single: Functions for handling single request execution.nreq:unit: Executes a single request within a bulk context.nbulk:inputs: Captures all inputs submitted within a bulk context.nbulk-collect,bulk-process,bulk: Functions for managing and processing bulk requests.ntransact: Enables transactional execution for a block of requests.ntransact-prep: Prepares request options for transactional contexts.nreq-fn: The core function for dispatching requests based on current context (*bulk*,*transact*).nbulk:transact: Combines bulk and transactional behavior.nreq(macro): The primary macro for submitting requests.nreq:in(macro): Captures all requests submitted within its body.nreq:bulk(macro): Defines a block of requests to be executed in bulk.nreq:transact(macro): Defines a block of requests to be executed transactionally within a bulk context.nbulk:map,transact:map: Apply a function across a collection of inputs in bulk or transactionally.n*fn-request-single,fn-request-bulk,fn-process-bulk,fn-process-single: Default implementations ofIRequestfor Clojure functions.
4.12 std.concurrent.request-apply (Applicative Request)
This sub-namespace integrates the std.concurrent.request framework with the std.lib.apply applicative pattern, allowing requests to be defined and executed as applicatives.
Key Functions:
req-call(multimethod): An extensible multimethod for dispatching request calls based on their:type(e.g.,:single,:bulk,:transact,:retry).nreq-apply-in: Runs a request applicative, handling transformations and options.nReqApplicative(defimpl record): A record that implementsstd.protocol.apply/IApplicablefor requests, allowing them to be invoked like functions.n*req:applicative: Constructs aReqApplicativeinstance.
4.13 std.concurrent.request-command (Command-based Requests)
This sub-namespace provides a framework for defining and running "commands" as a structured way to encapsulate requests. Commands can have predefined functions, options, and formatting rules for inputs and outputs.
Key Functions:
format-input,format-output: Helper functions for formatting command inputs and outputs.nrun-request(multimethod): An extensible multimethod for running commands based on their:type(e.g.,:single,:bulk,:transact,:retry).nreq:run: The main function for executing a command, applying input/output formatting and dispatching torun-request.nCommand(defimpl record): A record that encapsulates a command's definition (type, name, arguments, function, options, format, process).nreq:command: Constructs aCommandinstance.
4.14 Usage Pattern:
The std.concurrent module is essential for building high-performance, responsive, and scalable Clojure applications. It provides:
- Structured Concurrency: Tools for managing threads, executors, and message queues in an organized manner.n Asynchronous Operations: Support for non-blocking I/O and task execution.n Resource Management: Efficient pooling of expensive resources.n Inter-process Communication: A message bus for communication between different parts of an application or external processes.n Request Abstraction: A flexible framework for defining and executing requests against various services, with advanced features like bulk processing and transactions.
By offering these powerful concurrent programming primitives, std.concurrent empowers developers to tackle complex concurrency challenges with greater ease and reliability.