outfn

A macro for clearer, more declarative functions (that are actually macros) through an emphasis on names.

If you’ve thought to yourself any of the following, read on:

• What is this function and what do the arguments mean?
• I want to name these functions the same thing, because they do the same thing!
• If only this function’s arguments were in a different order, I could use a threading macro.
• I wish my code failed faster when I have a typo or refactoring error with static validation.
• I want validated code, but without writing validation code.
• We need one single place in the code that a new person could read and understand all the data that passes through.
• Why did you name these things the exact same thing??? That’s not what it means.
• This let block is so ugly and imperative, but I can’t break it down.
• I need to pass $DATA all the way to $DEEP_INNER_FUNCTION? This is going to be ugly.
• I know, a :dynamic var is the perfect solution to this!
• I already have all the functionality to do this, I just need to glue it together.
• We have too much code, I want less!

Status

This is all very alpha, and I’m still testing how well this works in practice. Feedback would be very much appreciated though. (A better name would be cool too, since the outfn macro generates a macro to get the validation before runtime.)

Features

vanilla/base outfn

• keyword arguments for clarity of function arguments

  (do-something :user-map x :password-list y)
      

• function dispatch by argument instead of arity

  (defoutfn outfn
    "what's up doc"
    ([x] (clojure.string/join (repeat x "x")))
    ([y] (clojure.string/join (repeat y "y"))))
  
  (outfn :x 2) ;; "xx"
  (outfn :y 3) ;; "yyy"
      

• decomplecting order from function arguments

  (defoutfn map-v2.0
    "Like map, but better"
    [f coll]
    (map f coll))
  
  ;; threading in a collection
  (->> 5
       range
       (map-v2.0 :f inc :coll)) ;; (1 2 3 4 5)
  
  ;; threading in a function
  (->> 5
       (partial +)
       (map-v2.0 :coll (range 5) :f)) ;; (5 6 7 8 9)
      

• 0-overhead function dispatch based on keys
  • No intermediate maps are made.
  • All the work is done at macroexpand-time.
  • Positional argument function calls are generated.
• macroexpand-time validation that a function is called properly - no waiting for (un)lucky input at runtime needed! hello easier refactoring and goodbye typos at run/test time

  (defoutfn outfn
    "what's up doc"
    [foo]
    :foo)
  
  (when nil
    ;; this doesn't get run
    (outfn :fo 2))
  ;; Error!  Invalid set of keys #{:fo} for #'outfn.core/outfn
      

Glossary

Naming, meaning, and validation shouldn’t be separated. Doing so hurts readability and encourages subtle bugs (confusing a variable for a different one because the name is the same) and boilerplace (validating the same name the same way in different places). Let’s avoid using the same name to refer to different things and enjoy simpler programming.

• DRY, non-intrusive validation

  ;; all the validation in one place!
  (def a-glossary {:foo {:validator #(= :foo %)}
                   :choo {:schema {:a {:b s/Int}}}})
  
  (defoutfn outfn1 {:glossary a-glossary}
    "Some outfn"
    ([foo] 3.5)
    ([choo] :lochness))
  
  (defoutfn outfn2 {:glossary a-glossary}
    "Some outfn"
    ([foo] :hello)
    ([choo] :world))
  
  (outfn1 :foo :foo) ;; => 3.5
  (outfn1 :foo 42) ;; => Error!
  (outfn1 :choo {:a {:b 3}}) ;; => :lochness
  (outfn1 :choo {:a {:b "3"}}) ;; => Error!
  (outfn2 :foo :foo) ;; => :hello
  (outfn2 :foo :bar) ;; => Error!
  (outfn2 :choo {:a {:b 3}}) ;; => :world
  (outfn2 :choo {:a {:b "3"}}) ;; => Error!
      

• encourages consistent naming which encourages thinking more about names (due to the centralized validation based on naming)

Implicits

By creating a glossary or “concept map”, we create a central place for understanding the types of data passing through a system, their meaining, and their names. Forcing you to think about and create this then allows more declarative programming, by implicitly calling these functions leading to simpler, less coupled logic.

• declarative programming

  (defoutfn foo-fn {:output :foo}
    "Docstring"
    ([a] 3)
    ([b] 4)
    ([c d] 5))
  
  (defoutfn bar-fn {:implicits #{#'foo-fn}}
    "what's up doc"
    [foo] foo)
  
  ;; Look ma, no glue code
  (bar-fn :foo 2) ;; 2
  (bar-fn :a nil) ;; 3
  (bar-fn :b 42) ;; 4
  (bar-fn :c 11 :d 22) ;; 5
  
  ;; I don't need to know what bar-fn needs - decoupling
  (bar-fn :foo 2 :a 3 :b 4 :c 5 :d 6 :q 72) ;; 2
  
  ;; Laziness is a virtue
  (bar-fn :foo 42 :a (throw Exception.)) ;; 42
      

• macroexpand-time validation of implicits

  (when nil
    ;; No errors here:
    (bar-fn :c 11 :d 22)
    ;; I don't run, but still error out
    (bar-fn :q 11)) ;; ERROR
      

• solves the “big let” problem by breaking the problem down into domain-level concepts

  ;; imperative code that is hard to break down
  (def imperative-ans (let [a 42
                            b (+ a 16)
                            c (* a b)]
                        (+ a b c)))
  
  ;; no imperative code here:
  (defoutfn a {:output :a} "Returns an a" [] 42)
  (defoutfn b {:output :b} "Returns a b" [a] (+ a 16))
  (defoutfn c {:output :c} "Returns a c" [a b] (* a b))
  (defoutfn result {:implicits #{#'a #'b #'c}}
    "Returns the answer"
    [a b c]
    (+ a b c))
  
  (assert (= (result)
             imperative-ans))
      

• solves the problem of dependency passing (normal solutions involve explicitly declaring dependencies of all your dependencies or passing an opaque options map or using dynamic vars; parameters are coupled either way)
  • eg. x calls y
    • former:
      • the arguments of x contain the arguments of y
      • when changing the logic of y, also have to change x
    • later:
      • the arguments of x contains some map containing the arguments of y
      • in order to understand how x is called, you need to understand y
  • Let’s have an example: Start with some functions that are all nice and understandable. Note that they can be understood in their entirety based on their contents.

    (defn gimme-data
      "Returns raw data"
      [db some-options]
      ;; do some stuff with all these inputs...
      :data)
    
    (defn filter-data
      "Returns filtered data"
      [data]
      (let [top-secret-filter identity]
        (top-secret-filter data)))
            

  • what if we want an api call to get filtered data? how do we call them together?
    • Option A: have a different function that wraps the two functions together. We have a problem though: What’s some-options? What kind of db? In order to know how to call this function, I need to know how to call the functions it calls.

      (defn gimme-filtered-data
        [db some-options]
        (filter-data (gimme-data db some-options)))
      
      (defn api-call-coupled1
        []
        (let [db :just-kidding
              some-options {:something :something}]
          (gimme-filtered-data db some-options)))
                  

    • Option B: Call them at the top level and connect them there. Problem: Do I need to have all my db calls in my top-level calls just to have decoupled code?

      (defn api-call-coupled2
        []
        (let [db :just-kidding
              some-options {:something :something}]
          (filter-data (gimme-filtered-data db some-options))))
                  

    • Option C: Use implicits and not have to write any glue code! Note that the insides of each function can be understood in their entirety based on their content, except for the one call to filtered-data-outfn, and you can look up in a glossary to see what it returns.

      (defoutfn data-outfn {:output :data}
        "Returns raw data"
        [db some-options]
        ;; do some stuff with all these inputs...
        :data)
      
      (def db-calls #{#'data-outfn #'some-other-db-call #'put-all-db-calls-together})
      
      (defoutfn filtered-data-outfn {:implicits db-calls}
        "Returns filtered data"
        [data]
        (let [top-secret-filter identity]
          (top-secret-filter data)))
      
      (defn api-call-with-implicits
        []
        (let [db :just-kidding
              some-options {:something :something}]
          (filtered-data-outfn :db db :some-options some-options)))
                  

• free error handling in glue code, which can lead to easier debugging

  (defoutfn a {:output :a} "Returns an a" [] 42)
  (defoutfn b {:output :b} "Returns a b" [a] (assert false) (+ a 16))
  (defoutfn c {:output :c :implicits #{#'a #'b}} "Returns a c" [a b] (* a b))
  
  (c)
  ;; clojure.lang.ExceptionInfo: throw+:
  ;; {:thrown #<AssertionError>
  ;;  :error-time :runtime
  ;;  :outfn-var #'c
  ;;  :computation-order [[:a #{}] [:b #{:a}] [:c #{:b :a}]]
  ;;  :intermediate-var #'b
  ;;  :computation-step [:b #{:a}]
  ;;  :input-keys #{:a}}
      

Future Benefits of Implicits

• optimizations
  • batching (eg. group together DB calls)
  • parallelization (evaluating non-dependent function calls in parallel)
  • determining the lowest cost route from the input to the desired data (assigning cost to each function call)
• auto-testing : by having multiple ways to solve for the same thing, a test suite can be automatically run to make sure each way returns the same result
  • eg. if there are multiple way to compute the same value, they can all be computed and compared to each other for validation

Usage

The library is available on clojars, just add [outfn "0.1.0"] to your project.clj.

Declaring an outfn is of the form:

(defoutfn $NAME $PARAMS
  $DOCSTRING
  ([$INPUTS...]
     $LOGIC)
  ([$INPUTS...]
     $LOGIC)
  ...)

or

(defoutfn $NAME $PARAMS
  $DOCSTRING
  [$INPUTS...]
  $LOGIC)

where:

$NAME

Standard function name. A macro is created with the same name.

$PARAMS

An optional map of extra data for the outfn, with the following optional keys:

:output

A keyword specifying the output of the outfn. It is used for both validation with the glossary and knowing the output for an implicit call.

:implicits

A set of vars of the outfn’s that can be implicitly chained together in order to create the necessary input for the current outfn.

:glossary

A function that takes in a keyword and returns a map with the following optional keys:

:validator

a validation function that the concept corresponding to the keyword must follow

:schema

a Prismatic/schema for the concept to follow

$DOCSTRING

A non-optional standard docstring.

$INPUTS

a series of symbols in a vector. Using & and destructuring is not permitted.

$LOGIC

Standard function implementation.

Things to note

• When calling outfn’s, the keywords must be there at macroexpand time to allow for the validation. This means no (apply some-outfn [:foo foo]).
• When multiple matches are possible, but not an exact match, the first declared match is chosen:

  (defoutfn outfn1
    "Docstring"
    ([foo] {:foo foo})
    ([bar] {:bar bar})
    ([foo bar] {:foobar (+ foo bar)}))
  
  (outfn1 :foo 3 :bar 2) ;; {:foobar 5}
  (outfn1 :foo 3) ;; {:foo 3}
  (outfn1 :bar 2 :foo 3 :choo 18) ;; {:foo 3}
      

Development

To run tests, perform lein midje or lein midje :autotest for the command line.

TODOs

• separate implicits from outfn definition
• consider outputs in the map for the function being called implicitly
  • pros:
    • params look better, and multiple outfn’s can use the exact same params
  • cons:
    • can’t do output valdiation
    • output seems to belong to the same params as the input (as in it makes sense)
• consider special syntax for calls with implicits, to make the intention clear
• implicits improvements
  • function to generate visualize call graph (w/ loom)
  • prismatic graph for parallel evaluation
• consider default glossary value to be a validator if it isn’t a map
  • pros:
    • cleaner
  • cons:
    • more “magic”
• should metadata be used instead of an atom? seems idiomatic/normal and using an atom might have issues with versioning/compiled jars
• figure out how to add cost
  • should each subfunction have its own cost
    • add metadata to each subfunction
      • eg. cost to have implicits find the lowest cost route to get an answer
  • should the toplevel outfn have a cost
• clojurescript compatibility
  • possibilities
    • somehow get the macro version working
    • non-macro version
• consider implicit transitivity (can call implicits of functions that can be called implicitly)
  • seems like it could be a bad idea because it can lead to really big graphs (making finding the best path slow) and difficulty to understand what is going on
• allow destructuring in outfn calls if :as is given
• better implicits algorithm
  • problem: how to find the optimal set of calls
    • there is an exponential number of states relative to the number of concepts
      • each concept is either present or not
    • the caching problem: local suboptimal routes may be part of a global optimal route because the intermediates may be shared
  • resources
    • maven optaplanner
    • optaplanner github
    • optaplanner docs
• refactoring tools
  • detecting/eliminating unused arguments
  • renaming arguments throughout the whole codebase
• internal quality
  • use something like defnk to make validation DRY
  • go through all the code TODOs for low hanging fruit, add the rest here
  • docstrings
  • common-data-map validation

Probably Won’t-s

• make both docstring and params optional
  • because there can be ambiguity between the params and the functions
• including the intermeidate value in the error handling of implicits
  • example
    • a is implicitly computed
    • then b is computed with a
    • but an exception occurs in b
    • logging the results of a might be helpful in debugging b
  • this doesn’t seem to be able to be done in general because if a contains a lazy value that causes an error or is an infinite seq, then the logging with break with absolutely no information