How Json.Decode.Pipeline chaining works

2018-07-10

•

Json.Decode.Pipeline is a popular package for building JSON decoders. It allows you to build decoders using the forward function application operator (|>) to chain field decoders in a convenient, DSL-like way. For example, we could have a decoder like this:

type alias Node =
    { nodeType : String
    , relationName : String
    , plans : Plans
    , schema : String
    , startupCost : Float
    , totalCost : Float
    }


decodeNode : Decode.Decoder Node
decodeNode =
    decode Node
        |> required "Node Type" Decode.string
        |> optional "Relation Name" Decode.string ""
        |> optional "Plans" (Decode.lazy (\_ -> decodePlans)) (Plans [])
        |> optional "Schema" Decode.string ""
        |> required "Startup Cost" Decode.float
        |> required "Total Cost" Decode.float

If you’re coming from an OOP background, at first glance it looks like that pattern where you return this from the object’s methods to allow chaining. One example of this is jQuery call chains:

$('button')
    .removeClass('off')
    .addClass('on')
    .html('Submit')

But when we get past the superficial similarity, there’s obviously something else going on because there are no objects in Elm! So how does this actually work? Clearly, each of those required and optional calls should be returning a different type, so how is it that they can be chained together? What exactly are we chaining here?

To figure this out, let’s look at some function signatures and definitions. The |> operator is forward function application, so x |> f is equivalent to f x. It has left associativity which means that this expression

decode Node
    |> required "Node Type" Decode.string
    |> optional "Relation Name" Decode.string ""

is equivalent to:

optional "Relation Name" Decode.string "" (required "Node Type" Decode.string (decode Node))

In other words, you have to walk the chain back-to-front to rewrite the expression using parentheses.

required and optional have these signatures:

required : String -> Decoder a -> Decoder (a -> b) -> Decoder b
optional : String -> Decoder a -> a -> Decoder (a -> b) -> Decoder b

decode is an alias for Json.Decode.succeed which returns a decoder that always produces a particular value (in our case, Node).

succeed : a -> Decoder a

At first, this can make things more confusing: decode is a function of one argument, and we’ve passed it Node, hence decode Node is a value (Decode Node), but the third argument to required should be a Decoder with a function (Decoder (a -> b)). What’s going on?

Here, we have to remember that type alias names are used in two ways: as the type alias name and as a constructor. The Node constructor is in fact a function of two arguments which returns a record (I’m referring to the actual record returned by the Node constructor as NodeR to avoid confusion):

Node : String -> String -> NodeR

Like any function, it can be partially applied, so we can also look at it as a function of one argument returning another function:

Node : String -> (String -> NodeR)

For example, Node "Result" returns a function of type String -> NodeR. You’ll see why this is useful in a second.

So let’s look at the types of arguments and return values (separated from the arguments with a blue arrow) in this chain of functions starting from the innermost function application, decode Node:

Types of arguments in the chain

As you can see, every application of required or optional “consumes” one more argument of the Node constructor so that at the end, we end up with a Decoder NodeR which is the decoder we need.

It works exactly the same for a larger number of fields because then the Node constructor correspondingly takes more arguments: for the full Node record defined at the start of the post, it’s String -> String -> Plans -> String -> Float -> Float -> Node, matching the six field definitions in decodeNode.

This also explains why the order of the field definitions made with required/optional has to match the order of the fields in the resulting record.

If you would like to learn more about working with JSON, check out my book, Practical Elm. In addition to JSON coverage, I’ve just added a new section to the book, expanding it to 122 pages (there’s still more to come). The new section is huge and it deals with commands, ports, subscriptions, flags, server requests, as well as structuring code as it grows.

Technical note

If you like going down rabbit holes, you may be interested in another explanation:

decode (aka succeed) and required/optional fulfil the requirements for Decoder to be an applicative functor, allowing us to perform this sort of sequential application of functions shown in the diagram above.

The requirements for an applicative functor consist of the presence of the following two functions which interact in a way that follows a number of rules:

pure : a -> t a
apply : t (a -> b) -> t a -> t b

(I won’t go into the rules here - you can see further reading links for that.)

There is an alternative version of apply called <**> (which apparently works just as well):

<**> : t a -> t (a -> b) -> t b

Now, if we look at decode and required signatures again:

decode : a -> Decoder a 
required : String -> Decoder a -> Decoder (a -> b) -> Decoder b

then we can see (after disregarding the first String argument to required) that these have the same types as pure and <**> if we set t = Decoder.