Pike - Asset Pipeline and Make Tool

04 June 2014


I was building a website recently and I decided to use coffeescript and less, just to try them out. Since I am deeply in love with Pyramid, I was using Python on the backend. Turns out, webassets is the only decent choice for compiling web assets in python. Without going into too much detail, I found it to be acceptible for extremely simple use cases, and a huge pain for anything more complex.

Initially, I wrote Pike in Python. I wanted the asset pipeline to be an arbitrary graph instead of an enforced linear pipeline. That not only allows for users to customize the behavior more, it handles source maps in a very reasonable way. I also opted to use Python for the configuration, rather than YAML or a DSL. This makes it more flexible for unanticipated use cases. The end result was the ability to produce a graph that looks exactly what you imagine when you picture your asset pipeline.

Below is a graph that watches coffeescript files for changes, compiles them, writes the js file paths to a json file, and writes the js, source maps, and original coffeescript to a directory.

I say pike was initially written in Python, because some time afterwards I decided that I wanted to learn Go. This project was a great candidate for porting because it's relatively small and the structure of the graph makes a lot of sense with goroutines. If each node is a goroutine, then every operation runs concurrently with all others and the pipelining should be close to optimal. Let's take a look at the final result.

Here is the Go code that I used to generate the image above.

package main

import "github.com/stevearc/pike"

func main() {
    n := pike.Glob("app/src", "*.coffee")
    // Watch for changes
    n = n.Pipe(pike.ChangeFilter())
    // Compile coffeescript in parallel across all cores
    coffee := n.Fork(pike.Coffee(), 0, 3)
    // Write file paths to a json file
    n = coffee.Pipe(pike.Json("app.js"))

    terminus := n.Pipe(pike.FanIn())
    // send the source maps to the terminus
    coffee.Pipe(terminus)
    // send the original coffeescript to the terminus
    coffee.Pipe(terminus)
    // Write the js, source maps, and coffeescript to a directory
    n = terminus.Xargs(pike.Write("build"), 0)

    g := pike.NewGraph("app.js")
    g.Add(n)
    g.Render("/home/stevearc/stevearc.github.io/assets/pike_graph.png")
}

Overall, I'm pretty happy with the port to Go. The code for generating assets isn't as neat and concise as the Python version, but the fact that it's compiled means that you're more likely to catch errors in a reasonable way. It also runs significantly faster, which is nice.

Having now looked around outside of the Python spectrum, it seems like Grunt and Gulp are the most popular tools available. Pike shares many similarities with Gulp (code over configuration, parallelized pipeline). It looks good enough to where if I could go back in time, I would probably just use Gulp and call it a day. But since I've already put in the time to make Pike, I'll be using that for the foreseeable future.

As always, code is on github. If you want additional features or support for more tools, file me an issue!

Code: github.com/stevearc/pike

comments powered by Disqus
© 2014 Steven Arcangeli - Hosted by Github Pages