Netflix Builds a Pipeline for Polyglot Programming

Once Netflix was a Java shop, and its focus on one programming language allowed the company to streamline its pipeline so developers to speed code to production as quickly as possible. But programmers at the video streaming service have been increasingly using languages other than Java — JavaScript (Node.js), Python and Ruby-on-Rails in particular.

So the tools team for the company had to work to build out a pipeline that could accommodate these languages, and others as well, noted Mike McGarr, Netflix developer productivity manager, speaking at the O’Reilly Velocity systems engineering conference in New York last week.

The result is a new workflow developer tool called NEWT (Netflix Workflow Toolkit), a programming language-independent application that provides a way for developers to specify their own chosen Build tools to package their applications according to the official Netflix specifications.

“I can tell NEWT to run pretty much anything I want and NEWT doesn’t care. It is unopinionated about builds. It is un-opinionated about dependency management for your application,” McGarr said.

When it started as a company that shipped DVDs by mail, Netflix was a Java shop running on Oracle software in a private data center. As the company moved to streaming, however, it experienced some “challenges of scale” that this monolithic Java-based setup was ill-equipped to handle. “We started to reassess what our architecture should look like,” he said, noting that this re-engineering had led the company, like many others, down the path of cloud computing, microservices and NoSQL.

This path proved to be problematic in that most of the developer tools in its pipeline were built around the Java stack. “Building” the apps that the developers created is overseen by the Java-centered Gradle. They were “baked” in a virtual machine where it can be shipped to the cloud as an “immutable artifact” (meaning it is not tampered with once deployed), using Gradle plugin developed by Netflix called Nebula. The company developed its own continuous integration and deployment software, called Spinnaker, to automate the deployments.

The deployment side of operations is fairly un-opinionated — it doesn’t need to know programming languages were used. The build side of operations was pretty wired into Java, however. But what if a dev would want to build a Node app? At first, the tools team adjusted Gradle for these use cases, asking developers to set the configuration to use Apache instead of Tomcat, the default Java application server.

This approach, however, left JavaScript developers as second-class citizens within Netflix, McGarr explained. They didn’t get the full range of support Java developers did, and the tools team found themselves making many minor modifications to support non-Java apps.

The tipping point came with a new initiative to re-architect a major Netflix component, using Node.js. “We couldn’t ignore this growing trend of non-Java,” McGarr said. “At Netflix, we don’t generally mandate tools. Tools have to win out,” among developers, McGarr said.

In short, the company needed a way to package non-Java apps in its preferred .deb packaging format. Though finding the non-Java Build tool to package up all the dependencies, however, proved to be problematic. So many choices! Webpack, Brunch, Browserifly, Grunt, Gulp, npm’s Bower, and Broccoli.js, for starters.

In his research, McGarr found, that in many cases, development shops were using more than one of these tools, with each executing a piece of the dependency management job. “As a Java developer, this was a little bit of a shock to me. I didn’t see this coming,” he confessed. He foresaw that the JavaScript community would probably continue to crank out new build tools at this rapid pace, making it all but impossible for an enterprise to settle on just one to the satisfaction of its developers.

Instead of worrying about providing a specific build tool, McGarr’s team then began thinking about developer workflow tools instead. “We wanted to build something that would enable engineers to do their job, but having opinions only in places that mattered the most,” McGarr said. “We wanted something that felt native or native-like” to the developer, McGarr said. In other words, no more shoehorning Gradle into a JavaScript job.

So the team started building NEWT, a command line tool, written in the Go programming language. Go dependencies have no binaries, so they freed developers to work on any machine they preferred.

The NEWT package command checks to see what dependencies an application needs, and installs them, along with the application itself, in a Docker container. This ensures that each developer gets the correct support tools and system dependencies for each application. The team developed profiles for different applications, called “app types,” that specifies the required configuration and conventions in YAML files for Node.js apps, Python apps, or other types as well. Developers can specify in the YAML file which build tools they want to use to build out their apps.

Why build a developer workflow tool in-house when other options, such as Chef’s Habitat are just arriving on the marketplace? Netflix does try to buy rather than build anywhere possible, McGarr said, after the presentation. But other workflow tools were, for Netflix’s purposes, were opinionated with the wrong opinions. In this case, it was better to build from scratch.

Currently, NEWT is not open source, though it may be in the future. The project is only managed by a handful of developers. As more are hired, McGarr will have the sufficient manpower to prepare the software for external release, he said.