Gradle Team Perspective on Bazel

hansdockterThe Google team that is responsible for the internal Google build system (known as Blaze) has open sourced as alpha the Bazel build system today. Congratulations to them for their release. We got asked a few times what our opinion is on Bazel.

This post is a work in progress but reflects the current thoughts of the Gradle team on this industry development.
Why Did This Happen Now?

The global conversation around Build Automation is red hot.

Enterprises are understanding that the ability to rapidly develop and improve software is fundamental to competing in an environment where increasingly large and complex software is “eating the world”. The key drivers include Agile, SaaS software and APIs, Multichannel (e.g. Android, IOS, Responsive Web, etc), Polyglot Software (e.g. Scala, Go, etc), Containerization (e.g. Docker), Microservices, User Analytics (i.e. Minimum Viable Product methodologies, A|B Testing etc), and mixing of Open and Closed Source Libraries.

All of these emergent capabilities demand systems for coordinating distributed developers at a large scale. With the release of Bazel, Google joins this global conversation about what matters to Enterprises seeking to embrace high velocity build automation.

Why is this good?

The build domain deserves the smartest people working on it and inspiring each other. New voices and solutions in Enterprise build systems can strengthen this global conversation; Google is a hugely talented engineering organization with a unique perspective.

There is a great curiosity in the Fortune 1000 companies about how leading companies in silicon valley manage to build large-scale software applications with great frequency and innovate so rapidly. Innovations from this class of company often contain valuable insights that can be transferred, when appropriate, to other large scale software projects.

What problem is Bazel trying to solve?

Bazel is a subset of the internal Google build system called Blaze. As such, Bazel has evolved to solve a very big problem that is somewhat (but perhaps not entirely) unique to Google:

The situation at Google is very well documented in this research article by J. David Morgenthaler, Misha Gridnev, Raluca Sauciuc, and Sanjay Bhansali of Google:

http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/37755.pdf

This paper states:

“Google’s source code is big (hundreds of millions LOC) and for the most part monolithic. This is both an advantage and a disadvantage for software development. One of the biggest advantages is that it allows a uniform development style to be adopted and enforced across the company. For example, there are written, language-specific style guides that all engineers follow, a single multi-language build system to build code for all projects, a single repository for the source code, a single continuous testing infrastructure that runs all unit tests, and another single repository for the results of every build and test run. Software tool writers have access to and can run analysis on the entire source code and everyone uses the same tools for code reviews and a single index to search the codebase.”

The original internal Google build system has deeply contributed to the way people think about builds. They were real innovators when it came to scalability.

Similar to many enterprises Google has many languages. But unlike most, Google has a single source repository for all code. Everything is released very frequently from head. This means no way of varying integration pressure between teams and decouple things across organizational boundaries like location, cooperation model, … Again, Google is one of the few organizations on the planet where such a model scales. Of course in this context without radical parallelization, you will die. But it is an unusual context.

Because of this huge code base performance is king, other requirements are subservient, especially requirements that are not relevant to Google. Most companies are not Google, and have broader requirement sets.

How does Bazel contribute to this conversation?

Bazel’s creation validates the notion that the best software companies in the world care deeply about the core competency of build automation.

The most telling part of the Bazel FAQ for those seeking to compare Gradle to Bazel is their team’s answer to the question: Why doesn’t Google use Gradle?

From the Bazel FAQ

  • Bazel configuration files are much more structured than Gradle’s, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility.

It’s also important to note here that the Google internal build system has been in development for over 7 years, so the decision to build their own may have come from a more historical comparison.

There are two dimensions here, one about “more structure” and the result enabling increased parallelism and better reproducibility. Both of these end goals are definitely desirable in a build system.

What Bazel is referring to is the low level execution and toolchain model. Bazel captures more information currently regarding the relations between inputs and outputs for tasks/actions. This also enables it to be reliably parallelizable on the lowest level. These are great improvements in the configuration model for providing gains in build performance when executed in a parallel environment.

The Google focus on parallelization may be the largest conversation to the global conversation on build performance. Making such a profound investment into build performance is an act of great industry leadership. Gradle is not as parallelizable as Bazel is today. It parallelizes on the project level, not on the task level. But within the next few months this will be history. With our upcoming new configuration and component model Gradle will be reliably and fundamentally parallelizable to the deepest level, including the configuration actions.

As far as structure, in many ways Gradle provides a strong model informed by more considerations that parallel performance. In these dimensions, Gradle is already more structured than Bazel. Try to create a variant aware Android build with Bazel and you will see this is no structure for this. You are back to the days of Ant. For all these reasons and more Gradle is and will be the default system for Google Android. The reasons why the Android team choose Gradle are the key qualities that distinguish it from tools like Bazel (or Bucks or Pants in that respect). They were created for being very good for a very specific environment and purpose.

With the parallelism required to process a very large monolithic source repository, it’s clear that Google needed to build their own system at the time this decision was taken. But Gradle will be adding this level of parallelism in the near future alongside our other design features and considerations.

Is Bazel developed fully in the open?

According to the Bazel governance page:

“Unfortunately not. We have a significant amount of code that is not open source; in terms of rules, only ~10% of the rules are open source at this point. We did an experiment where we marked all changes that crossed the internal and external code bases over the course of a few weeks, only to discover that a lot of our changes still cross both code bases.

This complicates the development process and means that some changes will simply appear in the Bazel repository without clear explanation. At the same time, it is difficult to collaborate or get actual users without actually having the code in the open. Therefore, we are opening up the code, even though some of the development is still happening internal to Google.”

It is clear from this statement that there is a boundary between the open and closed components which may produce complexity for implementers.

How does Gradle participate in this conversation?

Gradle recognizes strongly the need for standard builds across the Enterprise in order to normalize a frequently highly heterogeneous set of projects, languages, platforms, delivery channels and software development styles. We also recognize that the extraordinary techniques and technologies that come from the leading edge of software can contribute greatly to the rest of the world. Gradle has seen this deep commitment and leadership from our own customers including Linkedin, Netflix, Google Android and many others.

Alongside the Google Blaze team, we recognize the role of a single multi-language build system to provide a convergent solution for unifying the Enterprise build pipeline. However, we are keenly aware that most enterprises do not take as monolithic of an approach as Google, and that most enterprises don’t have the same kind of dedicated build services team and development culture that Google has. Serving a wide variety of Enterprise customers have created the design requirements that have informed the development of Gradle.

Here are some of the observations that can be made by studying the Bazel Alpha.

  • Bazel does not have a high level declarative build language that makes the build easy to use for developers. At Google this can be compensated with a specialized service team that owns the build tool.
  • Not built for extensibility*. Since Bazel was built for only one customer, If the Google engineers need deep extensions, they have a build team that can bake this into the tooling they own.

* Note 3/25/15: we received a gracious note through our web site from the Bazel team that they have been working on extensibility and they cited this work of almost a year to invite extension http://bazel.io/docs/skylark/index.html

  • Speed is optimised around the idea that all source transitive dependencies are stored in one big repo; all libraries and tools are checked in to this central repository. Most Enterprises have more distributed dependency management requirements.
  • Bazel is *nix only, it does not run on Windows. This eliminates a large number of potential Enterprises including a large bank which recently identified itself to us as “51% .Net”.
  • No plugin ecosystem.

The goal of Gradle has always been to find the ultimate path between performance, extensibility, richness in modelling the continuous integration process and usability.

Another significant part of this conversation is about extensibility and integration across a variety of tools in the build toolchain. For starters, Gradle has deep integration into popular IDE systems including Eclipse, IntelliJ and Android Studio. Beyond these deep IDE integrations, the Gradle ecosystem is massive. A plugin portal with more than 300+ community plugins that connect to every imaginable tool, platform and system in the Continuous Delivery market plus many more in the wild, 400K downloads per month, a global and elite group of committers and a mighty alliance of elite software organizations that use it strategically.

Where does the conversation go?

The global conversation about build automation continues to ignite. Developers everywhere are realizing the necessity to move from diverging, disparate and unmaintainable build systems towards a unifying paradigm for continuous software delivery. Leading software companies in Silicon Valley and beyond are participating in upping the industry’s game around the continuous delivery of software. Gradle is excited to contribute to this global conversation and partner with world-class Enterprises to enable them to embrace the core competency of shipping bigger and more complex software more dynamically and iteratively.

Gradle already has many satisfying answers that unifies build systems across hundreds of leading software companies. One of the core areas where we seek to improve over the next few months is in the rearchitecture of our configuration system that will bring Blaze-like parallelization to organizations who want to benefit from the large ecosystem and diverse feature set of Gradle.

We welcome scrutiny of our upcoming component model and distributed build capabilities that should enable Gradle to bring Google-class distributed parallelism to our customers and the rest of the industry through open source.