Revolutionary new Gradle Features on the 2014 Roadmap
|Persistable Build Configuration||Started|
|Variant Aware Dependency Management||Started|
|Better Discoverability & Error Reporting||Started|
|Improved C/C++ support||Started|
|Improved Tooling & IDE support||Not Started|
|Parallel Task Execution||Not Started|
|Watcher Mode for Daemon||Not Started|
|Global Cache||Not Started|
We are convinced that Gradle 2.0 is the best available enterprise build system. Yet we are far from done. Finally, we have the R&D bandwidth to make deep improvements to Gradle in some areas where it is still lacking. We also have the bandwidth to make some fundamental innovations in the domain of build and continuous delivery. These things will bring Gradle much closer to our vision of the ultimate build system. With the improvements noted below Gradle will revolutionize build system performance and extensibility, multi-platform dependency management, IDE integration and native builds. Work on this has commenced and significant portions will be finished by the end of the year. As per usual, all changes are happening on the master branch and shipped incrementally via our regular Gradle releases, every 6-8 weeks.
There is no single metric that can describe the performance of a build system. Build performance has many facets: small or large project, what command you are executing, is it a clean or incremental build, etc .... All aspects of build performance are of utmost importance to us. Since the Gradle 1.0 release we have tremendously improved the performance of Gradle in almost all relevant areas. Gradle shines in many aspects of build performance. Gradle has for example introduced incremental builds into the Java world. But there is enormous potential for further improvements.
Improving the performance will be a major focus for the next 6 months. What is good will become even better and where there are bottlenecks they will be removed. We are talking about revolutionary performance improvements that will make Gradle an extremely fast build system for all types of projects.Result of that will be lightning fast Android Studio Gradle integration and dramatic speed improvements for the largest enterprise builds in the industry like the Gradle build at LinkedIn with over 4000 subprojects. Our roadmap for performance improvements continues into 2015 with plans for fully distributed builds.
Here is what we are going to do in the area of performance:
Much faster configuration time
Right now there is a trade off between the richness and extendability of the Gradle build language and the time it takes to configure the build. The way Gradle configures a build has not changed since its inception. Every time you execute a Gradle build everything is configured, regardless of what command you are finally going to execute. During this configuration phase a build model is constructed, but it is not persisted. This means the configuration has to be re-run every time a Gradle build is executed. Obviously this is wasteful and for larger projects this can mean a significant delay before the actual build execution starts.
This configuration delay is also painful when it comes to IDE integration. Android Studio demonstrates the enormous potential of a deep integration between the IDE and Gradle. At the moment, the price we pay for this is often mediocre responsiveness: every time the IDE makes a query of Gradle, the configuration phase is run before an answer can be given. The folks at Google are implementing some smart workarounds for this. But what is ultimately needed is a fundamental solution within Gradle that eliminates the overhead in the IDE-to-Gradle communication.
Gradle already has a mechanism to alleviate the configuration time problem somewhat for multi-module builds: configuration-on-demand. And the Gradle daemon also caches certain aspects of the model. But this is not a complete solution to the problem.
Currently, the configuration phase runs a bunch of scripts against the Gradle Model objects in order to set up the executional model. What we are working on right now is removing the concept of a distinct configuration phase altogether, resulting in a single execution phase. Configuration will be modelled as tasks which have inputs and outputs like normal Gradle tasks. These “tasks” may not create a jar but instead create a classpath or a library definition. In practice, this will mean:
- Only configuration logic that is required for the current build command will be executed
- All configuration outputs will be persisted. So when build description doesn’t change, the configuration model can be read from cache. Only when one of the configuration inputs change will the outputs need to be rebuilt. See also the section Caching and Sharing Everything. .
- In cases where configuration does need to be rebuilt, this building will be both incremental and parallelizable. See also below the section Fundamentally Parallelizable.
These changes will dramatically reduce or in many cases even eliminate the configuration time for Gradle builds. Expect to see the first results in the coming months.
The new model for build configuration is not driven simply by performance requirements. It is more intuitive and logically simpler than the current model. We expect many additional benefits to follow from this innovation, some of which are discussed below.
Gradle offers already parallel builds. Many of our users make use of this feature on a daily basis. But there are limitations:
- Granularity of parallelization is per project.
- Users must ensure that projects are properly decoupled, allowing them to be built independently.
- The configuration phase for projects is not run in parallel.
The improvements that will make the configuration time much faster will also pave the way for making Gradle fundamentally parallelizable. By providing Gradle with deep knowledge about the interactions between different elements of the build, this will enable Gradle to easily and reliably parallelize on the task level without requiring the build author to care about decoupling. This means that parallel Gradle builds will be easier to maintain and the parallelization will be more fine grained. This will provide in general a much better CPU utilization. In a complex build domain like Android (or C++) where you build many variants of a component within one project, task level parallelism will significantly speed up building even for smaller projects. Parallel execution will be eventually switched on by default. In 2015 we will work on making parallelization even more fine-grained than the task level.
Caching and Sharing Everything
Gradle already does a great job when it comes to caching build outputs and building incrementally. We want to push this two steps further. Currently build outputs are cached locally per build, so the first step for us is to have a single cache for the build outputs for all builds on a machine, similar to how we cache external dependencies. This will mean that outputs can be reused across different builds, which will be particularly helpful on CI machines. As a next step this cache will become distributed, allowing any build output to be shared across machines. For large enterprise builds this will mean further significant performance improvements. Cached outputs will include not only traditional task outputs, but also the outputs of the new configuration tasks.
The base concept for such a cache is that every task is a deterministic function from input to output, with the input including all relevant parameters. In the case of compilation for example, the input is not only the source code but also the version and the type of compiler, and the OS being used. When all of those parameters are captured, the cache output can be reliably shared across machines. Having a single task cache is something we plan to commence this year, continuing work into 2015. The internal Google Blaze build system introduced the concept of such a cache years ago. We are glad that Gradle will be able to offer it’s own version of this very powerful concept.
Gradle has already the most powerful dependency management in the industry. We are now taking this one step further:
Work has started on making the concept of variants a first class citizen for Gradle dependency management. For the native world and for Android, this support is a core requirement permitting true dependency management. For any other domain it will be a very valuable addition.
Besides variants, there are other aspects we are planning to improve in the area of dependency management. We want to make arbitrary custom metadata a first class citizen in dependency resolution. We also want to provide better solutions for many use cases that currently rely on snapshot versions. This work is not yet scheduled, however the Gradle community has started to tackle some of them already. See for example the Netflix Nebula Gradle Dependency Locker plugin and Publishing Plugin.
One of the original contributions of Gradle to the build domain is its declarative and yet flexible build description. A build description that also comes with a sane and rich extendability model. We don't know any other build system that provides something similar. This model is for example a major reason that Google choose Gradle as the new official build system for Android. We want to take this quality of Gradle to the next level as part of our work to improve the way Gradle builds are configured.
Over time, the Gradle plugin ecosystem is become broader and deeper. More and more we see rich transitive relationships between plugins, with one plugin extending another. The Google Android plugin provides the base build framework for Android. Many other Gradle plugins build on top of the Android plugin to extend the Android build ecosystem (e.g. plugins from various Android cloud testing providers). Next, a particular organization may need to further extend and customise the behaviour of those plugins. Although Gradle makes such scenarios possible by providing many hooks, we have reached the point where this solution does not have the elegance and expressiveness we would expect.
The new configuration model will make it easier to deal properly with complex configuration time ordering issues. But we won't stop there. Gradle domain objects will be first class citizens within the new configuration model. For example a SourceSet will know exactly which configuration tasks will affect its state. You will be able to hook in custom configuration tasks directly with the domain object, for example enforcing that your custom configuration of the SourceSet should happen after all other configuration of the SourceSet object has been taking place.
Better Discoverability & Error Reporting
As already mentioned, with the new configuration model connections between the elements of a Gradle build are much stronger modelled. Currently quite a few relations within Gradle builds are implicit. For example between Sources and Jars. Or between configuration blocks and the actual task that gets configured. Our DSL suggests that they are connected, but in the underlying model this connection can not be made. So the new model is very important to provide much better discoverability to what we can do today. Furthermore the current Gradle model works a lot with lazy evaluation to deal with configuration time ordering. With the new configuration model we don't use lazyness in the model anymore but provide a way to order configurations actions explicitly. Again, this will make it much easier to discover what is going on in a Gradle build. For the same reasons we will be able to provide much better and deeper error reporting. For Gradle IDE integration this also provides wonderful opportunities to navigate you precisely through a complex builds extended by many plugins.
Gradle has already excellent C/C++ support, and we will continue to improve this. We want to turn Gradle into the most powerful native build system on the planet. The new configuration model will add capabilities that are particularly helpful for the complex C/C++ domain. The dependency management work will make Gradle the first true C/C++ binary dependency management solution. And work on performance is particularly crucial for the adoption within pure native stacks.
We are excited that the Google Android team is currently working on moving their NDK support from Make to the Gradle C/C++ plugins. Once released, this will immediately turn Gradle into one of the most widely used C/C++ build systems in the world.
Tooling and IDE support
With it’s tight Gradle integration, Android Studio is pioneering how modern IDE's should integrate with build systems. Gradle enables this kind of deep IDE integration with the Gradle Tooling API, which provides a mechanism for programmatically interacting with Gradle and lies at the heart of this effort on the Gradle side. We continually add to these capabilities in cooperation with the Google Android team, for example by enabling rich test execution from the IDE via Gradle. Later this year, Gradleware will offer a Gradle Eclipse plugin which will provide outstanding Gradle integration with Eclipse.
Daemon with Watcher Mode
We take backwards compatibility very seriously. We will continue to support the current configuration model for a long time, and we will provide a way to map the current DSL to the new configuration model. However, the benefits of the new configuration model will be so outstanding that we expect many people will migrate towards it very quickly, and we will try to make this as painless as possible.
Software code bases are growing in size and complexity. Teams must deal with a mix of different languages and technologies, navigate the build and release challenges of a micro service architectures, and deal with expectations of continuous delivery.
The build system must enable a continuous integration process for such software stacks, by connecting all of the different modules together and giving fast feedback on the correctness of any change. Without such a build infrastructure (and most larger projects don't have this) the productivity of the software development process suffers terribly. The mission of Gradle is to provide a tightly integrated and super fast build infrastructure for even the largest and most complex software stacks on the planet. At the same time it should be approachable for smaller and simpler projects. The features above are a massive step towards those goals.