Jenkins workflow

Managing Jenkins workflow with Gradle: Gradle Summit 2015 Video

Jenkins is a core part of your continuous delivery pipeline and should be managed with a degree of discipline. Managing large numbers of Jenkins Jobs can be challenging and tedious, especially when many of your jobs are nearly identical and only differ by a few values. This session will introduce the Gradle Jenkins Plugin and demonstrate how it can be used to take the pain out of maintaining Jenkins workflow from small projects with a few jobs up to large environments with hundreds of jobs.

Recently, Jenkins has brought the “Jenkins Workflow” plugin into the core of Jenkins Continuous Integration pipeline automation and renamed it the “Pipeline Plugin”. Pipeline allows users of Jenkins to script automations using the Groovy programming language. This talk does not specifically address these newest features, rather we use the term workflow more generically to describe build automation pipelines defined in Gradle and how to integrate that with Continuous Integration in Jenkins.

This session will compare various tools for maintaining a Jenkins environment and will introduce the basics of using the Gradle Jenkins plugin. It will cover using templates to generate jobs, techniques for keeping scripts clean, and mechanisms for customizing the build. The format will be a mixture of both discussion and live examples.

All right, let’s go ahead and get started. Welcome to the Managing Jenkins With Gradle talk. If you came here looking for IntelliJ talk, unfortunately, the IntelliJ speaker could not make it into the country. We moved this talk to this slot.

First, a little bit about me. My name’s Gary Hale. I am a principle engineer at Gradle, Inc., formally Gradleware. Before I joined with the Gradle team, I was a user of Gradle for many years. One of the things that I did was I wrote this Gradle Jenkins plugin, which we’ll be talking about today.

Personally, I’m somebody who, I style myself as a compulsive automater. I feel like you should automate as much as possible. Automate until there’s nothing left to automate, and then you start automating the automation. I think it’s the only way to create a livable, bearable, continuous delivery life cycle.

What you should do in this talk, please ask questions. If there’s something that’s not clear, ask questions. The more you ask questions, the more you’re gonna get out of this talk. The more interesting it’s going to be for you, the more interesting it’s going to be for me. Please, feel free to interrupt and interact at any point.

A little bit about what we’re gonna talk about. First I want to talk about kind of an evolution, or at least my evolution of trying to figure out how to manage and make sense of a large Jenkins environment. Then I’m gonna talk about the Gradle Jenkins plugin, itself. Talk about some of the basics, a little bit about the DSL that’s there. Then I’d like to demonstrate just a couple of quick samples of how we’ll use it. Or how we can use it.

What is it that we like about Jenkins? What makes Jenkins, you know, awesome? Well, first of all, it’s open source. Which means that the source is available to us. We can always go out and look at it, and if we have questions, try to understand how it works. It also has a nice distribute model, so I can create a farm of a bunch of different servers. I can push my jobs out to all these servers, and do quite a bit of work from a centralized location. It’s extensible, so if it doesn’t have functionality that I’m looking for, I can write my own plugins. I can extend Jenkins and put things in there. It has an active ecosystem, so the core of Jenkins itself is actively under development. They’re adding new features, fixing bugs. There’s also this very active plugin community, so there’s, you know, a vast number of plugins that are available out there for Jenkins.

Why is Jenkins not awesome? I mean, what tends to be the problem? Is that once we get beyond a few jobs, it becomes very tedious to maintain. We start trying to find better solutions for managing our environment.

When I was looking at trying to make sense of this environment that I had. Where I had several hundred jobs, I think we were up to about 600 jobs, something like that. I was kind of looking at what are some of the things that I want to be able to do. Right, so what are some of the motivations.

The first category of motivation I would present is discipline motivations. The first thing I would want to do is be able to track changes. As things get changed within my Jenkins environment, I want to be able to know when it as changed, who changed it, that sort of thing. Now, there are some things that kind of come out of the box with Jenkins. You know, there’s like the audit log plugin, and a couple other things, the job config history plugin. These give me a little bit of information about that … But tracking changes is something we’ve done for a long time in the SCM world. The source control world. If I can up with a solution that allows me to leverage all of my SCM tools, well, then I can just track those changes the way I would any other source control change.

I’d like to be able to couple my configuration of my Jenkins job with my project. As my project is moving through its life cycle, as I’m making changes to how it works, I want to be able to couple the changes to the Jenkins jobs with the changes to the project itself.

I want to be able to perform validations, right. If I’m making a change to my Jenkins environment, but I only expect to change three jobs, I want to be able to validate before I push it out there that I’m only gonna change three jobs. What I don’t want to do is push out a change to 100 jobs, and then I have a bunch of angry developers asking me why are they receiving all these emails about failed builds, and things like that. If I can do something where it allows me to validate my changes before I push them out, then that’s going to help with my discipline.

Another category of motivations here would be maintenance, right. The first thing, minimize overhead. I want to make my job easier. I don’t want to spend a whole lot of time clicking around in the Jenkins Gooey to make a bunch of changes. I want to be able to do those very efficiently and very fast. If at all possible, I want to leverage my distant skill set. If I know Groovy, I want to be able to make my changes, you know, with a Groovy tool. There’s nothing wrong with a Polyglot environment, or having you know, a lot of different tool sets out there … But if I’m leveraging an existing skill set, I might be able to find some synergies between some of the different automation that I have from my projects or my organization.

I want it to be as DRY as possible, right. If I’m making a change to 100 different things, what I really want to do is make that change once and have it be applied to 100 different things. I don’t want to have to make 100 different changes.

Ideally, I’d like to be able to rebuild from scratch, as well. There’s probably a bunch of different use cases for this. One of those might be to bring up a brand new server somewhere, and bring down an old server. Maybe I’m growing the size of my environment, or the size of my master, or something. It’d be nice if I could just make some changes to a script, or something like that. I bring up new server, and then push out all of my jobs in one fell swoop.

Lastly, I might want to be able to delegate some responsibilities out. If I’m, say, a build engineer in a larger environment, I’ve got a lot of development teams, I might want to delegate out some responsibilities to them for configuring their jobs as well. I would like to have a tool that allows me to do that.

Kind of part and parcel to that is the ability to enforce standards, you know. If I’m delegating a lot of stuff out, there might be still things that I want to have a standard configuration within my environment. I want to be able to enforce that, and I’d like a tool that helps me to do so.

Okay, so what I want to talk about now is just a little bit like my evolution, in terms of learning how to better manage my Jenkins environment. It’s probably a pretty typical evolution. One thing I will say, though, is I’m not trying to position the Gradle Jenkins plugin as being the best possible solution. What I’d like to do is position it so that it’s clear where it … What its strengths and its qualities are.

If you’re like me, you found Jenkins. Pretty neat, you start creating jobs. Pretty soon you realize that there’s this copy from existing. I don’t have to reconfigure every job every time. I can just copy a job, and you know, make a few edits. That’s fine, but pretty soon, I’ve got 30, 40, 50 jobs, and I decide that I want to add a new plugin. Well, I have 30 or 40, 50 jobs that I now have to go back in and edit manually. I realize that I have to have, you know, a better solution.

One of the fist things that you might come across are what I would call scripting solutions. These are things like the script console within Jenkins. The ability to run system Groovy scripts. The Scriptler plugin. These are basically Groovy scripting mechanisms that allow you to modify the Jenkins configuration.

Then at some point, you might start looking at what I would term job-like solutions. These are things like the Jenkins Template plugin, or the very fine Jenkins Job DSL. These kind of run as jobs inside our Jenkins environment, and allow you to manage a single Jenkins instance. If you’ve got a very large organization, or something about how you do your business where you might have multiple Jenkins server farms, different masters … These jobs just kind of run inside the box. They’re inside a single Jenkins instance.

When you start talking about multiple instances, you have to start thinking about external solutions, things that operate outside of the box that can manage multiple instances. One thing that’s out there is the Jenkins Job Builder, and then there’s also the Gradle Jenkins plugin, which is what I’m going to talk about today.

There’s not a whole lot to be said about manual solutions, other than to say at some point, you know, the number of jobs that you have just become overwhelming. You need to find a higher order solution to, you know, how am I going to manage this environment.

With scripts solutions, again, these are a way to run Groovy scripts within Jenkins that modify the environment. They’re extremely powerful. What you’re really doing is modifying the objects within the running Jenkins instance, and making changes that way. As if you were making those changes through the Gooey. It does require significant understanding of the Jenkins model, right. You have to understand to make this change to this particular attribute, or this job, what are the objects that I have to traverse to get to that particular change. What you find yourself doing is ultimately, there’s not a lot of good information on this. You ultimately end up going out and pulling down the Jenkins source, and looking through all of that. That becomes a little tedious. If I’m only changing one thing, you know, do I really want to spend 20 minutes looking through source code trying to figure out how to do it. It is written Groovy, so it does leverage our existing skill set. I mean, you’re here at the Gradle conference, you’re probably doing a little bit of Groovy development.

One of the down sides of this is that it has limited capabilities for parameterization, right. What you find yourself doing is copying scripts around. Or you know, kind of creating little blocks at the beginning of your scripts that have all of the parameters, and you edit them before you run them, that sort of thing. You know, it’s really kind of clunky. It’s not very elegant.

What these scripts usually are good for are one-off type things. Where I’m gonna change one thing across my entire environment. I’m gonna do it once, and then I’m never gonna look back … But not necessarily for things that I might want to reuse scripts over and over and over again in different contexts.

When we start talking about job-like solutions, there’s the Job Template plugin. Now what this is, is this is the ability to create a job within Jenkins that have kind of … I might configure a particular builder, or a publisher, or something like that. Then what I do, is I point to this thing from all these other jobs. You create this many to one relationship, right. Whenever those jobs run, what they’re actually doing is grabbing that configuration from the template job.

These tend to work out better when they’re combined. By that, what I mean is instead of creating one big template, what I want to do is create a lot of little templates, and combine them together in jobs. When I have one big template, the problem that I run into is that, sooner or later, I have, say, 400 jobs that use this single template, and I decide that 120 of these actually should be different. I want to do something slightly different with them, so I split that template up. Now, I’m back in that situation, I’ve got 120 jobs that I want to change. These tend to work a lot better when you keep them very small, very granular.

The Job DSL plugin is a pretty nice tool. A lot of people get to the point where they’re using the Job DSL plugin, and they’re pretty good with that. They don’t really move too much further int heir evolution. What this is is a very powerful DSL for describing jobs, modeling jobs. In fact, this DSL is what we use in Gradle Jenkins plugin to allow you to configure jobs.

Now, this is DSL. It’s just Groovy script, right, so it’s storable. I can put this into source control and track it in that way. It has the support for template jobs, which are just, you know, jobs that provide various templates. Unlike the job template plugin, it’s much easier to kind of split these up and reuse them in different ways.

One of the down sides of this is that there isn’t a good way to validate our jobs before you push them out there. There are some things that are out there that may allow you to, say, take your Job DSL, generate the XML for the jobs, and then you can maybe do some validation that way. It’s not really easy to use. It’s not kind of a first class citizen for the Job DSL plugin.

The other things is it really only operates within one instance. If I have multiple instances, if I have a dozen different Jenkins servers, I have a dozen different instances of my Job DSL seed jobs, and things like that. Sure.

Actually what I’m saying is that what I want to do is make some changes, and then run my validation and find out well, what’s actually going to change, right. Is it just these three jobs, or is it 100 jobs. You know, I want to have some kind of idea before I actually make my change, that you know, what is it going to do. It’s you know, kind of like a dry run sort of thing. Does that make sense?

Oh yeah, sorry. The question was the validation, what is that. Is it something like pushing out a sample job, and then being to validate it that way? The answer is no, actually what it is, is the ability to kind of do a dry run before I push my jobs out there. See that only the jobs that I expect to change will actually change.

Yeah, so the idea is that … The question was can you go over what I meant by the one instance again?

With the Job DSL, right, it’s kind of contained within a single Jenkins master, right. If I had 12 masters, I would have 12 different seed jobs, and you know, all of the stuff that goes into making the job DSL stuff work. Does that make sense? Okay.

Okay, yeah, and there may be, you know, some solutions to some of these things. You know, I’m just kind of pointing out some of the ways that it differs from the Gradle Jenkins plugin can work, too.

When we start talking about external solutions, the, you know, first thing you might find is Jenkins job builder. What this is, is it’s a Python tool that has a Yaml configuration. It’s part of the open stack, a suite of solutions. It produces some very clean configuration, but kind of the down side of this is that it’s purely declarative. Now, there are some ways to do some macros, they have like a macro functionality, but you don’t really have the capabilities of a programming language, like you would with something like Gradle. Where you can do looping and you know, make use of all of the goodness that Groovy brings to the table.

We get to the Gradle Jenkins plugin. This is, you know, ultimately what we ended up doing. This is a purely programmatic solution, so it is a Gradle script, right. It runs in there. You’ve got all the programming capabilities of Groovy. It allows us to do direct XML manipulation of jobs, if I wanted to. Or I can make use of the Jenkins Job DSL. What I can do is actually put Jenkins Job DSL into my Gradle script, or I can leave Job DSL out into like script files, and the such, and just pull them and reference them for my script. I can easily take an existing Jenkins Job DSL installation, and convert it over to a Gradle Jenkins plugin.

It does have support for template jobs. These exist only in the Gradle configuration, and are just used as templates for the jobs that get pushed out there. Unlike something like the job template plugin, it doesn’t actually have kind of like these null jobs that sit in your environment. These all exist just in your Gradle script.

It has the ability to do creation, deletion, and validation of the environment. What it actually does for validation … The question was how do you validate it? What it actually does for validation is an XML diff. By that, I mean that it checks to see if the same elements, attributes, and values are there. You know, it’s not just a straight textual diff, but it’s a little bit more smart. What it’s doing is it’ll go and grab the job that’s out there, compare it to the job that would be pushed out there if you did, and say are there any differences between these two. Is there another question here? Yeah.

Yeah, so the question was would it better to push this into the Job DSL, or leave it in the Gradle Jenkins plugin. Yeah, I think the answer is the Job DSL really is just for describing the jobs themselves, but this is kind of a layer on top of it. You can think about the Gradle Jenkins plugin s being kind of way to pull the Jenkins Job DSL functionality outside of the container, outside of the Jenkins instance, and manage, you know, externally, all right.

Okay, so before we move forward, I want to ask a quick question. Which is, does it make sense? Is it a good fit to manage Jenkins with a build automation tool? Just think about that. We’ll come back to that.

Some kind of core concepts here that I want to talk about. Kind of key here is this idea of a static model, versus a live model. The static model is what exists in my Gradle build. This is how I modeled my jobs, my views. I’ve also got templates in there, as well. The live model’s what actually exists up on my server. Whenever I’m doing functions, or I’m running tasks with the Gradle Jenkins plugin, I’m operating on one or both of these models. For instance, updating would be taking everything that’s in the static model, pushing it up to the live model.

Validation would be comparing everything that’s in the static model with what’s in the live model, that sort of thing.

Yes, there is some limited functionality for that. There is a task called dumpRemoteJenkins Jobs. What you can do, yeah … So the question was … I’m terrible about this, so I got to keep doing this. The question was is there a way to take what’s in the live model, pull it down into the static model.

The answer is yes, you can. What you can do is model, but what you would have to do is model the job in the static model, at least from the job name perspective. Then you can do a dumpRemoteJenkins Jobs. What that does is pulls whatever is in the live model down, and dumps it on your file system so you get, you know, you would have the XML at that point.

The other way, and you know part of the reason that isn’t really a first class citizen, is that you’ve got a really nice interface for pulling that stuff down already. I can just use the job name/config., and that’s going to pull down the XML for my job already. I can easily get to those jobs. Or an existing job within Jenkins. Does that answer your question? Okay.

All right, so let’s start with talking about some of the tasks in the Gradle Jenkins plugin. The first task is updateJenkinsItems. This is just gonna take whatever is in my static model, push it up to the live model. It’s gonna make everything appear. If a job already exists up in the Jenkins instance, it’s gonna override it.

The next task is deleteJenkinsItems. What this allows me to do is delete whatever is in my static model, from the live model, right. I might have additional things in my live model that I’ve created for some other purpose. It’s not going to delete those, but it’s gonna delete everything that I’ve got defined in my static model.

I’ve also got dumpJenkinsItems. What this does is takes everything in the static model, everything that’s been modeled in my Gradle script and dumps it out to file. It’s gonna take whatever I’ve defined, turned it into the actual XML that would have been pushed up to Jenkins, and then dump it into a directory in my project.

Then there’s validateJenkinsItems. This is what we talked about for validation. It’s gonna do an XML comparison, look at what’s in the static model, what’s in the live model, and decide whether all of the same elements, attributes, and values exist. Is it functionally the same job, is what it’s trying to determine.

Finally, there’s dumpRemoteJenkinsItems. What this does is it says for all of the jobs that are in my static model, find that same job in the live model, dump it down into a XML file on the file system.

Okay, so start by talking about kind of the high level conventions here. First of all, I’ve got the Jenkins block, which is all of the configuration for this is contained inside this Jenkins closure here. Now the first thing that I can configure are servers. These are the servers that I might push out to. I can have as many servers defined here as I want to. I can point different jobs at different servers. You know, this is just going to represent all of the servers in my farm that I’m managing with this particular build.

I’ve also got templates, and these represent template jobs that I can then reference from my jobs themselves. I can create one template, spin off as many jobs as I want to, that just maybe vary that job in a few minor attributes.

Yeah, so you can configure, authenticate it, as well as unauthenticate it. The question was basically, can I configure authenticated servers in the model here? The answer is yes, and I’ll show you an example of that in a minute.

When I servers, do I mean the masters? Yeah, yeah, so these are the masters. I don’t care about the slaves. I’m pushing configuration into the master, the master knows about the slaves. You know, in the Gradle Jenkins plugin, it’s really not material.

Yeah … No. The question is do I have a dependency on the template plugin itself? The answer is no. These exist within the Jenkins plugin itself. There’s no dependency on an external plugin.

Okay, so the next thing is jobs. These represent the jobs that are gonna exist in my environment. As well as the views block, which is gonna represent any views, and will reference the jobs that I specify.

First we have servers. To specify a server, I really only need the URL, as well as a username and password. Now, I probably wouldn’t put username and password in the script. I’d probably have this stored off into a Gradle.properties, or something like that. If I’m running from the console, and I haven’t specified username or password, it’ll prompt for a username and password.

I can also do unsecured servers, as well. The default is to expect a server to be secured. If I want to specify that it’s unsecured, I have to explicitly tell it that secure is equal to false.

Are there any mechanisms for tokens or keys? Right now, there aren’t. That’s something that, you know, I would like to add at some point, but no, there isn’t.

Templates … With most of the different templates, jobs, views, there’s multiple forms of that configuration. In this, the XML form, what I’m doing is basically I’ve got a raw XML file somewhere on my file system. I’m telling it that for this template, read in this XML file, and that’s going to represent the template. Anything that uses that template is gonna have all of the configuration attributes of the job described in this file. This job is going to be, this is simply the configuration that I might pull down from a Jenkins configuration.

Now, this is showing the file form for this. I could also specify this as a string. Or I could specify it as like a XML slurper configuration. There’s also a DSL form, which allows me to read in Jenkins Job DSL, so dsl file (template.dsl). This would read in whatever DSL has been configured in this template.dsl file, and make that the basis of the template.

For jobs, again, there’s also this XML form of the job. Which would read in a raw XML file and specify the job. Now notice that I’ve got servers.server1 here. What this is saying is that build_job1 is gonna be deployed to the server 1 that I specified before in my servers block … All right, one more … So server1 is how I specified. These are all named domain object containers. Servers.server1 is how I would reference this server form my jobs, or my views. I don’t necessarily … There we go … I don’t necessarily have to specify a server for each and every job. If I’m deploying only the one server, I can set up a default server. What that default says is that for every job that hasn’t specified its servers, use this server.

DSL form … So in this case, I’m actually embedding DSL into the script. This is obviously very, very simplistic. In a real scenario, what I would do is apply this template, and then add additional configuration on top of that. This is using the actual Jenkins Job DSL to access the template1 template. Template1 is created in our templates block, and then I’m referencing it here inside of the DSL block for the job.

No … The questions was can you accumulate multiple templates? The answer is no. You really have one template, and then you build on top of that. Now, the reality is, is that you could work around that with some clever programming. Where your template is actually some sort of function or closure that I apply, and I can apply multiple of those if I wanted to.

Views, same sort of thing. There’s an XML form. There’s also a DSL form to our views.

The question is am I pulling these files down from Jenkins, or am I pushing them up to Jenkins, right? The answer is yes. You know, the reality is that I’m pushing up to Jenkins most of the time … But what I’m pushing up there may have been something that I pulled down from Jenkins previously, right. My template may have been something, you know that XML file, that raw XML file may have been something I configured in Jenkins, then pulled it down, and then used that as a template for all of the jobs that I’m pushing up. That’s one way you could utilize it.

Yep, yeah … The question being, you know, so you’d pull it and then change, and you know, that’s what you push out there? The answer is that’s one way you could do it, right. Pull the XML down, change the XML, and then have, you know, the Jenkins plugin push it up there. The other is that you do your configuration, you configure your template in Jenkins, then pull it down, and then have the Gradle Jenkins plugin use that template, and then build on top of that for, you know, some larger of jobs that you’re pushing out there. There’s a lot of different ways that you can approach it. You know, and that’s why, you know, you’ve got an XML form, you’ve also got a DSL form. You’ve got a lot of flexibility in how you create your jobs.

Uh-huh, yes … Yes. Yeah, so the question was do we support built pipeline views within Jenkins? The answer is yes. The Job DSL, the Jenkins Job DSL has support for that. By virtue of we can use Jenkins Job DSL, then we can support pipeline views, and that sort of thing. I think you can support kind of like the classic view. You can support the nested views, and then you can support pipeline views. There may be some others there that I’m just not thinking of off the top of my head.

Yeah … So the question was how about CloudBees folders? That is one place where the Jenkins plugin is a little bit clunky. There is the ability to override what the different URLs look like. That’s the problem with CloudBees, is that it actually changes the URLs to something that’s slightly different than classic Jenkins. Within the Gradle Jenkins plugin, there is a way to override the different URLs for creating jobs, deleting jobs, et cetera. What you can do is create a job at a specific URL, right, that references that folder. It’s not as nice as I would like it to be, but you can do it. It is possible.

The last thing here that I want to talk about is the multiple DSL form. What this allows me to do, if I’ve got an existing Jenkins Job DSL environment, I might have a whole bunch of DSL scripts that I want to pull in. Instead of configuring each job individually, what I want to do is just pull in all of that DSL files in a particular directory. What this will do, read all those in, create a list of jobs based on what’s in there, and push that up there, too. This is just another way to organize your DSL and scripts.

Okay … Okay, so can I elaborate on that? If I’ve got multiple DSLs, what does that really mean? In this case, imagine that I’ve been using Jenkins Job DSL for the past year, and now all the sudden I’ve got five different servers that I want to manage. I decide I want to try out Gradle Jenkins plugin. Well, I don’t want to go into my Gradle and you know, recreate all of that configuration that I already have in existing DSL scripts out there. What I want to do is just pull all of those scripts in. What this is gonna do is look in these DSL files. They’re gonna have things like the job name, and all of the configuration there, and it’s just gonna pull it in from there instead of having to explicitly list out the jobs that I’m creating in some way. Does that answer your question?

No, not that I’m aware of. The question was is there a way to take manually configured jobs and exporting them to DSL? No, there’s not a way that I know of … But you could pull the raw XML down, and use those as templates, or something like that. I mean you know, that may be the way to do it, but I’m not aware of any way to translate existing XML into Jenkins Job DSL.

Uh-huh … Yeah. I mean, I’m sure that you could do some clever things with Groovy. I mean, it’s great at reading XML, and things like that, but it’s … There’s nothing that I know of out of the box that would do that.

Okay, so do some demonstrations here. Unfortunately, we will not have a crocodile involved. Apparently, there’s some problems with city ordinances and large reptiles in public spaces. Let’s go ahead and look at some quick examples here. We’ve got about 10 minutes left.

The first example I want to look at is just a basic config to give you an idea of what it looks like. Let me make this a little bit larger. Let’s go into … Okay, so what we see at the very beginning here is that I’ve got a buildscript block that pulls in my Gradle Jenkins plugin. You can also use the new 2.0 plugin DSL, which would be much more terse. You can also use the legacy form. You know, I apply the plugin com.terrafolio.jenkins. Is the font big enough for you to see it? You want me to make it a little bigger? It’s all right … Okay.

Then I have my Jenkins block. The first thing I do is define a server. I’m looking for a server on the local host at 8080. I’m telling it that the secure is equal to false, and I’m naming that server demo. Remember I said that I could define a default server. Here, I’m just saying that there’s a default server called servers.demo, and if any job doesn’t specify its own configuration, then just use that.

Now for my job definition, what I’ve got is a job, I’m gonna call it demoJob. Then, I’m referencing the XML file buildjob.xml that’s here locally. I look at buildjob.xml, this is an XML file that’s just been pulled down from Jenkins. It’s been configured in some way. Got a few things in here, I’ve got like a choice parameter in there, and a couple other things. A description, that sort of thing. Let’s just go ahead and run this guy. I do Gradle updateJenkinsItems. It starts a daemon and goes. It pushes that job up to my local server, 8080.

If we look at localhost:8080, we see that I didn’t have anything in there before. If I refresh it, we see that I’ve got my demo job in there. If I look at demojob, look at the configuration, we’ll see some of thee stuff that I had in my job. Like my choice parameter is in there. The description that I put in there for test. That’s just a very simple example of a single job.

Now, at this point I could validateJenkinsItems. We see that it comes back it just basically says built successful. Which means that the static model matched the live model. Let’s go ahead and make a quick change in here. Change to break validation there. Now I’m gonna run validate again, and we see that it failed. It tells me that the item demojob differs from what’s up on the server there.

Same time, I can do deleteJenkinsItems. This is gonna remove that job from the server. Now, if I go and look, we’ll see that … well, yeah, I’m getting a 404 now because the job is gone. If I look at here, we see that I don’t have any jobs configured in here anymore … Question.

Yeah, it’s doing an XML comparison. Yeah, it’s not a textual comparison. The question was when I do the validation, is all I’m doing comparing the job XML. The answer is yes, but it’s doing an XML comparison. It’s not doing a textual comparison. It’s actually trying to determine if the configuration is functionally the same.

Yeah, so if you were to run this with info, so if you did info, it’s gonna show you all the differences that it found. Some of those might not be functional differences, right. It’s going to give you a lot more information than probably what you want. You can find out from that information, you know, specifically what are those differences.

Yeah, so the question was if you’re using DSL instead of raw XML, is the DSL gonna generate XML to push up there. That’s exactly what it does. It uses that Jenkins Job DSL to kind of model it. Then it produces XML, and that’s what it actually pushes out onto the server.

Yes … So the question was does it do that for validation, too? That’s exactly what it does. It generates the XML, then it compares, you know, the two.

Okay, so in the interest of time, let just jump over to, we’re kind of running out of time here. Let me go to the last example here. I’ll show you something a little bit larger. Let’s look little.gradle.

What I’ve got in here, again I’ve my build scripts. Then for each project in this instance, what I’m gonna do is create multiple. I’m gonna create about 100 different jobs, all right. What we see here is that I’ve actually got two servers. I’ve got the non-prod server and then a prod server. I’m gonna push different jobs out to different places, depending on what’s in that job.

Now, you probably have something more sophisticated than just a loop of 0 to 100, but the point is you could get that information from someplace else. You could be querying something like your get management server, or something like that to find out what are all the branches that are available, or you know, some other source of information. What I’m gonna do here is that I’m creating a template called buildTemplate. I’m gonna create build jobs and deployment jobs.

My build jobs are going to be based off of the build template. They’ve got some common configuration here, so the SCM trigger is first. This is all Jenkins Job DSL here inside the DSL block. Then, I’m going to create a step to do a Gradle build. The Gradle build is going to run the build task. Then I’ve got publishers that do some things like send out emails, and the such. Then I’ve also got deploy templates. This is going to run a Gradle task, deploy, and also send out an email.

Now, this is all being done at the top level in the root script build, or the root build script. If we look at what I have in here, I’ve also got sub projects here of builds and deployments. These are gonna specify individual builds and deployments. If I look at builds, what we see here is that I’m looking at all the different projects that I’ve defined, those 100 different projects. I’m gonna create a build job for each branch. Now, my branches I’ve defined statically up here, so I’ve got a master, and a develop branch. Then I’ve got looping over all projects, all branches, and then create a job for each of those. That’s gonna create 200 jobs right there.

Then, I’m gonna create views here that are for each branch. What I’m gonna say is okay, if the branch is equal to master, then I’m pushing it to the server.prod. Otherwise, I’m pushing it to servers.nonprod. I should see all of my jobs for master up on the prod server, all of my jobs for develop on the non-prod server. I’ve done the same thing up there for the job itself. Then I just configure some things about the view, and I provide a regX there to specify my jobs. Let’s just go ahead and run this guy. We’re gonna push out like 300 jobs, something like that. Yes, question.

Yeah, so the Jobs DSL has the support. The questions was, if you’re using a Jenkins plugin, does it come with a DSL that you can use? Well, the Jenkins Job DSL has to support it. If you go out and look at what the Job DSL has out there, it supports a lot of the major plugins that are available. There may be some that it doesn’t support. In that case, what you can do is drop down into direct XML manipulation to manage that, as well.

Yeah, the question was, can you combine XML with DSL? The answer is yes. You know, ordering is important, right. Typically, what I would do is apply the DSL first, and then apply all of my XML transformations later. You can get into kind of some weird situations if you mix them together, but it is possible, there’s no doubt about it.

I pushed out about 300 jobs in something like 20 seconds. Probably be even faster on a more highly powered machine. If I look out here, this is my non-prod server. See that I’ve got all of my jobs here. I can look at develop builds, these are all of my develop builds. The deployments, I split those up between different environments, so I’ve got a staging environment and a test environment. If I look at the other server, which is on board 8081. We’ll see that here are all my prod jobs. I’ve got my master builds as well as my production deployments. Those are some examples of this.

Let me skip back to … Let’s come back to our question. Does it make sense to manage Jenkins with a build automation tool? I think it’s the wrong question. I think the right question is, is Gradle a build automation tool? Is a Swiss army knife a knife? Yes, it is, of course, it’s a knife … but it’s also so much more. I think that’s true of Gradle, is that it’s much more than a build automation tool. It happens to be very good at build automation, but it’s also a project automation platform. You can use it to solve just about any automation in your continuous delivery pipeline.

If you take away nothing from this, I hope you take away that mindset. That you can use Gradle for a lot of other things than just building and testing my software. Got some references here. The reference to the Gradle Jenkins plugin, as well as the demos from this demonstration. You had a question before, or were you trying to answer my question?

When you say matrix, like a matrix build within Jenkins? Yeah, I’m pretty sure that the DSL supports matrix builds, as well. The question was, can I redo the configuration, but using a Jenkins matrix build, instead of just freestyle build? The answer is yes.

Meaning starting Jenkins up from within Gradle? Yeah … Yes, yeah. The question or the comment was that you could potentially start Jenkins from within Gradle. Yeah, in fact, look at the Gradle Jenkins plugin. You’ll see an example of that for my integration test. I start up a Jenkins instance before I run my integration test, you know, run my test, and then stop it afterwards. You know, it’s as easy as starting any other process within Gradle. Which is just using like an exec task, or something like that. Question.

Mm-hmm (affirmative) … Okay, so the question is, if I wanted to provision my Jenkins instance, and not just pushing jobs up there, could I declare what Jenkins plugins need to be available before I push my jobs out there? The answer is right now, no … But just recently, well I’d say relatively recently, say in the last six months Jenkins supports the ability to specify what plugins are necessary when you push a job up. You push a job up, and then it’ll actually install that plugin as part of that whole process. I haven’t built that into the Gradle Jenkins plugin yet. It’s fully on my roadmap, my intentions to do so, but it’s not there quite yet. It hasn’t historically been supported by Jenkins, so that was the blocking.

Right, yeah. Yeah, and the comment was that you could potentially do integration tests with new versions of Jenkins, and new versions of plugins. The reality is that you could potentially do that now, right. As part of your Gradle integration test that you are writing, the first thing you do is explode that or somewhere, copy all of the appropriate plugins in there, and then start up the instance, and do that. If you were gonna track all of those things, those plugins and the such, outside of the jobs themselves, you could potentially do that.

Yes, yes, and that’s the ultimate target here. I mean, you know, we’re not quite there in terms of the configuration for Jenkins. You know, there’s not a lot of facilities for doing the configuration, but ultimately we want to get to the point where its infrastructure is code.

Yeah … The question was, do I envision that as being DSL, where I define the plugins themselves? I don’t envision that, but I believe the way … I’m trying to remember off the top of my head how Jenkins has exposed that capability. I want to say that there’s a bit like a URL that you hit with a job, and it will install what’s necessary there. I don’t know, I don’t think it’s going to be in DSL. I think it’s gonna be implicit in the job. The fact that the job uses it, then tells Jenkins enough so that he can install the necessary … Right, right.

The question was, you know, in the instance where I’m actually going to update an existing job, is there any way to determine which job I’m gonna update? Yeah, so when you run that Jenkins validation, it’s going to tell you which jobs specifically, you know, are different and would be changed.

The question is, can I specify like a subset of jobs or a subset of servers? Yeah, there are some filters that you can apply, that allow you to filter, you know, the jobs that I’m actually going to operate on. You can also create tasks, and I didn’t cover this in the talk just because of time, but you can also configure tasks that will do the same thing, right. They filter by job and by server.

Since last year, not much. The question was, what new features have been added to the Gradle Jenkins plugin since last year? The answer is that there hasn’t been a lot, mostly because of time constraints on my side. There has been updates to newer versions of Job DSL, and other dependencies, and the such, but that’s about it. There’s nothing of particular note since last year.

Okay, well, thank you for coming, guys.