Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Continuous Deployment - Lean LA
Next
Download to read offline and view in fullscreen.

Share

Large scale automation with jenkins

Download to read offline

My JavaOne 2012 talk about choreography/workflow related improvements/plugins in Jenkins.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Large scale automation with jenkins

  1. 1. Large-Scale Automation with JenkinsKohsuke Kawaguchi / kk@kohsuke.org / @kohsukekawaArchitect, CloudBees, Inc. ©2010 CloudBees, Inc. All Rights Reserved
  2. 2. Have you met Jenkins?http://jenkins-ci.org/• #1 OSS CI server – Written in Java – 47K+ installations• About 7 years old• Two key emphasis – Easy to install/use – Extensible via 600+ plugins ©2010 CloudBees, Inc. All Rights 2 Reserved
  3. 3. Context• First steps in Jenkins usage – Build your project – Run tests on your project – Maybe run code analysis tools Where to go from here? ©2010 CloudBees, Inc. All Rights 3 Reserved
  4. 4. Workflow!• By which I mean interconnected jobs• Why? – Lots of manual process – Dividing by the responsibility boundary – Single job isn’t meant for complex things – Reusing pieces ©2010 CloudBees, Inc. All Rights 4 Reserved
  5. 5. Parameterized Builds• Plain jobs can be thought of like a procedure without any input void buildAcmeLibrary() { … }• Ability to pass parameters make it more useful void buildAcmeLibrary(targetPlatform) { … } ©2010 CloudBees, Inc. All Rights 5 Reserved
  6. 6. Three Things You Need To Do• #1: Define Parameters Lots of different parameter types ©2010 CloudBees, Inc. All Rights 6 Reserved
  7. 7. Three Things You Need To Do• #2: Refer to parameter values – As variable expansions: ${Browser} – As environment variables from your builds – Some parameter types expose data in different ways • File parameter ©2010 CloudBees, Inc. All Rights 7 Reserved
  8. 8. Three Things You Need To Do• #3: Specify actual values when you run ©2010 CloudBees, Inc. All Rights 8 Reserved
  9. 9. Parameterized Trigger Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin• Call other jobs (with parameters) – Wait for their completions (optional) Foo Sub1 Sub2 Sub3 Foo Sub1 Sub2 ©2010 CloudBees, Inc. All Rights 9 Reserved
  10. 10. Multi-Configuration Project• You often do the same thing with slight variations – Compile C++ code for different platforms – Test with different browsers – Or more generally, think of it as for (x in [a,b,c]) { for (y in [d,e,f]) { doSomethingWith(x,y,...); } } ©2010 CloudBees, Inc. All Rights 10 Reserved
  11. 11. Model• Define axes – One axis ≈ one for loop• Choose from pre-defined types of Axis – Generic axis: arbitrary values exposed as environment variables – Slave axis: pick slaves by their names or their labels • e.g., linux, solaris, and windows – JDK axis ©2010 CloudBees, Inc. All Rights 11 Reserved
  12. 12. Multi-Configuration Project Gimmicks• Filtering – Otherwise combinations increase exponentially – Not all combinations make sense – Use boolean expression to trim down the size (label==“windows”).implies(browser==“iexplore”) && (label==“mac”).implies(browser==“safari”) – Or tell Jenkins to cut the workload to N% • Jenkins will thin out the combinations by itself ©2010 CloudBees, Inc. All Rights 12 Reserved
  13. 13. Demo ©2010 CloudBees, Inc. All Rights Reserved
  14. 14. Other Simple Choreography Toolshttps://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin• Join Plugin ©2010 CloudBees, Inc. All Rights 14 Reserved
  15. 15. When Jobs Start Working Together…https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin• Copy Artifact Plugin – Copy artifacts into a workspace – By using various criteria acme-build #15 acme-test #24 foo.jar foo.jar ©2010 CloudBees, Inc. All Rights 15 Reserved
  16. 16. Copy Artifact vs External Repository• Almost as if artifacts are versioned acme-build #10 Last saved build acme-build #11 acme-build #12 Last stable build acme-build #13 acme-build #14 acme-build #15 foo.jar ©2010 CloudBees, Inc. All Rights 16 Reserved
  17. 17. Labeling Builds Is Useful• Especially when labels have semantics Deployed to production acme-build #10 Signed off to QA acme-build #11 acme-build #12 Last stable build acme-build #13 acme-build #14 acme-build #15 foo.jar ©2010 CloudBees, Inc. All Rights 17 Reserved
  18. 18. Labeling Builds Is Useful• More so when you automate them• Take “Signed off to QA” label for example acme-devtest #15 acme-build #12 foo.jar acme-findbugs #22 Signed off to QA ©2010 CloudBees, Inc. All Rights 18 Reserved
  19. 19. Introducing Promoted Builds Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin• Promotion = act of giving a build a label• You specify: – Promotion criteria – what happens after promotion• Label is a nice hand-off between teams – It’s like sausage making process ©2010 CloudBees, Inc. All Rights 19 Reserved
  20. 20. Demo ©2010 CloudBees, Inc. All Rights Reserved
  21. 21. Maven Repository Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Maven+Repository+Server• Virtual Maven repository – Expose artifacts from specific build – And its upstream builds ©2010 CloudBees, Inc. All Rights 21 Reserved
  22. 22. http://www.flickr.com/photos/andresmusta/7990193487©2010 CloudBees, Inc. All Rights 22 Reserved
  23. 23. Challenge: Visualization• Edge traversal breaks down on large workflow ©2010 CloudBees, Inc. All Rights 23 Reserved
  24. 24. Dependency Graphhttps://wiki.jenkins-ci.org/display/JENKINS/Dependency+Graph+View+Plugin• Uses GraphViz to draw a graph of relations• Supports copy-artifact plugin ©2010 CloudBees, Inc. All Rights 24 Reserved
  25. 25. Build Pipeline Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin• It shows how far each change has gone ©2010 CloudBees, Inc. All Rights 25 Reserved
  26. 26. Fingerprint• Yet another angle to look at data• Think of it as a lifelog for a blobAir travel JenkinsAirline check-in/TSA/immigration Jobs and workflowMy going through immigration Build #13My travel experience Fingerprint view ©2010 CloudBees, Inc. All Rights 26 Reserved
  27. 27. Fingerprint: Mechanism• MD5 checksum of a file – Recorded against builds that it appeared – (And actions that were taken) acme-build #15 acme-test #13 MD5= MD5= 604e7e07 604e7e07 ©2010 CloudBees, Inc. All Rights 27 Reserved
  28. 28. Fingerprint: Why?• Track down where it came from – My component integrates to product XYZ, and a bug was reported against XYZ 3.0.5. Which build of the component did it contain?• Cross-correlate jobs that aren’t directly related ©2010 CloudBees, Inc. All Rights 28 Reserved
  29. 29. Next Step in Workflow ✓ Aggregation of results ? Aggregation of definitions ©2010 CloudBees, Inc. All Rights 29 Reserved
  30. 30. Build Flow Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin• Groovy DSL for kicking builds – High-level primitives – Ability to define abstractions b = build(“acme-build”) guard { parallel ( { build(“acme-test1”, param1:b.number) }, { build(“acme-test2”, param1:b.number) } ) } rescue { build(“acme-teardown”) } ©2010 CloudBees, Inc. All Rights 30 Reserved
  31. 31. Demo ©2010 CloudBees, Inc. All Rights Reserved
  32. 32. Jenkow Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Jenkow+Plugin• Embed BPMN workflow engine in Jenkins – Timeout, fork, join, …• Workflow is version controlled in Git – Push to Jenkins to load them up ©2010 CloudBees, Inc. All Rights 32 Reserved
  33. 33. Next Step in Workflow ✓ Choreography defined in one place ? Everything defined in one place ©2010 CloudBees, Inc. All Rights 33 Reserved
  34. 34. Job DSL Pluginhttps://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin• Groovy DSL for defining jobs def project = "jenkinsci/jenkins" def branchApi = new URL("https://api.github.com/repos/${project}/branches") def branches = new JsonSlurper().parse(branchApi.newReader()) branches.each { b -> job { name "${project}-${b.name}".replaceAll(/,-) scm { git("git://github.com/${project}.git", b.name) } steps { maven("install") } } } ©2010 CloudBees, Inc. All Rights 34 Reserved
  35. 35. Or More Likely…• Take Existing Job, Make Adjustments def project = "jenkinsci/jenkins" def branchApi = new URL("https://api.github.com/repos/${project}/branches") def branches = new JsonSlurper().parse(branchApi.newReader()) branches.each { b -> job { using "jenkins-build" name "${project}-${b.name}".replaceAll(/,-) scm { git("git://github.com/${project}.git", b.name) } } } ©2010 CloudBees, Inc. All Rights 35 Reserved
  36. 36. Job DSL Plugin• You can go down to XML definitions• The program itself executes as Jenkins job – Control over when it executes – Store definitions in VCS ©2010 CloudBees, Inc. All Rights 36 Reserved
  37. 37. Or just a bit of Perl/Python/Ruby scripts• Programmatically CRUD jobs $ ssh jenkins get-job foo | sed -e s/old.gitserver.com/new.gitserver.com/g’ | ssh jenkins update-job foo ©2010 CloudBees, Inc. All Rights 37 Reserved
  38. 38. Templates (in Jenkins Enterprise byCloudBees)http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-overview.cb• Share some traits with Job DSL – Define job once, generate many variations – Update definition, and reflect it everywhere• But different – Templates are defined in GUI, not in a program – Individual variations are manually updated by users ©2010 CloudBees, Inc. All Rights 38 Reserved
  39. 39. Conclusion• Lots of useful building blocks for automating even more – That means many people are doing this• Take your automation to the next level ©2010 CloudBees, Inc. All Rights 39 Reserved
  • AmolJadhav73

    Feb. 6, 2018
  • rll1956

    Mar. 31, 2016
  • JunshanHe

    Dec. 18, 2015
  • jorgearma1982

    Sep. 9, 2015
  • linekin

    Apr. 9, 2015
  • kanrangsan

    Feb. 5, 2015
  • teraslee0

    Jan. 19, 2015
  • nextnottakenusername

    Dec. 4, 2014
  • TaejeonKim

    Oct. 18, 2014
  • tvhung

    Sep. 30, 2014
  • BenjaminBRABANT

    Sep. 18, 2014
  • BenjaminBRABANT

    Sep. 18, 2014
  • LeeHyeongchae

    Aug. 12, 2014
  • hypermin

    Aug. 12, 2014
  • fbrnc

    Jul. 4, 2014
  • TomaszNowodzinski

    Apr. 29, 2014
  • thegiive

    Aug. 5, 2013
  • jollychang

    Jul. 21, 2013
  • suiaing

    May. 27, 2013
  • hshenry

    Jan. 26, 2013

My JavaOne 2012 talk about choreography/workflow related improvements/plugins in Jenkins.

Views

Total views

40,720

On Slideshare

0

From embeds

0

Number of embeds

30,288

Actions

Downloads

220

Shares

0

Comments

0

Likes

28

×