CI/CD Patterns for Kubernetes Environments
•
5 likes•928 views
This document discusses CI/CD patterns for Kubernetes. It begins with an agenda and speaker bios. It then discusses the challenges of CI/CD for Kubernetes, including that it is a new space with unclear standard practices. It introduces concepts like environment and application repositories, manifests, and applied manifest sets. It also debunks some myths. The main part describes four related CI/CD processes: application code/config updates, environment config updates, environment restore, and detecting environment config drift. It uses diagrams to illustrate how a tool could implement these patterns.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to CI/CD Patterns for Kubernetes Environments
Similar to CI/CD Patterns for Kubernetes Environments (20)
More from Andrew Phillips
More from Andrew Phillips (11)
Recently uploaded
Recently uploaded (20)
CI/CD Patterns for Kubernetes Environments
- 1. CI/CD Patterns for Kubernetes Kubernetes NYC, May 29th 2018 Andrew Phillips & Lars Wander
- 2. Agenda ● What’s the challenge? ● CI/CD for Kubernetes: concepts, mental models & mythbusting ● An example implementation using Spinnaker ● Other implementation options & trade-offs ● 3 Takeaways ● Q&A
- 3. The bio slide ● Been on most sides of this space: developer, infra builder, product owner, evangelist and more ● Long-standing open-source contributor ● Author and regular conference and meetup presenter ● Co-organizer of ContainerDays Boston & NYC ● Leading OSS Spinnaker team (5 engineers) ● Been on Spinnaker team about 2.5 years ● Wrote initial Kubernetes-Spinnaker integration, as well as the newest one (v2) ● Author and semi-regular conference and meetup presenter
- 5. Why is this hard? 1. New space, new technology, new conventions 2. Unclear with “standard” practices to carry over vs. which to revise 3. Early stage: lots of different and sometimes conflicting approaches out there 4. Tooling often still very early in the maturity cycle 5. One size does not fit all
- 6. Pattern vs. implementation ● In an early stage space, tooling choices and thus tooling recommendations are difficult ● Patterns are better! ○ Simpler to describe ○ Easier to reason about ○ Higher signal-to-noise ratio ● a.k.a. “let’s talk about the interface”
- 7. 3 takeaways 1. Code and config deployments in a Kubernetes environment are not fundamentally different from other “as code” scenarios 2. There is a mental model for thinking about code and config in a Kubernetes environment that enables four important CI/CD patterns: app code/config rollout, env config rollout, env restore and env drift detection 3. This model is compatible with various choices of implementation tools (one of which is a GitOps-style implementation), each with different trade-offs suitable to different environments
- 8. Concepts, mental models & (some) mythbusting
- 9. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod
- 10. Some concepts 1. Application repository/repositories a. “Owned” by development team b. Contains code and application config
- 11. Some concepts 1. Application repository/repositories a. “Owned” by development team b. Contains code and application config 2. Application config a. Meaningful only in the context of an application b. Linked to application version: if app is rolled back, setting should be rolled back c. Could be the same across all environments, or different per environment d. E.g. “login.message”, “background.color”, “session.timeout”
- 12. Some concepts 1. Application repository/repositories a. “Owned” by development team b. Contains code and application config 2. Application config a. Meaningful only in the context of an application b. Linked to application version: if app is rolled back, setting should be rolled back c. Could be the same across all environments, or different per environment d. E.g. “login.message”, “background.color”, “session.timeout” 3. Environment repository/repositories a. Contains environment config and record/”audit trail” of applied manifest sets b. Ownership depends on org culture, regulatory requirements etc. - could differ per env
- 13. Some concepts 1. Application repository/repositories a. “Owned” by development team b. Contains code and application config 2. Application config a. Meaningful only in the context of an application b. Linked to application version: if app is rolled back, setting should be rolled back c. Could be the same across all environments, or different per environment d. E.g. “login.message”, “background.color”, “session.timeout” 3. Environment repository/repositories a. Contains environment config and record/”audit trail” of applied manifest sets b. Ownership depends on org culture, regulatory requirements etc. - could differ per env 4. Environment config a. Meaningful only in the context of an environment b. Not linked to application version: if app(s) is/are rolled back, setting should be unchanged c. E.g. “ldap.url”
- 14. Some concepts (2) 1. Manifests are templates a. = need to be modified before they can be submitted to kubectl
- 15. Some concepts (2) 1. Manifests are templates a. = need to be modified before they can be submitted to kubectl 2. Hydrated manifests are completed a. = ready to be submitted to kubectl
- 16. Some concepts (2) 1. Manifests are templates a. = need to be modified before they can be submitted to kubectl 2. Hydrated manifests are completed a. = ready to be submitted to kubectl 3. Manifest set = multiple manifests
- 17. Some concepts (2) 1. Manifests are templates a. = need to be modified before they can be submitted to kubectl 2. Hydrated manifests are completed a. = ready to be submitted to kubectl 3. Manifest set = multiple manifests 4. Candidate = before submitting to kubectl a. = can’t be sure it’s even valid
- 18. Some concepts (2) 1. Manifests are templates a. = need to be modified before they can be submitted to kubectl 2. Hydrated manifests are completed a. = ready to be submitted to kubectl 3. Manifest set = multiple manifests 4. Candidate = before submitting to kubectl a. = can’t be sure it’s even valid 5. Applied = after submitting to kubectl a. = can be sure that it was accepted by Kubernetes at some point in time
- 19. Some myths 1. “The manifests in my app repository are my source of truth” a. Manifests == templates b. Applied manifests are closer, but are only an attempt to make something happen c. Record of intent != record of achieved state 2. “Every state of my cluster should be represented in my repository” a. Kubernetes cluster will auto-manage and modify resources! 3. “Kubernetes already handles deployments” a. might handle some proportion, but deployment is an 80/20 problem b. multi cluster, exponential rollout, explicit traffic shaping 4. “Helm is a deployment tool” a. Helm is a package manager b. Distribution != deployment
- 20. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod To be thought of as templates! “Known good” (i.e. successfully submitted at some point in time) states of the environments
- 21. 4 related processes 1. Application code/config update 2. Environment config update 3. Environment restore 4. Environment config drift
- 22. Application code or settings change
- 23. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 1. Trigger on change
- 24. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 2. Hydrate candidate for dev Candidate hydrated manifest set dev
- 25. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 2a. If review and approval needed, two- and optionally three-way diff Candidate hydrated manifest set dev
- 26. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool Candidate hydrated manifest set dev 3. Attempt to apply candidate hydrated set to K8s cluster/namespace/etc. for dev
- 27. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool Candidate hydrated manifest set dev 4. If successful, combine with previous applied manifest set for dev and add to environment repository as current applied manifest set
- 28. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 5. Hydrate candidate for test Candidate hydrated manifest set dev Candidate hydrated manifest set test
- 29. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool etc. for prod Candidate hydrated manifest set dev Candidate hydrated manifest set test
- 31. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 1. Trigger on change
- 32. Process tool Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod 2. Hydrate candidate for dev Candidate hydrated manifest set dev
- 33. Process tool Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Candidate hydrated manifest set dev 2a. If review and approval needed, two- and optionally three-way diff
- 34. Process tool Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Candidate hydrated manifest set dev 3. Attempt to apply candidate hydrated set to K8s cluster/namespace/etc. for dev
- 35. Process tool Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Candidate hydrated manifest set dev 4. If successful, combine with previous applied manifest set for dev and add to environment repository as current applied manifest set
- 36. Process tool Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Candidate hydrated manifest set dev Done
- 38. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 1. Trigger manually with unique ID of prior applied manifest set
- 39. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 2. Retrieve using unique ID Candidate hydrated manifest set test
- 40. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool Candidate hydrated manifest set test 2a. If review and approval needed, two- and optionally three-way diff
- 41. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool Candidate hydrated manifest set test 3. Attempt to apply candidate hydrated set to K8s cluster/namespace/etc. for test
- 42. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool Candidate hydrated manifest set test 4. If successful, add to environment repository as new applied manifest set for test
- 43. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool Candidate hydrated manifest set test Done
- 45. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 1. Compare and alert on meaningful diffs
- 46. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 2. Optionally (to be considered with care), update K8s to match applied manifest set
- 47. Environment repository Application repository Code App settings App settings for dev App settings for test App settings for prod Env settings for dev Env settings for test Env settings for prod K8s dev K8s test K8s prod Applied manifest set dev Applied manifest set test Applied manifest set prod Process tool 3. Optionally (to be considered with even more care), create new applied manifest set on diff detection
- 49. App code/config change 1. Code change -> build 2. Triggers “deploy to staging” pipeline a. Hydrates manifests relevant to staging environment b. Apply to staging cluster c. Store results in blobstore 3. Optionally trigger “deploy to production” pipeline a. Bake manifests relevant to production environment b. Diff options: production vs. intent vs. repo c. Apply to 3 production clusters
- 55. Env config change 1. Versioned vs. unversioned config maps a. Impact on rollback 2. Example rollout pipeline
- 57. Env rollback 1. Parameterized 2. Different rollback options a. app(s) vs. controller vs. everything 3. Env config a. unversioned configmaps
- 60. 1. Application repository = SCM repository 2. Environment repositories = one per app per environment 3. Automation = pull request, tag & merge SCM triggers + polling/watching Env repositories contain hydrated manifest set before application. Need some separate process (labels?) to indicate which commits were actually successfully applied. GitOps
- 61. 1. Application repository = SCM repository 2. Environment repositories for applied manifest sets = Helm repository 3. Automation = pull request, tag & merge SCM triggers + GPO calling Helm Applied manifest sets are not stored in hydrated form - depending on behaviour of templatizer in recovery path General-purpose orchestrator + Helm
- 62. “Manual” ● Useful for understanding the pattern ● Application repository: folder on file system ● Environment repository: folder on file system with one manifest template file per env ● Subfolder “applied” with hydrated env config and file per app ● “Automation” = cat … | kubectl apply
- 63. “The delete problem” ● If you remove some configuration in your source code repository, what does that mean? ○ “I don’t care about this anymore”? ○ “I want this to be removed”? ● Turning line removals into delete actions is hard ○ Dangling resources that are not actually dangling can cause nasty recovery issues ● “Not present == should not exist” is tricky if there are multiple sources of input for a Kubernetes cluster (e.g. different app repos)
- 64. Polling/watching vs. direct invocation ● Direct invocation: orchestrator calls Kubernetes API explicitly before storing applied manifest set ○ Makes more advanced orchestration possible (canary, gradual exponential rollouts) ○ Delete problem ambiguous ● Polling/watching: an external process monitors a repository and applies the state it finds there to a target asynchronously ○ Potential solution to the delete problem if only one source of truth ○ Hydrated manifest set needs to be stored before it’s clear whether it can even happen
- 65. Wrapping up
- 66. 3 Main Takeaways 1. Code and config deployments in a Kubernetes environment are not fundamentally different from other “as code” scenarios
- 67. 3 Main Takeaways 1. Code and config deployments in a Kubernetes environment are not fundamentally different from other “as code” scenarios 2. There is a mental model for thinking about code and config in a Kubernetes environment that enables four important CI/CD patterns: app code/config rollout, env config rollout, env restore and env drift detection
- 68. 3 Main Takeaways 1. Code and config deployments in a Kubernetes environment are not fundamentally different from other “as code” scenarios 2. There is a mental model for thinking about code and config in a Kubernetes environment that enables four important CI/CD patterns: app code/config rollout, env config rollout, env restore and env drift detection 3. This model is compatible with various choices of implementation tools (one of which is a GitOps-style implementation), each with different trade-offs suitable to different environments
- 69. Get in touch! Slack Forum Diagram* More detailed notes* *Please request access!
- 70. Questions?