A sane helm chart workflow
Kubernetes ·Intro
Helm is a templating and deployment mechanism for kubernetes resources which is endorsed by the kubernetes core team. There are a couple of other possibilities to template kubernetes resources, but using helm charts facilitates finding other developers familiar with the technology.
This blog post describes the helm chart workflow which worked best for us.
You find a repo with a basic helm charts, tests and a simple build file here: https://github.com/ecodia/helm-chart-example
Basic Principles
- one repo per chart
- usage of a helm chart repository (e.g. chartmuseum)
- automated jenkins build and push into repository
- unittests automated run during jenkins tests
- (optional) roboll/helmfile as an umbrella chart wrapper
- automated jenkins deployments of helmfile in different environments
One repo per chart
While even the official helm charts are managed in one large repo our experience was, that this makes handling change on single charts much more difficult and gives repo editors much more power to destroy stuff than is usually desired.
We wanted to let the developers edit and ship their helm charts themselves, devops style. With having a separate repo per chart, this was easy to coordinate and no team could destroy other teams’ charts by accident.
Helm charts should be treated like any other codebase and the reasons to separate the repos are the same like for all codebases. Decoupling and better control. The additional overhead of managing the release-cycles of all the repos is leveraged by the build automation.
Usage of a chart repository (e.g. chartmuseum)
Just like a docker-registry or maven repository the helm repository allows to easily and reliable manage versioned builds of the helm charts. This allows a clean release process and a clean hand over to the deployment pipeline. Chart repos can be synced between environments (by scripts).
Use docker to run a local chartmuseum for tests or to run it in production. Hardware requirements are minimal.
docker run --rm -it \
-p 8080:8080 \
-e DEBUG=1 \
-e BASIC_AUTH_USER=chartadmin \
-e BASIC_AUTH_PASS=changeme \
-e STORAGE=local \
-e STORAGE_LOCAL_ROOTDIR=/charts \
-v $(pwd)/charts:/charts \
--user 1000:1000 \
chartmuseum/chartmuseum:latest
Setup helm with all plugins
We use the following plugins when working with charts.
- helm-unittest : this awesome plugin allows to write simple yaml spec tests
- helm-template : dry run your helm files, nice for debugging
- helm-push : push charts to chartmuseum
- helm-diff : a helm plugin that shows a diff explaing what a helm upgrade would change
helm init --client-only
helm plugin install https://github.com/lrills/helm-unittest
helm plugin install https://github.com/technosophos/helm-template
helm plugin install https://github.com/chartmuseum/helm-push
helm plugin install https://github.com/databus23/helm-diff
Write some tests
Even only having some basic tests which show differences in the resulting yaml structure are a huge improvements against having no tests at all. The tests also allow to test the outcome with different set values, which is especially useful to test “on/off” switches ( `` ).
You find more details on the helm unittest github page: https://github.com/lrills/helm-unittest
And more examples in our demo repo: https://github.com/ecodia/helm-chart-example/tree/master/example-application/tests
A simple example
This simple example checks that the deployment resource exists after rendering the
template and that the spec
subpath of the yaml file matches the previously taken
snapshot. Remember to do a git add tests
after running helm unittest the first time,
so the snapshots are added to the gitrepo.
When you then run helm unittest .
again after some changes to the templates you will
see the tests fail and you see the changes in the rendered output.
If the changes were made intentionally you can run helm unittest -u
to update the snapshots.
Then again remember to commit the snapshots together with your chart changes.
suite: test deployment
templates:
- deployment.yaml
tests:
- it: should be of type Deployment
asserts:
- isKind:
of: Deployment
- it: manifest should match snapshot
set:
image:
repository: test-docker-registry
tag: 1.2.3-TEST
asserts:
- matchSnapshot:
path: spec
Publish the chart to chartmuseum
Assuming you want to push the chart to chartmuseum, first add your helm repository with the credentials, then push the chart to the registry:
helm repo add --username chartadmin --password changeme localtest http://localhost:8080
helm push . localtest
You can see all available chart versions in the yaml file:
curl --user chartadmin:changeme http://localhost:8080/index.yaml
Use helmfile as an umbrella chart
A helm umbrella chart is a helm chart containing only dependencies to other charts (via the
requirements.yaml
), so these can deployed as one complete deployment.
roboll/helmfile is a separate tool which allows to deploy multiple different helm charts and also adds a ton of useful features (like templating in the values files).
https://github.com/roboll/helmfile
Automated deployment
Build a jenkins job to deploy the chart to different environments.
You can deploy your charts directly via helm upgrade --install
after the chart build
and supply environment specific information by different values files which you provide
via the -f dev-values.yaml
option of the helm command.
You can also use helmfile which is suited very well for this task.
Just create a dropdown jenkins parameter with your environments (e.g. “dev”, “qa”, “test”) which match environments in your helmfile.
Tipps and Tricks
- If you are using “latest” docker-images new versions will not be automatically migrated by helm/kubernetes as it cannot detect the change. So always use versioned docker images. You can version them with the current date-time or a build counter e.g. myimage:20190202123 if you do not want to give them semantic versions.
- always use the
--concurrency=3
parameter while doing a helmfile sync, otherwise you might bring down your k8s api with the many parallel requests - use a wrapper shell script around standard helm to delete, as helmfile doesn’t support the –concurrency option for deletes