A sane helm chart workflowKubernetes ·
Helm is a templating and deployment mechanism for kubernetes resources which is endorsed by the kubernetes core team. There are a couple of other possibilities to template kubernetes resources, but using helm charts facilitates finding other developers familiar with the technology.
This blog post describes the helm chart workflow which worked best for us.
You find a repo with a basic helm charts, tests and a simple build file here: https://github.com/ecodia/helm-chart-example
- one repo per chart
- usage of a helm chart repository (e.g. chartmuseum)
- automated jenkins build and push into repository
- unittests automated run during jenkins tests
- (optional) roboll/helmfile as an umbrella chart wrapper
- automated jenkins deployments of helmfile in different environments
One repo per chart
While even the official helm charts are managed in one large repo our experience was, that this makes handling change on single charts much more difficult and gives repo editors much more power to destroy stuff than is usually desired.
We wanted to let the developers edit and ship their helm charts themselves, devops style. With having a separate repo per chart, this was easy to coordinate and no team could destroy other teams’ charts by accident.
Helm charts should be treated like any other codebase and the reasons to separate the repos are the same like for all codebases. Decoupling and better control. The additional overhead of managing the release-cycles of all the repos is leveraged by the build automation.
Usage of a chart repository (e.g. chartmuseum)
Just like a docker-registry or maven repository the helm repository allows to easily and reliable manage versioned builds of the helm charts. This allows a clean release process and a clean hand over to the deployment pipeline. Chart repos can be synced between environments (by scripts).
Use docker to run a local chartmuseum for tests or to run it in production. Hardware requirements are minimal.
docker run --rm -it \ -p 8080:8080 \ -e DEBUG=1 \ -e BASIC_AUTH_USER=chartadmin \ -e BASIC_AUTH_PASS=changeme \ -e STORAGE=local \ -e STORAGE_LOCAL_ROOTDIR=/charts \ -v $(pwd)/charts:/charts \ --user 1000:1000 \ chartmuseum/chartmuseum:latest
Setup helm with all plugins
We use the following plugins when working with charts.
- helm-unittest : this awesome plugin allows to write simple yaml spec tests
- helm-template : dry run your helm files, nice for debugging
- helm-push : push charts to chartmuseum
- helm-diff : a helm plugin that shows a diff explaing what a helm upgrade would change
helm init --client-only helm plugin install https://github.com/lrills/helm-unittest helm plugin install https://github.com/technosophos/helm-template helm plugin install https://github.com/chartmuseum/helm-push helm plugin install https://github.com/databus23/helm-diff
Write some tests
Even only having some basic tests which show differences in the resulting yaml structure are a huge improvements against having no tests at all. The tests also allow to test the outcome with different set values, which is especially useful to test “on/off” switches ( `` ).
You find more details on the helm unittest github page: https://github.com/lrills/helm-unittest
And more examples in our demo repo: https://github.com/ecodia/helm-chart-example/tree/master/example-application/tests
A simple example
This simple example checks that the deployment resource exists after rendering the
template and that the
spec subpath of the yaml file matches the previously taken
snapshot. Remember to do a
git add tests after running helm unittest the first time,
so the snapshots are added to the gitrepo.
When you then run
helm unittest . again after some changes to the templates you will
see the tests fail and you see the changes in the rendered output.
If the changes were made intentionally you can run
helm unittest -u to update the snapshots.
Then again remember to commit the snapshots together with your chart changes.
suite: test deployment templates: - deployment.yaml tests: - it: should be of type Deployment asserts: - isKind: of: Deployment - it: manifest should match snapshot set: image: repository: test-docker-registry tag: 1.2.3-TEST asserts: - matchSnapshot: path: spec
Publish the chart to chartmuseum
Assuming you want to push the chart to chartmuseum, first add your helm repository with the credentials, then push the chart to the registry:
helm repo add --username chartadmin --password changeme localtest http://localhost:8080 helm push . localtest
You can see all available chart versions in the yaml file:
curl --user chartadmin:changeme http://localhost:8080/index.yaml
Use helmfile as an umbrella chart
A helm umbrella chart is a helm chart containing only dependencies to other charts (via the
requirements.yaml), so these can deployed as one complete deployment.
roboll/helmfile is a separate tool which allows to deploy multiple different helm charts and also adds a ton of useful features (like templating in the values files).
Build a jenkins job to deploy the chart to different environments.
You can deploy your charts directly via
helm upgrade --install after the chart build
and supply environment specific information by different values files which you provide
-f dev-values.yaml option of the helm command.
You can also use helmfile which is suited very well for this task.
Just create a dropdown jenkins parameter with your environments (e.g. “dev”, “qa”, “test”) which match environments in your helmfile.
Tipps and Tricks
- If you are using “latest” docker-images new versions will not be automatically migrated by helm/kubernetes as it cannot detect the change. So always use versioned docker images. You can version them with the current date-time or a build counter e.g. myimage:20190202123 if you do not want to give them semantic versions.
- always use the
--concurrency=3parameter while doing a helmfile sync, otherwise you might bring down your k8s api with the many parallel requests
- use a wrapper shell script around standard helm to delete, as helmfile doesn’t support the –concurrency option for deletes