v7.14.0 - The First ARM Build of Concourse

v7.14.0 is out and with it also comes the first ARM build of Concourse! There were a lot of behind the scenes changes that were required to make this happen, so let’s dive into everything that’s in this release.

🫆Pipeline Identity Tokens

I figure we’ll start with the new features, because that’s always fun. #9035 added a whole new var_source called “Identity Tokens”. These are JWT’s that Concourse can generate for you and that you can then use to authenticate to third-party systems that support “identity federation”, such as Vault, AWS, and Azure.

@dbaumgarten did everything here: wrote the RFC, made the PR, AND wrote a comperhensive set of docs with examples on how to use it. Huge thank you to him for bringing this feature to the community. I think a lot of users will find it useful and help them migrate away from using static credentials.

🦾 The Road to ARM

This has been a longgggggg time coming. There was a community fork of Concourse out there for a while, specifically for building an ARM version of Concourse. I remember when Ciro, a co-worker from Pivotal, did a little exploration running Concourse on his Raspberry Pi. He showed it was possible, though the road wasn’t completely smooth.

Thankfully, the container ecosystem has continued to develop and mature these last few years while Concourse was in limbo. More workloads are running on ARM now and as a result more of our tools and libraries just work when trying to do cross-compilation or ARM stuff. The big win for us was Docker making multi-platform builds a thing. This saved us from having to manually build an ARM Concourse worker, like Ciro did, just so we could build ARM images of Concourse and all the resource-types that we ship with Concourse.

At this point, all we needed to do was update our pipelines to support building and releasing ARM versions of everything!

The journey went like this:

  • Update all 12 base resource-types to use Wolfi as their base image because the previous base image did not have an ARM variant. The concourse/concourse image also uses Wolfi as its base image now.
  • Update the OCI Build task and Registry Image resource to better support multi-platform images and workflows.
  • Update the pipeline used to build, test, and release the resource-types, and release ARM versions of all the resource-types.
  • Update the main Concourse pipeline and the release pipeline to build and test the ARM variant of Concourse.
  • Added a /download-fly page to support all variants of fly, replacing the three static download links previously located in the footer

Each of those steps took many hours to complete. I had to touch every repository we own and got to do a little clean-up everywhere.

One nice side-effect of moving to Wolfi for the base image is that the size of all the container images we produce dropped. Some by a lot, some by just a little. For example, the Git resource went from 218MB to 37MB, a massive drop in size! Some other resources dropped by only a few MB’s. The size drops weren’t 100% due to Wolfi. I took some extra time to ensure we were only adding what was necessary for each resource-type to function.

Collectively, this results in us shipping a much smaller concourse/concourse image. v7.14.0 clocks in at 928MB (x86_64) and 883MB (ARM), down from 1.41GB (x86_64). That’s a 34% drop in size for the x86_64 image and 37% for the ARM image. The ARM-based images are also always smaller, so you save a bit more disk space if you go for a fully ARM-based Concourse deployment.

Now there is finally an ARM version of Concourse and fly, that folks can run on their Raspberry Pi’s, M-Series macs, and ARM cloud servers. I’m excited to see what Concourse will end up running on now 🎉

☁️ Cloud Foundry Foundation

As a project member of the Cloud Foundry Foundation (CFF), Concourse has had two tasks assigned to it.

  1. Reduce project cost: We’ve reduced project costs by 40%, leaving us in a decent state. There’s some smaller things we can tackle, but urgency from CFF is gone.
  2. Running a shared Concourse cluster for CFF member projects. This is related to the CFF’s wider goal of reducing costs across all projects. This is something myself and Derek plan to work on over the next few months.

🧭 What’s Next

My goal with Concourse right now is to continue to refine and improve what we currently have. There are plenty of little bugs littered throughout the code base that I want to resolve to help make Concourse feel even more stable and reliable than it currently is.

I try to keep this GitHub project board up to date with what I plan to work on. Folks are free to look at the board, and if it isn’t in the “In Progress” column, feel free to pick up the issue and work on it.

I’ll being put a lot of my attention to the breaking changes stuff planned for v8. See this discussion post for details and leave any thoughts or comments there.

That’s all I have for everyone now. See everyone over on Discord or GitHub Discussions. Enjoy the new release!