So you want to run Docker in Concourse? Well this is the guide for you!
Let’ clarify what it is we want to do. We want to be able to run docker-compose inside a task in Concourse to bring up our application along side some other services (i.e. Redis, Postgres, MySQL, etc.).
Thankfully this challenge has been solved by the community! There are a few “Docker-in-Docker” images designed to run in Concourse that are maintained by the community.
In this blog post we’re going to cover how to use git and Concourse to automatically set, update, and archive your pipelines using the set_pipeline step. No longer will you need to use fly set-pipeline to update any of your pipelines!
For consistency we will refer to the pipeline that contains all the set_pipeline steps as the parent pipeline. The pipelines created by the set_pipeline steps will be called child pipelines.
Happy Friday! This one’s brief.
Merged RFCs RFC #37: prototypes has landed! …but it probably could have use more detail regarding the run step, which is the only immediately actionable part of it. š¤ I’ll draft another RFC for that; #37 mainly covered the protocol. RFC #38: resource prototypes is in! Its associated issue for implementation is #5870. RFCs to merge n/a - taking a breather for this round-up to focus on the below RFCs and “reset” the 2 week merge window so I can start publishing these posts earlier in the week.
With the four(!) RFCs from the last round-up now merged, it’s time to move on to the next RFC milestone: Prototypes!
Merged RFCs š RFC #31: set_pipeline step RFC #40: valid identifiers RFC #39: var sources RFC #27: var steps RFCs ready to merge š¤ RFC #37: Prototypes is finally ready to go! For (much) further reading, check out the Re-inventing resource types blog post. The importance of this RFC really cannot be overstated; it will be the most significant change to Concourse since its creation.
In this blog post we are going to show how to build and publish container images using the oci-build task and registry-image resource. This post assumes you understand how to build container images with Dockerfile’s and publish to Docker Hub or another image registry using the docker cli.
If you just want to see the pipeline, scroll to the bottom or click here. What follows is a detailed explanation of what each part of the pipeline does.
First off: sorry, I immediately failed to keep my target pace for these. š I got wrapped up in a deadline, and since I alternate weeks between engineering and community duties like this post, when I miss a week for RFC updates the 2-week interval can quickly turn into 4 or 5.
Owing to the missed round-up, and in hopes of burning through the backlog more quickly so that interested contributors may volunteer for merged RFCs, I’m going to expand the scope of this post to include more RFCs than the last one - primarily by proposing that we merge ones that are nearly certain for the v10 roadmap.
Understanding how task inputs and outputs work in Concourse can be a little confusing initially. This post will walk you through a few example pipelines to show you how inputs and outputs work within a single Concourse job. By the end you should understand how inputs and outputs work within the context of a single job.
Let’s define some jargon first.
step : A step is a container running code within the context of a Concourse job.
A little over a month ago, the Concourse team sent a survey out to the community. The purpose of this survey was to gain insight into our users as well as measure our year-over-year growth. In the process of learning about how you all deploy and manage Concourse, we also received tons of great feedback about what’s working well and what needs work in order to make Concourse even better. Weāre excited to share our findings!
Howdy, and welcome to our first RFC round-up! š¤
For those unaware, Concourse RFCs are a process for proposing and collaborating on improvements to core Concourse functionality, including pipeline behavior, new step types, new operator capabilities, etc.
In short, RFCs are where all the cool new stuff is planned. š
My goal is to provide an update at least every few weeks on the status of RFCs and shepherd them through the process via blog posts like this one.
Hasta la vista, stale bot.
tl;dr: We’ve been granted access to GitHub’s beta Discussions feature! š Discussions on the concourse repo will be used for questions and technical support. Discussions on the rfcs repo will be for incubating ideas for new workflows , which eventually turn into Pull Requests (also on the rfcs repo). From here on, Issues on the concourse repo are exclusively for project backlog and bug reports - i.
In March 2020, countless companies made a shift to have their employees work from home. For remote staff getting work done can be challenging enough, but staying connected to your team and company culture can be even more challenging. On the Concourse team, weāre working hard to keep our product development running smoothly with some additions to our tech stack and day to day workflow.
Concourse team composition At VMware (formerly Pivotal Software), there are fourteen engineers and five product folks working on theConcourse CI project full time.
Help shape the future of Concourse Since Concourse CI was created, thousands of users worldwide have helped the project by opening issues, committing code, and providing feedback to the team that develops the product. This community involvement is priceless - thank you, Concourse community! š
One of the ways the Concourse team collects feedback is through our annual Community Survey. This lets us gather crucial information about how users deploy Concourse, how different use cases scale, and various configuration patterns.
Photo: National Parks Service
The idea to build a dedicated resource types catalog has been long-discussed. Weād like to announce that the idea has come to fruition: the new Concourse resource types catalog is wheels up!
The catalog lists Concourse resource types that have recently been submitted to the resource types GitHub repo. Originally, resource types were listed on a GitHub wiki page. While the wiki page listed resource types, it didnāt provide much information about each resource.
Before the paint completely dries on the v10 roadmap, there is one last big unknown I want to explore in case it brings more clarity to our direction: generic tasks.
Resource types are a great way to share tools and integrations for others to use in their pipelines. Unfortunately, they’re basically the only way, and because resources are a very opinionated concept, the resource type interface is not always a good fit.
Phew, it’s been a while. I got lots of info to cover so let’s just get right into it
Concourse OSS Growth As some of you may know, the Concourse team switched over to a PR-based workflow at the beginning of the year. This change is in line with our objectives of being open and transparent with our community of contributors. Plus, its just good thing to do because that’s what most OSS projects do.
A long-term roadmap for the core design of Concourse, a general-purpose CI/CD tool.
Accompanying slides. Recommended viewing: episode 1 of Yu-Gi-Oh.
Concourse’s design philosophy is to be expressive, versatile, and safe while limited to a handful of simple, proven concepts. The design of these concepts should make good practices feel intuitive and bad practices feel uncomfortable.
Coming up with these designs can be very challenging. There are many different workflows and patterns across the software industry, and they each have to be deeply understood in order to know what the good and bad practices are.
How does the Concourse team go about solving the problem of pipeline monitoring? With the growing popularity of Concourse, we noticed that our development teams wanted to observe and monitor multiple pipelines simultaneously. This behaviour wasnāt limited to just Pivotal engineering teams; in fact, it was even more prevalent amongst our Open Source Community. Our users currently solve this by cramming multiple browser windows into TV their monitor view or they use the Concourse Summary (aka Crystal) by David Goddard of the Pivotal Buildpacks team.
The Concourse team had the opportunity to visit some Concourse users out in Montreal last week. We had a blast meeting everyone, including some folks from the Concourse OSS community. Thanks again for hosting us!
Iāll also be in Kansas City for two days next week to meet some other Concourse users as well, so give me a tap onTwitter or Discord (username jama) if you wanna meet up.
Parallel Input Streaming In addition to the work on Algorithm improvements from the Core track, the Runtime track tested out their new work on Parallel Input Streaming.
…and weāre back! Apologies for the lack of updates lately. Iāve just come back from some time off and work travel has taken up a lot of my time. Iām back in Toronto now so letās get back into it.
Release Engineering & Concourse 5.3.0 In the past, we relied a lot on Alex Suraci to handle a lot of our release engineering work. Release Engineering is incredibly important and valuable work for the Concourse team, but it can also very time consuming.
In case you missed it, weāve made some tweaks to the structure of the website. Iām happy to report that Alex Suraci drastically improved our site-wide search. This resolves #181 and weāre all the better for it!
Second, youāll notice that a lot of the community related comment that was on our homepage has now been moved to our Concourse GitHub Wiki. We hope this change will make contributor and community specific content more discoverable and more maintainable over time.
Sorry for missing the update last week. I was travelling out to the Bay area to attend the ConcourseCI Bay Area User Group. For those who missed it, you can find a recording of the event here. On to the update.
In case you missed it, Concourse 5.1.0 is out! Itās got icons on resources, better garbage collection, on_error on pipelines, and much more! As usual, you can read the full list of new features here.
Greetings from sunny Philadelphia! The team was there for the Cloud Foundry 2019 NA Summit for a few days; talking to Concourse users and attending talks. Recorded videos of the talks should be uploaded soon; so Iāll point you to the interesting Concourse-related ones next week.
On to the update.
For Active Discussion Please take some time to review and comment on the latest Concourse + k8s Runtime RFC Regarding the runtime, thereās been an active conversation around better build scheduling and load distribution.
Phew, itās been a while since I last wrote an update. For some background behind why I slowed down, hop on over to this thread on our forms: āWhat would you like to see on our blogā.
That said, I do have a lot of interesting updates to share, so letās get started
Concourse 5.0.0 In case you missed it, Concourse 5.0.0 and 5.0.1 came out a few weeks ago in March.
With the release of Concourse 5.0.0 this week I thought it would be a good time to review the evolving implementation of authorization in Concourse. Iāll also be covering some helpful debugging information for you to consider when configuring authorization in your own Concourse instance.
Read the Docs The revised Concourse Auth & Teams docsis a great place to start when diving into Concourse 5.0.0. The docs will cover important steps around provider configuration and team configuration for your cluster.
Its been a week since we switched over to the PR workflow and so far its been great! Weāre still working through some of the kinks with this process so please bear with us while we continue to burn down through the list of open PRs!
And nowā¦on to the update! I might have missed a few issues while Iām still getting used to our new workflow. Completed issues now appear as closed PRs in concourse/concourse
Some updates worth bringing up this week. As I had mentioned last week we began to do a re-organization of projects and issues in our concourse/concourse repo; you can read more about it on our wiki page here. With that said, you can find the issues and PRs that are slated for Concourse 5.0.0ās release in our 5.0.0 Milestones. If youād like to help us with documentation, weāve started a new branch in the docs repo under v5.
ā¦and weāre back! The teamās been pretty quiet over the past few weeks due to vacations and holidays. This was our first week back at full strength so weāve got some interesting updates for yaāll
How are issues managed? This is an issue that comes up a lot in our open source community, and Alex Suraci has taken some time to clean up our issues backlog and add in some bots.
Or, āwhy we havenāt shipped any new features since September.ā You may have noticed that our release cadence has slowed down significantly in the past few months. The bad news is we probably wonāt get a release out this year (mainly due to end-of-year vacations and slowing down in general), but the good news is the next release is hugeāāābig enough to bump us to v5.0āāāand itās just about ready.
2018 has been an action-packed year for us. We saw a major release (Concourse 4.0.0) with a lot of new features: new auth connectors and users, dashboard, distributed GC and other runtime improvements. At the same time our team grew from 3 engineering pairs at the start of 2018 to 8 engineering pairs and an additional PM (š Scott Foerster) working on Concourse OSS and supporting Concourse for PCF.
Weāre nearing the end on some UX refactoring work and finished off the issue regarding container scheduling. Between those improvements and the global resource caching, weāre hoping to see a lot of efficiency improvements in 5.0
That said, weāve decided that we need to perform some additional performance and load testing on Concourse 5.0 before we cut the release. And with the holidays coming up, its increasingly unlikely that weāll be able to push Concourse 5.
As I mentioned last week Iāve been doing story acceptance in our dev environments for the upcoming RBAC feature. The teamās been working through some of the new issues that come out of that to give some final polish on to the release.
Something that I havenāt talked too much about in the past weeks is our work on the Concourse k8s Helm chart. If you pull up some of the PRs under stable/concourse, youāll see that weāve been proposing some changes to the chart.
One of the big themes for Concourse in 2018 has been Users, multiple auth connectors, and role-based access control (aka RBAC). With RBAC in the final phases of development, I wanted to give you a preview of some of the functionality that you can expect in our upcoming release; Concourse 5.0
Admins, Owners, Members and Viewers Concourse 5.0 will come with 4 roles: Concourse Admin, Team Owner, Team Member, and Team Viewer.
It was a relatively light week this week due to some vacations. I did, however, get a chance to do some acceptance work on our upcoming feature for role-based access control in Concourse. You can read more about how thatāll work in our feature preview post.
On to the update:
API
Our investigation into the API continues and branches out into more areas of the codebase. If you havenāt already, make sure to check out the two related RFCS: https://github.
Right off the bat Iād like to give a shoutout to Jamie Klassen and his new post about the upcoming feature for pinning resources. You can check it out the new post here:https://medium.com/concourse-ci/resource-page-explained-eb99cf256fb5
I also wanted to mention that the Github Pull Request that was maintained by JT Archie (https://github.com/jtarchie/github-pullrequest-resource) has been officially deprecated.
The official docs for the resource types no longer point to jtarchie/pr for the PR resource.
As a part of our refactor of the prod pipeline, Alex Suraci cleaned up and refactored parts of the TSA to better support draining and rebalancing #2748. The numbers are looking really good!
On to the update:
API Weāre deep into investigations around our API documentation and management strategy. Our current investigation work is captured in #2739 but the original request comes from #1122 Core SPACCEEEE #2651 UX Continuing our work on supporting pinning of versions on resources from the UI.
This week the team got together to discuss the initial groundwork and investigations required to publish and maintain a supported API. If youāve built any tools against our API and have feedback for us, please let us know by commenting on the original issue #1122.
In another interesting update, the PivNet team has published an update to the pivnet-resource so āyou no longer need to specify the access key, secret access key, bucket and region for creating releases.
We finished our first implementation of Role Based Access Control (RBAC) this week! You can look forward to this change in our next release of Concourse.
Speaking of which, the next release of Concourse is currently blocked while we try to re-build our new release pipelines. Along with our move to the mono-repo, weāre focusing even more on making the binary distribution of Concourse the first-class distribution of Concourse. This means that youāll get everything you need for Concourse packaged into one nifty tgz!
The results of the Concourse 2018 Community survey is out! Thanks to everyone who took the time to fill it out; and to Scott Foerster and Lindsay Auchinachie for sifting through the data.
It was a relatively short week for us due to Thanksgiving celebrations, but hereās our update:
UX
Continued our rampage in fixing fly issues: #259, #267, #1038, #1062, #248 I also wanted to add that weāre trying to keep all issues under concourse/concourse.
Alex Suraci is still tackling the chores on our One Big Repo issue #2534. Specifically, Alex is re-writing a new pipeline (aka concourse) for our mono-repo structure so we can unblock ourselves from releasing updates.
In other news, Concourse engineer Saman Alvi wrote up a short article on her experience pairing with a product designer during a discovery into the PivNet resource; check it out: Design & Dev Pairing: What we learned during a one week technical discovery.
The Concourse team went out to Washington D.C. this week to attend Spring One Platform 2018. Thanks to all the Concourse fans who stopped by to say hi, we really enjoyed meeting yaāll. All of the talks were recorded and should be uploaded to the SpringDeveloper YouTube channel in the coming weeks. Some of the interesting talks to check out are:
Extreme Pipelines Zero to Multicloud and Spinnaker and the Distributed Monorepo …and of course Draupnir: A story about Managing Concourse in the Enterprise And now, on to the update:
Concourse 4.2.0 and Concourse 4.2.1 were released earlier this week. Thereās a lot of great fixes and features in this new release, so please upgrade now!
Reminder that The Great Project Restructuring of 2018 is now underway. Youāll notice that all our submodules (e.g. ATC, TSA fly)are now all under the root level of the concourse/concourse repo. Its cleaner.
Youāll also notice that the BOSH spec has moved from its usual place.
Following up from a discussion on our forums Scott Foerster has been looking at different options for selling Concourse swag online. Do you want Concourse leggings? or maybe a limited edition @vito pls pillow! Let us know in the thread Concourse merchandising.
Please also take some time to fill out our 2018 Concourse Community survey. Your feedback is really valuable to us and the information you provide will help us plan the future of Concourse.
Apologies for the break from the usual update schedule; I wanted to get one last update out before I take some personal time, starting Fri. Aug 31 and coming back Sept 10. In my absence Scott Foerster and Alex Suraci will be writing the product update next week. The Concourse team will also be taking Monday, Sept 3rd off in observance of Labour day as well.
On to the updates:
Kubernetes As we continue our sporadic work on Kubernetes and its Helm chart, weāre also starting to expand our thinking to cover the runtime aspects of Concourse + Kubernetes. Weāve already prioritized the need to have Kubernetes as a supporting backend in addition to Garden, but what about the spiffy new developments in the Kubernetes world? Weāre hearing a lot about knative and knative services like build and eventing. Are there any kubernetes users whoād like to weigh in on the topic?
Going to switch things up this week and start with some interesting community news:
Weāve decided to restructure our repositories to make things more understandable and less scary for contributors. Alex Suraci has laid out a good explainer on why and how weāre going to start in our PSA: the Great Code Restructing of 2018 Lindsay Auchinachie wrote up a blog post describing some of the visual elements of the Concourse pipeline view in a blog post titled Concourse Pipeline UI Explained marco-m has been updating a āconcourse-in-a-boxā formula that comes with a s3-compatible-store and a Vault.
As I mentioned last week, this was a short week for us in Canada due to the Civic Holiday. We did, however, manage to work on some pretty cool stuff!
With the release of 4.0.0, weāve been shifting our new feature focus towards Operations and Runtime. Weāre intentionally slowing down on UX to focus on regressions and UI polish for existing screens.
On to the update:
Core
Continued our work on āpinningā a version of a resource across the pipeline.
As a Product Manger at Pivotal Iām often called on to help with our customerās Concourse-related issues. I recently spent some time hunting down an issue around suspiciously high volume usage on Concourse workers. It was an interesting problem that I wanted to share with the broader Concourse community.
Platform Management Pipelines One of the primary use case for Concourse within the Pivotal Cloud Foundry (PCF) ecosystem is to automate the toil of manual maintenance against the platform; specifically PAS and PKS.
With the launch of Concourse 4.0.0, weāve been monitoring our typical communication channelscarefully to watch out for any glaring new bugs. So far we seem to be safe from any crazy issues, but we have noticed that there has been some confusion in how to set the basic auth users in the new deployment method (see #2421 for details). Thanks everyone for your patience and working through these issues with us!
Iām happy to announce that we released Concourse 4.0.0 this week! This was a HUGE release with over 28 new features and fixes. Iād encourage you to read through the full list of changes on our Downloads page.
Why did this release warrant a bump in the major version? Well, if youāve been following along closely youāll know that we had just finished our new auth work in 4.0.0. Users are now central to the authentication flows, not teams.
This week, the Concourse team went out to Portland to attend OSCON 2018. Topher Bullock gave a great intro to Concourse in the Open Source track. We even met some of the Concourse fans in person!
In other news, weāve begun to sketch out what RBAC might look like in Concourse. Please check out #2389 when you have some time!
On to the update:
UX
Team has been working on adding drag and drop re-arranging for the dashboard in #2364 We also found a weird quirk with the new team creation flow, where you wonāt see your team if it was just created and has no pipelines.
Weāre going dark themed for Concourse 4.0.0! In addition to the users work, weāre promoting the Dashboard to the / level to take over the home page. Youāll also notice that we added pipeline play/pause capabilities to the dashboard, NEAT!
To keep things consistent, weāre also propagating our new design to the existing pipeline views. You can play around with this new nav structure on our own CI: https://ci.concourse-ci.org/
Since July 1st was the official day of Canadaās birth, the Concourse team enjoyed a long weekend with no work on Monday. We were, however, able to get quite a bit done during this short week.
A big win is that we added a k8s-testflight job to our official ci pipeline (check it out here); this will let us know in advance when we have broken the Concourse Kubernetes Helm Chart.
If youāve been following along with our Auth changes, youāll know that weāve been doing a lot of work behind the scenes to make the upgrade into this new world as seamless as possible. This week, we were able to do our first large-scale upgrade test against our Wings instance. The upgrade went well and we were able to find a few more areas of polish before we push this feature.
Itās been a busy week for myself and Topher Bullock. We spent some time in Boston meeting with some users operating large-scale Concourses. We learned a lot about the issues they were running into operating Concourse a scaleā¦and we ate a lot of Lobster!
On to the update:
UX:
Made some improvements to the build page in issue #1543 that weāre hoping to test soon on our internal Concourse. You can read into some more of the details in our comments.
This was a post-release week, so we spent a lot of time merging in new code from the Users track, fixing our pipelines, and working on some neglected issues. All in all a solid weekās worth of work! On to the update
UX
Changed the behaviour of the breadcrumb so that clicking on the pipeline name resets the pipeline view and group settings (#2258) Fixed a bug with the breadcrumb where it wouldnāt render whitespace correctly (#2267) Fixed a bug with team name overflowing on breadcrumbs (#2241) Fixed a UI bug on the navigation arrows (#2276) Added JSON stdout to Fly CLI (#952) We havenāt done work on this yet, but based on our observations and feedback from the community, weāre planning to push the dashboard up to / level.
Big release this week! After lots internal load testing on Wings we finally felt comfortable releasing Concourse 3.14.0. In addition to the new Distributed Garbage Collection, breadcrumbs, responsive groups, and Windows worker, we have 14 new features a whole bunch of bug fixes. But wait! Donāt download that one; get Concourse v3.14.1 instead.
A few other updates. First, be sure to check out my write up on How We Build Concourse.
Building on some of our previous posts on the Concourse team mechanicsĀ¹, I wanted to spend some time going over how we actually build Concourse.
Tracking Features and Bugs Concourse tracks all of its bugs, features and epics through GitHub Issues.
For items regarding the core functionality of Concourse itself, you can find the master issues list here: https://github.com/concourse/concourse/issues
For issues regarding the Concourse website and documentation, you can find the backlog here: https://github.
If youāve been experiencing āAw Snapā errors on Chrome with Concourse 3.13.0 or 3.12.0 we traced the root case to two lines of CSS. This seems to happen only on Chrome 67; so a temporary workaround is to switch over to Chrome canary or use Firefox/Safari/Edge. You can follow along in our discussion at GitHub issue #2236
Now, on to the update
Runtime
We were able to successfully test our distributed volume GC collection code on our Wings environment this week.
It was a short week for us here in Canada, but we had a few interesting updates:
We attempted to deploy our distributed GC changes to our internal environment āWingsā last Friday. Turns out that was an incredibly bad idea. The deployment failed horrifically and we had to roll back all our changes. Weāre still investigating why our code worked in our āprodā environment but failed when deployed onto Wings. Weāre tracking this work in issue #2202.
In case you missed it, Iād encourage you to check out some of the recent posts from Shashwathi Reddy on āMy first month on Concourseā and Joshua Winters regarding upcoming changes to our authentication; āOh, Authā. Weād love to hear your feedback!
Heads up: the Concourse team will be taking Monday May 21st off for Victoria Dayholiday.
And now, on to the update:
Core
Continued banging our heads against new auth connectors with Dex.
Hi folks,
Joshua Winters has spent a lot of time refactoring Concourse so that it can finally support Users. Weāre finally at a point where we can share some our work with you, so Iād really encourage you to check out his recent blog post Oh, Auth
Outside of that, weāve got a bunch of vacations going on this week, so itās been more of the same three tracks of work:
Iāve gotten some questions about Freedom Friday from some readers after last weekās update. Well it turns out thatTopher Bullock wrote a great article about it this week; you read up on it here: https://medium.com/concourse-ci/freedom-fridays-319204dea834
We also release Concourse v3.13.0 earlier this week. Make sure you check it out if you were hit by the accumulating logs issue introduced in v3.12.0.
On to the update:
Space Weāve been building out some of the frontend code for representing Spaces as part of #2131.
Well, that was fun! Topher Bullock absolutely killed it last week on the CF Summit 2018 main stage with his demo of the experimental Concourse ā¤ ļøK8s runtime project. We also had a great time talking to companies who were using Concourse to continuously do things in the cloud. One of my favorite talks was from Jason Immerman and Derek Van Assche from Zipcar (Concourse All the Things, All the Time); really inspirational stuff!
Concourse v3.11.0 came out today! Go get it: https://concourse-ci.org/download.html#v3110
On another note: Iāve noticed some interesting articles and guides come out on writing custom resources for Concourse. Thereās one on the Pivotal blog https://content.pivotal.io/blog/developing-a-custom-concourse-resource and another from fellow Medium writer Shin Myung Yoon (https://itnext.io/writing-a-custom-resource-for-concourse-detecting-pull-request-close-merge-events-e40468eb2a81)!
Topher Bullock and I will also be travelling to Boston next week for CF Summit 2018. Weāll be around to meet some Pivotal PCF customers and answer questions about Concourse.
If you havenāt done so already please check out Alex Suraciās recent update post on āA renewed focus & community changesā. It covers all the recent changes that weāve been making; starting with the new styling of the website, our new discussion forum, and our migration to Discord chat.
Specifically, weāve been getting some mixed feedback on the new format of the site. Some folks love it, other folks miss the highly visual styling of the old site.
Phew, weāve been busy for the past couple of months! Thereās a lot to give yāall an update on.
New website First off, check out our new website! Weāve completely redesigned it and redone how we organize the documentation, in hopes that itāll be much easier to find what youāre looking for.
We also hope that the new style, language, and tone will feel a bit more inclusive and humble. For example, we got rid of the āConcourse vs.
Hi folks,
Had an interesting week talking to customers about how we might improve their Concourse operations and deployments. More info on that soon!
On to the update:
UX
Fixed an issue with timestamps #2088 Core
Continued our refactoring of the API to support dex and users #1888 Runtime
Finished the issue around custom resources on tagged workers, it should work now #1371 Restricted the list of allowed TLS ciphers for more security checkboxing #1997 Design
Its been a first week in a long time where we were back to full strength and fully co-located. It was nice!
Oh, and Concourse v3.9.2 was released this week as well,check it out!
On to our update:
UX
Started to make our dashboard a bit more mobile friendly #1712 Started to tackle the problem where our dashboard holds too many open connections #1806 Core
Removed a dependancy to provide an external URL for fly execute #2069.
Whelp, that felt like a long week. If you havenāt heard the news by now you should definitely read Alex Suraciās post regarding our domain.
I want to take this time to thank the Concourse fans out there who offered their help, support, and positive vibes throughout the whole ordeal. The team here really appreciates it š
Luckily for us this event didnāt consume our entire engineering team. We WERE able to get some issues resolved this week and are planning for an imminent release of Concourse 3.
(UPDATE March 9 @ ~10 AM: The old domain appears to now be hosting a very old snapshot of our website. This is either targeted or part of a phishing scam. Do not go to it.)
Well, that sucked.
Wednesday morning I woke up to a ton messages because Concourseās site was gone, and in its place was a blank domain registrar placeholder.
Before you say anything, I totally remembered to renew the domain.
After some wrestling with our production pipelines last week we managed to release a patch update in the form of Concourse v3.9.1. Weāve fixed some of the reported bugs from the previous release (3.9.0) so definitely go and check it out!
I donāt have much else regarding updates this week so hereās a fun fact for you to chew on: did you know that Concourse uses Concourse to deploy Concourse? Its true!
Monday, Feb 19 was Family Day for us here in Canada, so its been a relatively short work week for the Concourse team. With the release of v3.9.0 last week, weāve gotten some reports of new bugs and issues, so thanks to everyone who reported them in via our GitHub issues and Slack. Please make sure to check the updated release notes (here) for the full details! Weāre planning to cut a new patch release early next week with some of the fixes to the reported issues.
If you havenāt heard the news by now, we released Concourse v3.9.0 this week ššš! Two of the top-line features in this release are:
Concourse will now automatically propagate certificates from the worker machine into resource containers (GH issue #1027) Improved btrfs volume driver stability. So if youāre getting hit hard by overlay weirdness, Iād suggest you give the btrfs driver another shot! To find out what else weāve packed into this release, Iād encourage you to read the full release notes on the concourse.
We spent some time this week wrapping up additional testing on our certs management across workers. We also put down some of work on Spaces this week to play around with something fun: a high density dashboard view. A lot of you have been asking us when Concourse v3.9.0 will be available, and the answer is: very soon!
On to the update:
Features UX
High Density View! Original Git issue #1899 and a demo version of it up and running can be found in our production environment Merged PR #227, thanks for the contribution SwamWithTurtles!
As a Product Manager at Pivotal, one of my responsibilities is to write weekly updates to let Pivots know what the Concourse team has been up to for the past week. When the Concourse team got together earlier this month for our 2018 planning, we decided that we should be sharing these updates with our community as a whole. So, without further ado, hereās our first update of 2018!
Features UX
Topher Bullock, Lindsay Auchinachie, Alex Suraci and I will be travelling to San Francisco next week to attend the SpringOne Platform 2017 Conference. Not only are there a lot of exciting Spring talks this year, but there will be a lot of CI/CD talks from some amazing speakers.
Unfortunately none of the Concourse core contribution team will be speaking at SpringOne this year; but that doesnāt mean thereās a shortage of Concourse-related talks.
If youāve been paying close attention to our issues on GitHub you may have noticed a small flurry of activity around one specific issue: #1707 Spike: spatial resource flows.
What are Spatial Resources Flows (aka Space)? The first reference to āspatialā resources came up in a proposal between Alex Suraci and Christopher Hendrix for Multi-branch workflows ( #1172). In that issue we focused specifically on one recurring problem: itās a real pain to deal with the Git resource when you have multiple branches representing different streams of work.
Concourse v3.6.0 comes with two new features on the build output page: timestamps and keyboard shortcuts.
Timestamps and OutputĀ Sharing When looking at the build page, you will now see timestamps reported against each line of output using your browserās reported timezone. As you hover over the timestamp, you can select single line of output or you can SHIFT select multiple lines of output. Youāll also notice that the build page URL is updated to reflect the lines you have selected.
As the Concourse team continues to grow in size and in the # of incoming issues, the team has been experimenting with new ways of managing our backlog. So far we have tried three different setups:
GitHub issues + Customs/Tracksuit + Pivotal Tracker GitHub issues + aggressive labelling + CodeTree GitHub issues + GitHub Projects Weāve been using the third setup, GitHub issues + GitHub Projects, for the past few months and weāve been mildly happy with the experience.
In 2014 the Concourse CI project started with just two engineers; Alex Suraci and Chris Brown. At the time, both Alex and Chris were working on the Pivotal Cloud Foundry team. Over time, they became increasingly frustrated by existing CI/CD solutions. In response, Alex and Chris worked on designing a new CI/CD system in their spare time; imagining a new type of CI/CD system that would treat pipelines as first class citizens.