Help shape the future of Concourse Since Concourse CI was created, thousands of users worldwide have helped the project by opening issues, committing code, and providing feedback to the team that develops the product. This community involvement is priceless - thank you, Concourse community! đź‘Ź
One of the ways the Concourse team collects feedback is through our annual Community Survey. This lets us gather crucial information about how users deploy Concourse, how different use cases scale, and various configuration patterns.
The Concourse team had the opportunity to visit some Concourse users out in Montreal last week. We had a blast meeting everyone, including some folks from the Concourse OSS community. Thanks again for hosting us!
I’ll also be in Kansas City for two days next week to meet some other Concourse users as well, so give me a tap onTwitter or Discord (username jama) if you wanna meet up.
Parallel Input Streaming In addition to the work on Algorithm improvements from the Core track, the Runtime track tested out their new work on Parallel Input Streaming.
…and we’re back! Apologies for the lack of updates lately. I’ve just come back from some time off and work travel has taken up a lot of my time. I’m back in Toronto now so let’s get back into it.
Release Engineering & Concourse 5.3.0 In the past, we relied a lot on Alex Suraci to handle a lot of our release engineering work. Release Engineering is incredibly important and valuable work for the Concourse team, but it can also very time consuming.
In case you missed it, we’ve made some tweaks to the structure of the website. I’m happy to report that Alex Suraci drastically improved our site-wide search. This resolves #181 and we’re all the better for it!
Second, you’ll notice that a lot of the community related comment that was on our homepage has now been moved to our Concourse GitHub Wiki. We hope this change will make contributor and community specific content more discoverable and more maintainable over time.
Sorry for missing the update last week. I was travelling out to the Bay area to attend the ConcourseCI Bay Area User Group. For those who missed it, you can find a recording of the event here. On to the update.
In case you missed it, Concourse 5.1.0 is out! It’s got icons on resources, better garbage collection, on_error on pipelines, and much more! As usual, you can read the full list of new features here.
Greetings from sunny Philadelphia! The team was there for the Cloud Foundry 2019 NA Summit for a few days; talking to Concourse users and attending talks. Recorded videos of the talks should be uploaded soon; so I’ll point you to the interesting Concourse-related ones next week.
On to the update.
For Active Discussion Please take some time to review and comment on the latest Concourse + k8s Runtime RFC Regarding the runtime, there’s been an active conversation around better build scheduling and load distribution.
Phew, it’s been a while since I last wrote an update. For some background behind why I slowed down, hop on over to this thread on our forms: “What would you like to see on our blog”.
That said, I do have a lot of interesting updates to share, so let’s get started
Concourse 5.0.0 In case you missed it, Concourse 5.0.0 and 5.0.1 came out a few weeks ago in March.
With the release of Concourse 5.0.0 this week I thought it would be a good time to review the evolving implementation of authorization in Concourse. I’ll also be covering some helpful debugging information for you to consider when configuring authorization in your own Concourse instance.
Read the Docs The revised Concourse Auth & Teams docsis a great place to start when diving into Concourse 5.0.0. The docs will cover important steps around provider configuration and team configuration for your cluster.
Its been a week since we switched over to the PR workflow and so far its been great! We’re still working through some of the kinks with this process so please bear with us while we continue to burn down through the list of open PRs!
And now…on to the update! I might have missed a few issues while I’m still getting used to our new workflow. Completed issues now appear as closed PRs in concourse/concourse
Some updates worth bringing up this week. As I had mentioned last week we began to do a re-organization of projects and issues in our concourse/concourse repo; you can read more about it on our wiki page here. With that said, you can find the issues and PRs that are slated for Concourse 5.0.0’s release in our 5.0.0 Milestones. If you’d like to help us with documentation, we’ve started a new branch in the docs repo under v5.
…and we’re back! The team’s been pretty quiet over the past few weeks due to vacations and holidays. This was our first week back at full strength so we’ve got some interesting updates for ya’ll
How are issues managed? This is an issue that comes up a lot in our open source community, and Alex Suraci has taken some time to clean up our issues backlog and add in some bots.
2018 has been an action-packed year for us. We saw a major release (Concourse 4.0.0) with a lot of new features: new auth connectors and users, dashboard, distributed GC and other runtime improvements. At the same time our team grew from 3 engineering pairs at the start of 2018 to 8 engineering pairs and an additional PM (đź‘‹ Scott Foerster) working on Concourse OSS and supporting Concourse for PCF.
We’re nearing the end on some UX refactoring work and finished off the issue regarding container scheduling. Between those improvements and the global resource caching, we’re hoping to see a lot of efficiency improvements in 5.0
That said, we’ve decided that we need to perform some additional performance and load testing on Concourse 5.0 before we cut the release. And with the holidays coming up, its increasingly unlikely that we’ll be able to push Concourse 5.
As I mentioned last week I’ve been doing story acceptance in our dev environments for the upcoming RBAC feature. The team’s been working through some of the new issues that come out of that to give some final polish on to the release.
Something that I haven’t talked too much about in the past weeks is our work on the Concourse k8s Helm chart. If you pull up some of the PRs under stable/concourse, you’ll see that we’ve been proposing some changes to the chart.
It was a relatively light week this week due to some vacations. I did, however, get a chance to do some acceptance work on our upcoming feature for role-based access control in Concourse. You can read more about how that’ll work in our feature preview post.
On to the update:
API
Our investigation into the API continues and branches out into more areas of the codebase. If you haven’t already, make sure to check out the two related RFCS: https://github.
Right off the bat I’d like to give a shoutout to Jamie Klassen and his new post about the upcoming feature for pinning resources. You can check it out the new post here:https://medium.com/concourse-ci/resource-page-explained-eb99cf256fb5
I also wanted to mention that the Github Pull Request that was maintained by JT Archie (https://github.com/jtarchie/github-pullrequest-resource) has been officially deprecated.
The official docs for the resource types no longer point to jtarchie/pr for the PR resource.
As a part of our refactor of the prod pipeline, Alex Suraci cleaned up and refactored parts of the TSA to better support draining and rebalancing #2748. The numbers are looking really good!
On to the update:
API We’re deep into investigations around our API documentation and management strategy. Our current investigation work is captured in #2739 but the original request comes from #1122 Core SPACCEEEE #2651 UX Continuing our work on supporting pinning of versions on resources from the UI.
This week the team got together to discuss the initial groundwork and investigations required to publish and maintain a supported API. If you’ve built any tools against our API and have feedback for us, please let us know by commenting on the original issue #1122.
In another interesting update, the PivNet team has published an update to the pivnet-resource so “you no longer need to specify the access key, secret access key, bucket and region for creating releases.
We finished our first implementation of Role Based Access Control (RBAC) this week! You can look forward to this change in our next release of Concourse.
Speaking of which, the next release of Concourse is currently blocked while we try to re-build our new release pipelines. Along with our move to the mono-repo, we’re focusing even more on making the binary distribution of Concourse the first-class distribution of Concourse. This means that you’ll get everything you need for Concourse packaged into one nifty tgz!
The results of the Concourse 2018 Community survey is out! Thanks to everyone who took the time to fill it out; and to Scott Foerster and Lindsay Auchinachie for sifting through the data.
It was a relatively short week for us due to Thanksgiving celebrations, but here’s our update:
UX
Continued our rampage in fixing fly issues: #259, #267, #1038, #1062, #248 I also wanted to add that we’re trying to keep all issues under concourse/concourse.
Alex Suraci is still tackling the chores on our One Big Repo issue #2534. Specifically, Alex is re-writing a new pipeline (aka concourse) for our mono-repo structure so we can unblock ourselves from releasing updates.
In other news, Concourse engineer Saman Alvi wrote up a short article on her experience pairing with a product designer during a discovery into the PivNet resource; check it out: Design & Dev Pairing: What we learned during a one week technical discovery.
The Concourse team went out to Washington D.C. this week to attend Spring One Platform 2018. Thanks to all the Concourse fans who stopped by to say hi, we really enjoyed meeting ya’ll. All of the talks were recorded and should be uploaded to the SpringDeveloper YouTube channel in the coming weeks. Some of the interesting talks to check out are:
Extreme Pipelines Zero to Multicloud and Spinnaker and the Distributed Monorepo …and of course Draupnir: A story about Managing Concourse in the Enterprise And now, on to the update:
Concourse 4.2.0 and Concourse 4.2.1 were released earlier this week. There’s a lot of great fixes and features in this new release, so please upgrade now!
Reminder that The Great Project Restructuring of 2018 is now underway. You’ll notice that all our submodules (e.g. ATC, TSA fly)are now all under the root level of the concourse/concourse repo. Its cleaner.
You’ll also notice that the BOSH spec has moved from its usual place.
Following up from a discussion on our forums Scott Foerster has been looking at different options for selling Concourse swag online. Do you want Concourse leggings? or maybe a limited edition @vito pls pillow! Let us know in the thread Concourse merchandising.
Please also take some time to fill out our 2018 Concourse Community survey. Your feedback is really valuable to us and the information you provide will help us plan the future of Concourse.
Apologies for the break from the usual update schedule; I wanted to get one last update out before I take some personal time, starting Fri. Aug 31 and coming back Sept 10. In my absence Scott Foerster and Alex Suraci will be writing the product update next week. The Concourse team will also be taking Monday, Sept 3rd off in observance of Labour day as well.
On to the updates:
Kubernetes As we continue our sporadic work on Kubernetes and its Helm chart, we’re also starting to expand our thinking to cover the runtime aspects of Concourse + Kubernetes. We’ve already prioritized the need to have Kubernetes as a supporting backend in addition to Garden, but what about the spiffy new developments in the Kubernetes world? We’re hearing a lot about knative and knative services like build and eventing. Are there any kubernetes users who’d like to weigh in on the topic?
Going to switch things up this week and start with some interesting community news:
We’ve decided to restructure our repositories to make things more understandable and less scary for contributors. Alex Suraci has laid out a good explainer on why and how we’re going to start in our PSA: the Great Code Restructing of 2018 Lindsay Auchinachie wrote up a blog post describing some of the visual elements of the Concourse pipeline view in a blog post titled Concourse Pipeline UI Explained marco-m has been updating a “concourse-in-a-box” formula that comes with a s3-compatible-store and a Vault.
With the launch of Concourse 4.0.0, we’ve been monitoring our typical communication channelscarefully to watch out for any glaring new bugs. So far we seem to be safe from any crazy issues, but we have noticed that there has been some confusion in how to set the basic auth users in the new deployment method (see #2421 for details). Thanks everyone for your patience and working through these issues with us!
I’m happy to announce that we released Concourse 4.0.0 this week! This was a HUGE release with over 28 new features and fixes. I’d encourage you to read through the full list of changes on our Downloads page.
Why did this release warrant a bump in the major version? Well, if you’ve been following along closely you’ll know that we had just finished our new auth work in 4.0.0. Users are now central to the authentication flows, not teams.
This week, the Concourse team went out to Portland to attend OSCON 2018. Topher Bullock gave a great intro to Concourse in the Open Source track. We even met some of the Concourse fans in person!
In other news, we’ve begun to sketch out what RBAC might look like in Concourse. Please check out #2389 when you have some time!
On to the update:
UX
Team has been working on adding drag and drop re-arranging for the dashboard in #2364 We also found a weird quirk with the new team creation flow, where you won’t see your team if it was just created and has no pipelines.
We’re going dark themed for Concourse 4.0.0! In addition to the users work, we’re promoting the Dashboard to the / level to take over the home page. You’ll also notice that we added pipeline play/pause capabilities to the dashboard, NEAT!
To keep things consistent, we’re also propagating our new design to the existing pipeline views. You can play around with this new nav structure on our own CI: https://ci.concourse-ci.org/
Since July 1st was the official day of Canada’s birth, the Concourse team enjoyed a long weekend with no work on Monday. We were, however, able to get quite a bit done during this short week.
A big win is that we added a k8s-testflight job to our official ci pipeline (check it out here); this will let us know in advance when we have broken the Concourse Kubernetes Helm Chart.
If you’ve been following along with our Auth changes, you’ll know that we’ve been doing a lot of work behind the scenes to make the upgrade into this new world as seamless as possible. This week, we were able to do our first large-scale upgrade test against our Wings instance. The upgrade went well and we were able to find a few more areas of polish before we push this feature.
It’s been a busy week for myself and Topher Bullock. We spent some time in Boston meeting with some users operating large-scale Concourses. We learned a lot about the issues they were running into operating Concourse a scale…and we ate a lot of Lobster!
On to the update:
UX:
Made some improvements to the build page in issue #1543 that we’re hoping to test soon on our internal Concourse. You can read into some more of the details in our comments.
This was a post-release week, so we spent a lot of time merging in new code from the Users track, fixing our pipelines, and working on some neglected issues. All in all a solid week’s worth of work! On to the update
UX
Changed the behaviour of the breadcrumb so that clicking on the pipeline name resets the pipeline view and group settings (#2258) Fixed a bug with the breadcrumb where it wouldn’t render whitespace correctly (#2267) Fixed a bug with team name overflowing on breadcrumbs (#2241) Fixed a UI bug on the navigation arrows (#2276) Added JSON stdout to Fly CLI (#952) We haven’t done work on this yet, but based on our observations and feedback from the community, we’re planning to push the dashboard up to / level.
Big release this week! After lots internal load testing on Wings we finally felt comfortable releasing Concourse 3.14.0. In addition to the new Distributed Garbage Collection, breadcrumbs, responsive groups, and Windows worker, we have 14 new features a whole bunch of bug fixes. But wait! Don’t download that one; get Concourse v3.14.1 instead.
A few other updates. First, be sure to check out my write up on How We Build Concourse.
If you’ve been experiencing “Aw Snap” errors on Chrome with Concourse 3.13.0 or 3.12.0 we traced the root case to two lines of CSS. This seems to happen only on Chrome 67; so a temporary workaround is to switch over to Chrome canary or use Firefox/Safari/Edge. You can follow along in our discussion at GitHub issue #2236
Now, on to the update
Runtime
We were able to successfully test our distributed volume GC collection code on our Wings environment this week.
It was a short week for us here in Canada, but we had a few interesting updates:
We attempted to deploy our distributed GC changes to our internal environment “Wings” last Friday. Turns out that was an incredibly bad idea. The deployment failed horrifically and we had to roll back all our changes. We’re still investigating why our code worked in our “prod” environment but failed when deployed onto Wings. We’re tracking this work in issue #2202.
In case you missed it, I’d encourage you to check out some of the recent posts from Shashwathi Reddy on “My first month on Concourse” and Joshua Winters regarding upcoming changes to our authentication; “Oh, Auth”. We’d love to hear your feedback!
Heads up: the Concourse team will be taking Monday May 21st off for Victoria Dayholiday.
And now, on to the update:
Core
Continued banging our heads against new auth connectors with Dex.
Hi folks,
Joshua Winters has spent a lot of time refactoring Concourse so that it can finally support Users. We’re finally at a point where we can share some our work with you, so I’d really encourage you to check out his recent blog post Oh, Auth
Outside of that, we’ve got a bunch of vacations going on this week, so it’s been more of the same three tracks of work:
I’ve gotten some questions about Freedom Friday from some readers after last week’s update. Well it turns out thatTopher Bullock wrote a great article about it this week; you read up on it here: https://medium.com/concourse-ci/freedom-fridays-319204dea834
We also release Concourse v3.13.0 earlier this week. Make sure you check it out if you were hit by the accumulating logs issue introduced in v3.12.0.
On to the update:
Space We’ve been building out some of the frontend code for representing Spaces as part of #2131.
Well, that was fun! Topher Bullock absolutely killed it last week on the CF Summit 2018 main stage with his demo of the experimental Concourse ❤ ️K8s runtime project. We also had a great time talking to companies who were using Concourse to continuously do things in the cloud. One of my favorite talks was from Jason Immerman and Derek Van Assche from Zipcar (Concourse All the Things, All the Time); really inspirational stuff!
Concourse v3.11.0 came out today! Go get it: https://concourse-ci.org/download.html#v3110
On another note: I’ve noticed some interesting articles and guides come out on writing custom resources for Concourse. There’s one on the Pivotal blog https://content.pivotal.io/blog/developing-a-custom-concourse-resource and another from fellow Medium writer Shin Myung Yoon (https://itnext.io/writing-a-custom-resource-for-concourse-detecting-pull-request-close-merge-events-e40468eb2a81)!
Topher Bullock and I will also be travelling to Boston next week for CF Summit 2018. We’ll be around to meet some Pivotal PCF customers and answer questions about Concourse.
If you haven’t done so already please check out Alex Suraci’s recent update post on “A renewed focus & community changes”. It covers all the recent changes that we’ve been making; starting with the new styling of the website, our new discussion forum, and our migration to Discord chat.
Specifically, we’ve been getting some mixed feedback on the new format of the site. Some folks love it, other folks miss the highly visual styling of the old site.
Hi folks,
Had an interesting week talking to customers about how we might improve their Concourse operations and deployments. More info on that soon!
On to the update:
UX
Fixed an issue with timestamps #2088 Core
Continued our refactoring of the API to support dex and users #1888 Runtime
Finished the issue around custom resources on tagged workers, it should work now #1371 Restricted the list of allowed TLS ciphers for more security checkboxing #1997 Design
Its been a first week in a long time where we were back to full strength and fully co-located. It was nice!
Oh, and Concourse v3.9.2 was released this week as well,check it out!
On to our update:
UX
Started to make our dashboard a bit more mobile friendly #1712 Started to tackle the problem where our dashboard holds too many open connections #1806 Core
Removed a dependancy to provide an external URL for fly execute #2069.
Whelp, that felt like a long week. If you haven’t heard the news by now you should definitely read Alex Suraci’s post regarding our domain.
I want to take this time to thank the Concourse fans out there who offered their help, support, and positive vibes throughout the whole ordeal. The team here really appreciates it 🙏
Luckily for us this event didn’t consume our entire engineering team. We WERE able to get some issues resolved this week and are planning for an imminent release of Concourse 3.
After some wrestling with our production pipelines last week we managed to release a patch update in the form of Concourse v3.9.1. We’ve fixed some of the reported bugs from the previous release (3.9.0) so definitely go and check it out!
I don’t have much else regarding updates this week so here’s a fun fact for you to chew on: did you know that Concourse uses Concourse to deploy Concourse? Its true!
Monday, Feb 19 was Family Day for us here in Canada, so its been a relatively short work week for the Concourse team. With the release of v3.9.0 last week, we’ve gotten some reports of new bugs and issues, so thanks to everyone who reported them in via our GitHub issues and Slack. Please make sure to check the updated release notes (here) for the full details! We’re planning to cut a new patch release early next week with some of the fixes to the reported issues.
If you haven’t heard the news by now, we released Concourse v3.9.0 this week 🎉🎉🎉! Two of the top-line features in this release are:
Concourse will now automatically propagate certificates from the worker machine into resource containers (GH issue #1027) Improved btrfs volume driver stability. So if you’re getting hit hard by overlay weirdness, I’d suggest you give the btrfs driver another shot! To find out what else we’ve packed into this release, I’d encourage you to read the full release notes on the concourse.
We spent some time this week wrapping up additional testing on our certs management across workers. We also put down some of work on Spaces this week to play around with something fun: a high density dashboard view. A lot of you have been asking us when Concourse v3.9.0 will be available, and the answer is: very soon!
On to the update:
Features UX
High Density View! Original Git issue #1899 and a demo version of it up and running can be found in our production environment Merged PR #227, thanks for the contribution SwamWithTurtles!
As a Product Manager at Pivotal, one of my responsibilities is to write weekly updates to let Pivots know what the Concourse team has been up to for the past week. When the Concourse team got together earlier this month for our 2018 planning, we decided that we should be sharing these updates with our community as a whole. So, without further ado, here’s our first update of 2018!
Features UX
Topher Bullock, Lindsay Auchinachie, Alex Suraci and I will be travelling to San Francisco next week to attend the SpringOne Platform 2017 Conference. Not only are there a lot of exciting Spring talks this year, but there will be a lot of CI/CD talks from some amazing speakers.
Unfortunately none of the Concourse core contribution team will be speaking at SpringOne this year; but that doesn’t mean there’s a shortage of Concourse-related talks.
As the Concourse team continues to grow in size and in the # of incoming issues, the team has been experimenting with new ways of managing our backlog. So far we have tried three different setups:
GitHub issues + Customs/Tracksuit + Pivotal Tracker GitHub issues + aggressive labelling + CodeTree GitHub issues + GitHub Projects We’ve been using the third setup, GitHub issues + GitHub Projects, for the past few months and we’ve been mildly happy with the experience.
In 2014 the Concourse CI project started with just two engineers; Alex Suraci and Chris Brown. At the time, both Alex and Chris were working on the Pivotal Cloud Foundry team. Over time, they became increasingly frustrated by existing CI/CD solutions. In response, Alex and Chris worked on designing a new CI/CD system in their spare time; imagining a new type of CI/CD system that would treat pipelines as first class citizens.