Running Docker in Concourse

So you want to run Docker in Concourse? Well this is the guide for you! Let’ clarify what it is we want to do. We want to be able to run docker-compose inside a task in Concourse to bring up our application along side some other services (i.e. Redis, Postgres, MySQL, etc.). Thankfully this challenge has been solved by the community! There are a few “Docker-in-Docker” images designed to run in Concourse that are maintained by the community.

GitOps For Your Pipelines

In this blog post we’re going to cover how to use git and Concourse to automatically set, update, and archive your pipelines using the set_pipeline step. No longer will you need to use fly set-pipeline to update any of your pipelines! For consistency we will refer to the pipeline that contains all the set_pipeline steps as the parent pipeline. The pipelines created by the set_pipeline steps will be called child pipelines.

RFC round-up: July 10th, 2020

Happy Friday! This one’s brief. Merged RFCs RFC #37: prototypes has landed! …but it probably could have use more detail regarding the run step, which is the only immediately actionable part of it. šŸ¤” I’ll draft another RFC for that; #37 mainly covered the protocol. RFC #38: resource prototypes is in! Its associated issue for implementation is #5870. RFCs to merge n/a - taking a breather for this round-up to focus on the below RFCs and “reset” the 2 week merge window so I can start publishing these posts earlier in the week.

RFC round-up: June 24th, 2020

With the four(!) RFCs from the last round-up now merged, it’s time to move on to the next RFC milestone: Prototypes! Merged RFCs šŸŽ‰ RFC #31: set_pipeline step RFC #40: valid identifiers RFC #39: var sources RFC #27: var steps RFCs ready to merge šŸ¤ž RFC #37: Prototypes is finally ready to go! For (much) further reading, check out the Re-inventing resource types blog post. The importance of this RFC really cannot be overstated; it will be the most significant change to Concourse since its creation.

How To Build and Publish a Container Image

In this blog post we are going to show how to build and publish container images using the oci-build task and registry-image resource. This post assumes you understand how to build container images with Dockerfile’s and publish to Docker Hub or another image registry using the docker cli. If you just want to see the pipeline, scroll to the bottom or click here. What follows is a detailed explanation of what each part of the pipeline does.

RFC round-up: June 10th, 2020

First off: sorry, I immediately failed to keep my target pace for these. šŸ˜“ I got wrapped up in a deadline, and since I alternate weeks between engineering and community duties like this post, when I miss a week for RFC updates the 2-week interval can quickly turn into 4 or 5. Owing to the missed round-up, and in hopes of burning through the backlog more quickly so that interested contributors may volunteer for merged RFCs, I’m going to expand the scope of this post to include more RFCs than the last one - primarily by proposing that we merge ones that are nearly certain for the v10 roadmap.

Introduction to Task Inputs and Outputs

Understanding how task inputs and outputs work in Concourse can be a little confusing initially. This post will walk you through a few example pipelines to show you how inputs and outputs work within a single Concourse job. By the end you should understand how inputs and outputs work within the context of a single job. Let’s define some jargon first. step : A step is a container running code within the context of a Concourse job.

Concourse 2020 Community Report

A little over a month ago, the Concourse team sent a survey out to the community. The purpose of this survey was to gain insight into our users as well as measure our year-over-year growth. In the process of learning about how you all deploy and manage Concourse, we also received tons of great feedback about what’s working well and what needs work in order to make Concourse even better. Weā€™re excited to share our findings!

RFC round-up: May 6th, 2020

Howdy, and welcome to our first RFC round-up! šŸ¤  For those unaware, Concourse RFCs are a process for proposing and collaborating on improvements to core Concourse functionality, including pipeline behavior, new step types, new operator capabilities, etc. In short, RFCs are where all the cool new stuff is planned. šŸ˜Ž My goal is to provide an update at least every few weeks on the status of RFCs and shepherd them through the process via blog posts like this one.

Community update: enter Discussions! šŸŽ‰

Hasta la vista, stale bot. tl;dr: We’ve been granted access to GitHub’s beta Discussions feature! šŸŽ‰ Discussions on the concourse repo will be used for questions and technical support. Discussions on the rfcs repo will be for incubating ideas for new workflows , which eventually turn into Pull Requests (also on the rfcs repo). From here on, Issues on the concourse repo are exclusively for project backlog and bug reports - i.

Developing Concourse (from home šŸ”)

In March 2020, countless companies made a shift to have their employees work from home. For remote staff getting work done can be challenging enough, but staying connected to your team and company culture can be even more challenging. On the Concourse team, weā€™re working hard to keep our product development running smoothly with some additions to our tech stack and day to day workflow. Concourse team composition At VMware (formerly Pivotal Software), there are fourteen engineers and five product folks working on theConcourse CI project full time.

Concourse 2020 Community Survey

Help shape the future of Concourse Since Concourse CI was created, thousands of users worldwide have helped the project by opening issues, committing code, and providing feedback to the team that develops the product. This community involvement is priceless - thank you, Concourse community! šŸ‘ One of the ways the Concourse team collects feedback is through our annual Community Survey. This lets us gather crucial information about how users deploy Concourse, how different use cases scale, and various configuration patterns.

A New Hangar For Resource Types

Photo: National Parks Service The idea to build a dedicated resource types catalog has been long-discussed. Weā€™d like to announce that the idea has come to fruition: the new Concourse resource types catalog is wheels up! The catalog lists Concourse resource types that have recently been submitted to the resource types GitHub repo. Originally, resource types were listed on a GitHub wiki page. While the wiki page listed resource types, it didnā€™t provide much information about each resource.

Re-inventing resource types

Before the paint completely dries on the v10 roadmap, there is one last big unknown I want to explore in case it brings more clarity to our direction: generic tasks. Resource types are a great way to share tools and integrations for others to use in their pipelines. Unfortunately, they’re basically the only way, and because resources are a very opinionated concept, the resource type interface is not always a good fit.

Concourse Mid-year Update

Phew, it’s been a while. I got lots of info to cover so let’s just get right into it Concourse OSS Growth As some of you may know, the Concourse team switched over to a PR-based workflow at the beginning of the year. This change is in line with our objectives of being open and transparent with our community of contributors. Plus, its just good thing to do because that’s what most OSS projects do.

Core roadmap: towards v10

A long-term roadmap for the core design of Concourse, a general-purpose CI/CD tool. Accompanying slides. Recommended viewing: episode 1 of Yu-Gi-Oh. Concourse’s design philosophy is to be expressive, versatile, and safe while limited to a handful of simple, proven concepts. The design of these concepts should make good practices feel intuitive and bad practices feel uncomfortable. Coming up with these designs can be very challenging. There are many different workflows and patterns across the software industry, and they each have to be deeply understood in order to know what the good and bad practices are.

Designing a Dashboard for Concourse

How does the Concourse team go about solving the problem of pipeline monitoring? With the growing popularity of Concourse, we noticed that our development teams wanted to observe and monitor multiple pipelines simultaneously. This behaviour wasnā€™t limited to just Pivotal engineering teams; in fact, it was even more prevalent amongst our Open Source Community. Our users currently solve this by cramming multiple browser windows into TV their monitor view or they use the Concourse Summary (aka Crystal) by David Goddard of the Pivotal Buildpacks team.

Concourse Update (July 21 2019)

The Concourse team had the opportunity to visit some Concourse users out in Montreal last week. We had a blast meeting everyone, including some folks from the Concourse OSS community. Thanks again for hosting us! Iā€™ll also be in Kansas City for two days next week to meet some other Concourse users as well, so give me a tap onTwitter or Discord (username jama) if you wanna meet up. Parallel Input Streaming In addition to the work on Algorithm improvements from the Core track, the Runtime track tested out their new work on Parallel Input Streaming.

Concourse Update June 7

…and weā€™re back! Apologies for the lack of updates lately. Iā€™ve just come back from some time off and work travel has taken up a lot of my time. Iā€™m back in Toronto now so letā€™s get back into it. Release Engineering & Concourse 5.3.0 In the past, we relied a lot on Alex Suraci to handle a lot of our release engineering work. Release Engineering is incredibly important and valuable work for the Concourse team, but it can also very time consuming.

Concourse Update April 29ā€“3

In case you missed it, weā€™ve made some tweaks to the structure of the website. Iā€™m happy to report that Alex Suraci drastically improved our site-wide search. This resolves #181 and weā€™re all the better for it! Second, youā€™ll notice that a lot of the community related comment that was on our homepage has now been moved to our Concourse GitHub Wiki. We hope this change will make contributor and community specific content more discoverable and more maintainable over time.

Concourse Update (April 8ā€“18)

Sorry for missing the update last week. I was travelling out to the Bay area to attend the ConcourseCI Bay Area User Group. For those who missed it, you can find a recording of the event here. On to the update. In case you missed it, Concourse 5.1.0 is out! Itā€™s got icons on resources, better garbage collection, on_error on pipelines, and much more! As usual, you can read the full list of new features here.

Concourse Update (April 1ā€“5)

Greetings from sunny Philadelphia! The team was there for the Cloud Foundry 2019 NA Summit for a few days; talking to Concourse users and attending talks. Recorded videos of the talks should be uploaded soon; so Iā€™ll point you to the interesting Concourse-related ones next week. On to the update. For Active Discussion Please take some time to review and comment on the latest Concourse + k8s Runtime RFC Regarding the runtime, thereā€™s been an active conversation around better build scheduling and load distribution.

Concourse Update (šŸ¤·-April 1, 2019)

Phew, itā€™s been a while since I last wrote an update. For some background behind why I slowed down, hop on over to this thread on our forms: ā€œWhat would you like to see on our blogā€. That said, I do have a lot of interesting updates to share, so letā€™s get started Concourse 5.0.0 In case you missed it, Concourse 5.0.0 and 5.0.1 came out a few weeks ago in March.

An Overview of Authorization in Concourse 3, 4 and 5

With the release of Concourse 5.0.0 this week I thought it would be a good time to review the evolving implementation of authorization in Concourse. Iā€™ll also be covering some helpful debugging information for you to consider when configuring authorization in your own Concourse instance. Read the Docs The revised Concourse Auth & Teams docsis a great place to start when diving into Concourse 5.0.0. The docs will cover important steps around provider configuration and team configuration for your cluster.

Concourse Update (Jan 21ā€“25)

Its been a week since we switched over to the PR workflow and so far its been great! Weā€™re still working through some of the kinks with this process so please bear with us while we continue to burn down through the list of open PRs! And nowā€¦on to the update! I might have missed a few issues while Iā€™m still getting used to our new workflow. Completed issues now appear as closed PRs in concourse/concourse

Concourse Update (Jan 14ā€“18)

Some updates worth bringing up this week. As I had mentioned last week we began to do a re-organization of projects and issues in our concourse/concourse repo; you can read more about it on our wiki page here. With that said, you can find the issues and PRs that are slated for Concourse 5.0.0ā€™s release in our 5.0.0 Milestones. If youā€™d like to help us with documentation, weā€™ve started a new branch in the docs repo under v5.

Concourse Update (Jan 7ā€“11)

ā€¦and weā€™re back! The teamā€™s been pretty quiet over the past few weeks due to vacations and holidays. This was our first week back at full strength so weā€™ve got some interesting updates for yaā€™ll How are issues managed? This is an issue that comes up a lot in our open source community, and Alex Suraci has taken some time to clean up our issues backlog and add in some bots.

The Great Process Update of 2018

Or, ā€œwhy we havenā€™t shipped any new features since September.ā€ You may have noticed that our release cadence has slowed down significantly in the past few months. The bad news is we probably wonā€™t get a release out this year (mainly due to end-of-year vacations and slowing down in general), but the good news is the next release is hugeā€Šā€”ā€Šbig enough to bump us to v5.0ā€Šā€”ā€Šand itā€™s just about ready.

Concourse 2018 Year in Review

2018 has been an action-packed year for us. We saw a major release (Concourse 4.0.0) with a lot of new features: new auth connectors and users, dashboard, distributed GC and other runtime improvements. At the same time our team grew from 3 engineering pairs at the start of 2018 to 8 engineering pairs and an additional PM (šŸ‘‹ Scott Foerster) working on Concourse OSS and supporting Concourse for PCF.

Concourse Update (Dec 3ā€“7)

Weā€™re nearing the end on some UX refactoring work and finished off the issue regarding container scheduling. Between those improvements and the global resource caching, weā€™re hoping to see a lot of efficiency improvements in 5.0 That said, weā€™ve decided that we need to perform some additional performance and load testing on Concourse 5.0 before we cut the release. And with the holidays coming up, its increasingly unlikely that weā€™ll be able to push Concourse 5.

Concourse Updates (Nov 26ā€“30)

As I mentioned last week Iā€™ve been doing story acceptance in our dev environments for the upcoming RBAC feature. The teamā€™s been working through some of the new issues that come out of that to give some final polish on to the release. Something that I havenā€™t talked too much about in the past weeks is our work on the Concourse k8s Helm chart. If you pull up some of the PRs under stable/concourse, youā€™ll see that weā€™ve been proposing some changes to the chart.

Concourse RBAC Preview

One of the big themes for Concourse in 2018 has been Users, multiple auth connectors, and role-based access control (aka RBAC). With RBAC in the final phases of development, I wanted to give you a preview of some of the functionality that you can expect in our upcoming release; Concourse 5.0 Admins, Owners, Members and Viewers Concourse 5.0 will come with 4 roles: Concourse Admin, Team Owner, Team Member, and Team Viewer.

Concourse Update (Nov 19ā€“23)

It was a relatively light week this week due to some vacations. I did, however, get a chance to do some acceptance work on our upcoming feature for role-based access control in Concourse. You can read more about how thatā€™ll work in our feature preview post. On to the update: API Our investigation into the API continues and branches out into more areas of the codebase. If you havenā€™t already, make sure to check out the two related RFCS: https://github.

Concourse Update (Nov 5ā€“9)

Right off the bat Iā€™d like to give a shoutout to Jamie Klassen and his new post about the upcoming feature for pinning resources. You can check it out the new post here:https://medium.com/concourse-ci/resource-page-explained-eb99cf256fb5 I also wanted to mention that the Github Pull Request that was maintained by JT Archie (https://github.com/jtarchie/github-pullrequest-resource) has been officially deprecated. The official docs for the resource types no longer point to jtarchie/pr for the PR resource.

Concourse Update (Oct 29ā€” Nov 2)

As a part of our refactor of the prod pipeline, Alex Suraci cleaned up and refactored parts of the TSA to better support draining and rebalancing #2748. The numbers are looking really good! On to the update: API Weā€™re deep into investigations around our API documentation and management strategy. Our current investigation work is captured in #2739 but the original request comes from #1122 Core SPACCEEEE #2651 UX Continuing our work on supporting pinning of versions on resources from the UI.

Concourse Update (Oct 22ā€“26)

This week the team got together to discuss the initial groundwork and investigations required to publish and maintain a supported API. If youā€™ve built any tools against our API and have feedback for us, please let us know by commenting on the original issue #1122. In another interesting update, the PivNet team has published an update to the pivnet-resource so ā€œyou no longer need to specify the access key, secret access key, bucket and region for creating releases.

Concourse Update (Oct 15ā€“19)

We finished our first implementation of Role Based Access Control (RBAC) this week! You can look forward to this change in our next release of Concourse. Speaking of which, the next release of Concourse is currently blocked while we try to re-build our new release pipelines. Along with our move to the mono-repo, weā€™re focusing even more on making the binary distribution of Concourse the first-class distribution of Concourse. This means that youā€™ll get everything you need for Concourse packaged into one nifty tgz!

Concourse Update (Oct 9ā€“12)

The results of the Concourse 2018 Community survey is out! Thanks to everyone who took the time to fill it out; and to Scott Foerster and Lindsay Auchinachie for sifting through the data. It was a relatively short week for us due to Thanksgiving celebrations, but hereā€™s our update: UX Continued our rampage in fixing fly issues: #259, #267, #1038, #1062, #248 I also wanted to add that weā€™re trying to keep all issues under concourse/concourse.

Concourse Update (Oct 1ā€“5)

Alex Suraci is still tackling the chores on our One Big Repo issue #2534. Specifically, Alex is re-writing a new pipeline (aka concourse) for our mono-repo structure so we can unblock ourselves from releasing updates. In other news, Concourse engineer Saman Alvi wrote up a short article on her experience pairing with a product designer during a discovery into the PivNet resource; check it out: Design & Dev Pairing: What we learned during a one week technical discovery.

Concourse Update (Sept 24ā€“28)

The Concourse team went out to Washington D.C. this week to attend Spring One Platform 2018. Thanks to all the Concourse fans who stopped by to say hi, we really enjoyed meeting yaā€™ll. All of the talks were recorded and should be uploaded to the SpringDeveloper YouTube channel in the coming weeks. Some of the interesting talks to check out are: Extreme Pipelines Zero to Multicloud and Spinnaker and the Distributed Monorepo …and of course Draupnir: A story about Managing Concourse in the Enterprise And now, on to the update:

Concourse Update (Sept 17ā€“21)

Concourse 4.2.0 and Concourse 4.2.1 were released earlier this week. Thereā€™s a lot of great fixes and features in this new release, so please upgrade now! Reminder that The Great Project Restructuring of 2018 is now underway. Youā€™ll notice that all our submodules (e.g. ATC, TSA fly)are now all under the root level of the concourse/concourse repo. Its cleaner. Youā€™ll also notice that the BOSH spec has moved from its usual place.

Concourse Update (Sept 10 ā€” Sept 14)

Following up from a discussion on our forums Scott Foerster has been looking at different options for selling Concourse swag online. Do you want Concourse leggings? or maybe a limited edition @vito pls pillow! Let us know in the thread Concourse merchandising. Please also take some time to fill out our 2018 Concourse Community survey. Your feedback is really valuable to us and the information you provide will help us plan the future of Concourse.

Concourse Update (Aug 27ā€“31)

Apologies for the break from the usual update schedule; I wanted to get one last update out before I take some personal time, starting Fri. Aug 31 and coming back Sept 10. In my absence Scott Foerster and Alex Suraci will be writing the product update next week. The Concourse team will also be taking Monday, Sept 3rd off in observance of Labour day as well. On to the updates:

Concourse Update (August 20ā€“24)

Kubernetes As we continue our sporadic work on Kubernetes and its Helm chart, weā€™re also starting to expand our thinking to cover the runtime aspects of Concourse + Kubernetes. Weā€™ve already prioritized the need to have Kubernetes as a supporting backend in addition to Garden, but what about the spiffy new developments in the Kubernetes world? Weā€™re hearing a lot about knative and knative services like build and eventing. Are there any kubernetes users whoā€™d like to weigh in on the topic?

Concourse Update (August 13ā€“17)

Going to switch things up this week and start with some interesting community news: Weā€™ve decided to restructure our repositories to make things more understandable and less scary for contributors. Alex Suraci has laid out a good explainer on why and how weā€™re going to start in our PSA: the Great Code Restructing of 2018 Lindsay Auchinachie wrote up a blog post describing some of the visual elements of the Concourse pipeline view in a blog post titled Concourse Pipeline UI Explained marco-m has been updating a ā€œconcourse-in-a-boxā€ formula that comes with a s3-compatible-store and a Vault.

Concourse Update (Aug 7ā€“10)

As I mentioned last week, this was a short week for us in Canada due to the Civic Holiday. We did, however, manage to work on some pretty cool stuff! With the release of 4.0.0, weā€™ve been shifting our new feature focus towards Operations and Runtime. Weā€™re intentionally slowing down on UX to focus on regressions and UI polish for existing screens. On to the update: Core Continued our work on ā€œpinningā€ a version of a resource across the pipeline.

Suspicious Volume Usage on Workers

As a Product Manger at Pivotal Iā€™m often called on to help with our customerā€™s Concourse-related issues. I recently spent some time hunting down an issue around suspiciously high volume usage on Concourse workers. It was an interesting problem that I wanted to share with the broader Concourse community. Platform Management Pipelines One of the primary use case for Concourse within the Pivotal Cloud Foundry (PCF) ecosystem is to automate the toil of manual maintenance against the platform; specifically PAS and PKS.

Concourse Update (July 30 ā€” Aug 3)

With the launch of Concourse 4.0.0, weā€™ve been monitoring our typical communication channelscarefully to watch out for any glaring new bugs. So far we seem to be safe from any crazy issues, but we have noticed that there has been some confusion in how to set the basic auth users in the new deployment method (see #2421 for details). Thanks everyone for your patience and working through these issues with us!

Concourse Update (Jul 23ā€“27)

Iā€™m happy to announce that we released Concourse 4.0.0 this week! This was a HUGE release with over 28 new features and fixes. Iā€™d encourage you to read through the full list of changes on our Downloads page. Why did this release warrant a bump in the major version? Well, if youā€™ve been following along closely youā€™ll know that we had just finished our new auth work in 4.0.0. Users are now central to the authentication flows, not teams.

Concourse Update (Jul 16ā€“20)

This week, the Concourse team went out to Portland to attend OSCON 2018. Topher Bullock gave a great intro to Concourse in the Open Source track. We even met some of the Concourse fans in person! In other news, weā€™ve begun to sketch out what RBAC might look like in Concourse. Please check out #2389 when you have some time! On to the update: UX Team has been working on adding drag and drop re-arranging for the dashboard in #2364 We also found a weird quirk with the new team creation flow, where you wonā€™t see your team if it was just created and has no pipelines.

Concourse Update (Jul 9ā€“13)

Weā€™re going dark themed for Concourse 4.0.0! In addition to the users work, weā€™re promoting the Dashboard to the / level to take over the home page. Youā€™ll also notice that we added pipeline play/pause capabilities to the dashboard, NEAT! To keep things consistent, weā€™re also propagating our new design to the existing pipeline views. You can play around with this new nav structure on our own CI: https://ci.concourse-ci.org/

Concourse Updates (July 3ā€“6)

Since July 1st was the official day of Canadaā€™s birth, the Concourse team enjoyed a long weekend with no work on Monday. We were, however, able to get quite a bit done during this short week. A big win is that we added a k8s-testflight job to our official ci pipeline (check it out here); this will let us know in advance when we have broken the Concourse Kubernetes Helm Chart.

Concourse Update (Jun 25ā€“29)

If youā€™ve been following along with our Auth changes, youā€™ll know that weā€™ve been doing a lot of work behind the scenes to make the upgrade into this new world as seamless as possible. This week, we were able to do our first large-scale upgrade test against our Wings instance. The upgrade went well and we were able to find a few more areas of polish before we push this feature.

Concourse Update (Jun 18ā€“22)

Itā€™s been a busy week for myself and Topher Bullock. We spent some time in Boston meeting with some users operating large-scale Concourses. We learned a lot about the issues they were running into operating Concourse a scaleā€¦and we ate a lot of Lobster! On to the update: UX: Made some improvements to the build page in issue #1543 that weā€™re hoping to test soon on our internal Concourse. You can read into some more of the details in our comments.

Concourse Update (Jun 11ā€“15)

This was a post-release week, so we spent a lot of time merging in new code from the Users track, fixing our pipelines, and working on some neglected issues. All in all a solid weekā€™s worth of work! On to the update UX Changed the behaviour of the breadcrumb so that clicking on the pipeline name resets the pipeline view and group settings (#2258) Fixed a bug with the breadcrumb where it wouldnā€™t render whitespace correctly (#2267) Fixed a bug with team name overflowing on breadcrumbs (#2241) Fixed a UI bug on the navigation arrows (#2276) Added JSON stdout to Fly CLI (#952) We havenā€™t done work on this yet, but based on our observations and feedback from the community, weā€™re planning to push the dashboard up to / level.

Concourse Update (June 4ā€“8)

Big release this week! After lots internal load testing on Wings we finally felt comfortable releasing Concourse 3.14.0. In addition to the new Distributed Garbage Collection, breadcrumbs, responsive groups, and Windows worker, we have 14 new features a whole bunch of bug fixes. But wait! Donā€™t download that one; get Concourse v3.14.1 instead. A few other updates. First, be sure to check out my write up on How We Build Concourse.

How We Build Concourse

Building on some of our previous posts on the Concourse team mechanicsĀ¹, I wanted to spend some time going over how we actually build Concourse. Tracking Features and Bugs Concourse tracks all of its bugs, features and epics through GitHub Issues. For items regarding the core functionality of Concourse itself, you can find the master issues list here: https://github.com/concourse/concourse/issues For issues regarding the Concourse website and documentation, you can find the backlog here: https://github.

Concourse Update (May 28 ā€” June 1)

If youā€™ve been experiencing ā€œAw Snapā€ errors on Chrome with Concourse 3.13.0 or 3.12.0 we traced the root case to two lines of CSS. This seems to happen only on Chrome 67; so a temporary workaround is to switch over to Chrome canary or use Firefox/Safari/Edge. You can follow along in our discussion at GitHub issue #2236 Now, on to the update Runtime We were able to successfully test our distributed volume GC collection code on our Wings environment this week.

Concourse Update (May 22ā€“25)

It was a short week for us here in Canada, but we had a few interesting updates: We attempted to deploy our distributed GC changes to our internal environment ā€œWingsā€ last Friday. Turns out that was an incredibly bad idea. The deployment failed horrifically and we had to roll back all our changes. Weā€™re still investigating why our code worked in our ā€œprodā€ environment but failed when deployed onto Wings. Weā€™re tracking this work in issue #2202.

Concourse Update (May 14ā€“18)

In case you missed it, Iā€™d encourage you to check out some of the recent posts from Shashwathi Reddy on ā€œMy first month on Concourseā€ and Joshua Winters regarding upcoming changes to our authentication; ā€œOh, Authā€. Weā€™d love to hear your feedback! Heads up: the Concourse team will be taking Monday May 21st off for Victoria Dayholiday. And now, on to the update: Core Continued banging our heads against new auth connectors with Dex.

Concourse Update (May 7ā€“11)

Hi folks, Joshua Winters has spent a lot of time refactoring Concourse so that it can finally support Users. Weā€™re finally at a point where we can share some our work with you, so Iā€™d really encourage you to check out his recent blog post Oh, Auth Outside of that, weā€™ve got a bunch of vacations going on this week, so itā€™s been more of the same three tracks of work:

Concourse Update (April 30 ā€” May 4)

Iā€™ve gotten some questions about Freedom Friday from some readers after last weekā€™s update. Well it turns out thatTopher Bullock wrote a great article about it this week; you read up on it here: https://medium.com/concourse-ci/freedom-fridays-319204dea834 We also release Concourse v3.13.0 earlier this week. Make sure you check it out if you were hit by the accumulating logs issue introduced in v3.12.0. On to the update: Space Weā€™ve been building out some of the frontend code for representing Spaces as part of #2131.

Concourse Update (April 23ā€“27)

Well, that was fun! Topher Bullock absolutely killed it last week on the CF Summit 2018 main stage with his demo of the experimental Concourse ā¤ ļøK8s runtime project. We also had a great time talking to companies who were using Concourse to continuously do things in the cloud. One of my favorite talks was from Jason Immerman and Derek Van Assche from Zipcar (Concourse All the Things, All the Time); really inspirational stuff!

Concourse Update (April 9ā€“13)

Concourse v3.11.0 came out today! Go get it: https://concourse-ci.org/download.html#v3110 On another note: Iā€™ve noticed some interesting articles and guides come out on writing custom resources for Concourse. Thereā€™s one on the Pivotal blog https://content.pivotal.io/blog/developing-a-custom-concourse-resource and another from fellow Medium writer Shin Myung Yoon (https://itnext.io/writing-a-custom-resource-for-concourse-detecting-pull-request-close-merge-events-e40468eb2a81)! Topher Bullock and I will also be travelling to Boston next week for CF Summit 2018. Weā€™ll be around to meet some Pivotal PCF customers and answer questions about Concourse.

Concourse Updates (April 2ā€“6)

If you havenā€™t done so already please check out Alex Suraciā€™s recent update post on ā€œA renewed focus & community changesā€. It covers all the recent changes that weā€™ve been making; starting with the new styling of the website, our new discussion forum, and our migration to Discord chat. Specifically, weā€™ve been getting some mixed feedback on the new format of the site. Some folks love it, other folks miss the highly visual styling of the old site.

A renewed focus & community changes

Phew, weā€™ve been busy for the past couple of months! Thereā€™s a lot to give yā€™all an update on. New website First off, check out our new website! Weā€™ve completely redesigned it and redone how we organize the documentation, in hopes that itā€™ll be much easier to find what youā€™re looking for. We also hope that the new style, language, and tone will feel a bit more inclusive and humble. For example, we got rid of the ā€œConcourse vs.

Concourse Update (April 19ā€“23)

Hi folks, Had an interesting week talking to customers about how we might improve their Concourse operations and deployments. More info on that soon! On to the update: UX Fixed an issue with timestamps #2088 Core Continued our refactoring of the API to support dex and users #1888 Runtime Finished the issue around custom resources on tagged workers, it should work now #1371 Restricted the list of allowed TLS ciphers for more security checkboxing #1997 Design

Concourse Update (Mar 12ā€“16)

Its been a first week in a long time where we were back to full strength and fully co-located. It was nice! Oh, and Concourse v3.9.2 was released this week as well,check it out! On to our update: UX Started to make our dashboard a bit more mobile friendly #1712 Started to tackle the problem where our dashboard holds too many open connections #1806 Core Removed a dependancy to provide an external URL for fly execute #2069.

Concourse Update (Mar 5ā€“9)

Whelp, that felt like a long week. If you havenā€™t heard the news by now you should definitely read Alex Suraciā€™s post regarding our domain. I want to take this time to thank the Concourse fans out there who offered their help, support, and positive vibes throughout the whole ordeal. The team here really appreciates it šŸ™ Luckily for us this event didnā€™t consume our entire engineering team. We WERE able to get some issues resolved this week and are planning for an imminent release of Concourse 3.

Weā€™re switchinā€™ domains.

(UPDATE March 9 @ ~10 AM: The old domain appears to now be hosting a very old snapshot of our website. This is either targeted or part of a phishing scam. Do not go to it.) Well, that sucked. Wednesday morning I woke up to a ton messages because Concourseā€™s site was gone, and in its place was a blank domain registrar placeholder. Before you say anything, I totally remembered to renew the domain.

Concourse Update (Feb 26 ā€” Mar2)

After some wrestling with our production pipelines last week we managed to release a patch update in the form of Concourse v3.9.1. Weā€™ve fixed some of the reported bugs from the previous release (3.9.0) so definitely go and check it out! I donā€™t have much else regarding updates this week so hereā€™s a fun fact for you to chew on: did you know that Concourse uses Concourse to deploy Concourse? Its true!

Concourse Update (Feb 20ā€“23)

Monday, Feb 19 was Family Day for us here in Canada, so its been a relatively short work week for the Concourse team. With the release of v3.9.0 last week, weā€™ve gotten some reports of new bugs and issues, so thanks to everyone who reported them in via our GitHub issues and Slack. Please make sure to check the updated release notes (here) for the full details! Weā€™re planning to cut a new patch release early next week with some of the fixes to the reported issues.

Concourse Update (Feb 12ā€“16)

If you havenā€™t heard the news by now, we released Concourse v3.9.0 this week šŸŽ‰šŸŽ‰šŸŽ‰! Two of the top-line features in this release are: Concourse will now automatically propagate certificates from the worker machine into resource containers (GH issue #1027) Improved btrfs volume driver stability. So if youā€™re getting hit hard by overlay weirdness, Iā€™d suggest you give the btrfs driver another shot! To find out what else weā€™ve packed into this release, Iā€™d encourage you to read the full release notes on the concourse.

Concourse Updates (Feb 5 ā€” Feb9)

We spent some time this week wrapping up additional testing on our certs management across workers. We also put down some of work on Spaces this week to play around with something fun: a high density dashboard view. A lot of you have been asking us when Concourse v3.9.0 will be available, and the answer is: very soon! On to the update: Features UX High Density View! Original Git issue #1899 and a demo version of it up and running can be found in our production environment Merged PR #227, thanks for the contribution SwamWithTurtles!

Concourse Updates (Jan 29 ā€” Feb 2, 2018)

As a Product Manager at Pivotal, one of my responsibilities is to write weekly updates to let Pivots know what the Concourse team has been up to for the past week. When the Concourse team got together earlier this month for our 2018 planning, we decided that we should be sharing these updates with our community as a whole. So, without further ado, hereā€™s our first update of 2018! Features UX

Concourse at SpringOne 2017

Topher Bullock, Lindsay Auchinachie, Alex Suraci and I will be travelling to San Francisco next week to attend the SpringOne Platform 2017 Conference. Not only are there a lot of exciting Spring talks this year, but there will be a lot of CI/CD talks from some amazing speakers. Unfortunately none of the Concourse core contribution team will be speaking at SpringOne this year; but that doesnā€™t mean thereā€™s a shortage of Concourse-related talks.

Sneak Peek: Spatial Resources

If youā€™ve been paying close attention to our issues on GitHub you may have noticed a small flurry of activity around one specific issue: #1707 Spike: spatial resource flows. What are Spatial Resources Flows (aka Space)? The first reference to ā€œspatialā€ resources came up in a proposal between Alex Suraci and Christopher Hendrix for Multi-branch workflows ( #1172). In that issue we focused specifically on one recurring problem: itā€™s a real pain to deal with the Git resource when you have multiple branches representing different streams of work.

Build Page Improvements

Concourse v3.6.0 comes with two new features on the build output page: timestamps and keyboard shortcuts. Timestamps and OutputĀ Sharing When looking at the build page, you will now see timestamps reported against each line of output using your browserā€™s reported timezone. As you hover over the timestamp, you can select single line of output or you can SHIFT select multiple lines of output. Youā€™ll also notice that the build page URL is updated to reflect the lines you have selected.

How the Concourse Team Organize Issues

As the Concourse team continues to grow in size and in the # of incoming issues, the team has been experimenting with new ways of managing our backlog. So far we have tried three different setups: GitHub issues + Customs/Tracksuit + Pivotal Tracker GitHub issues + aggressive labelling + CodeTree GitHub issues + GitHub Projects Weā€™ve been using the third setup, GitHub issues + GitHub Projects, for the past few months and weā€™ve been mildly happy with the experience.

The Concourse Crew (2017)

In 2014 the Concourse CI project started with just two engineers; Alex Suraci and Chris Brown. At the time, both Alex and Chris were working on the Pivotal Cloud Foundry team. Over time, they became increasingly frustrated by existing CI/CD solutions. In response, Alex and Chris worked on designing a new CI/CD system in their spare time; imagining a new type of CI/CD system that would treat pipelines as first class citizens.