Github and Infinite Builds

I heard you like builds so I built inside my build

There is very little educational value in this post, it's more just a mea-culpa admission of guilt! TL;DR though I inadvertently managed to create an infinite loop of builds in Github.

The story begins not at work — and this is the primary thing I should re-learn — but one evening at home, idly hacking away at a simple task involving Github Actions. I was updating an existing build, and just wanted to additionally create a release from the build, for later installation by our Ansible playbooks. I didn't have to be doing it in the evening, it wasn't urgent, I just wanted to tie off a loose-end and knew it was a simple task.

I have done this multiple times before, so I simply copied the relevant lines from another project and pasted them in. It involves a secret, so I created a personal-access token and a new secret to accompany it, and… that's odd, it was rejected because of the GITHUB_ prefix, presumably that is a new change but not to worry, I renamed the secret in the workflow, created a new secret with matching name, pushed my changes and went to bed.

…the disappointment

I woke up to roughly 800 successive builds — and it's even worse than that because I didn't notice at the time! Other members of that organisation received an email overnight for every single one, and strangely I didn't so I only realised when I rolled into work, a bit late to boot.

What happened

Two things:

  • The release process involves creating a tag for the release (as well as uploading 150M .war file).1

  • The workflow was triggered by a push.

To tie it all together, the token I inadvertently created was not a recent change, it was a special token to avoid just this scenario. When my workflow created the tag it generated a push event, which in turns triggers an entire new build. Look, it's right there in the documentation:

When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run. This prevents you from accidentally creating recursive workflow runs. For example, if a workflow run pushes code using the repository's GITHUB_TOKEN, a new workflow will not run even when the repository contains a workflow configured to run when push events occur.

Meta disappointment

So, oops. But more than that, I should have been way smarter than that. I don't feel I have improved as developer in quite some time2, but the single thing I have noticed is: an improved ability to go to bed! We've all that situation where it's late but we feel fine, I'm really close to finding this bug, perhaps if I just try this next thing… and then actually solving it within 5 minutes the next day. Now, I'm disciplined enough to realise that I'm probably just doing stupid things, and going to bed. Trust the process.

I did not do that here; I was programming for the heck of it, when there was no need to. That's the biggest disappointment3, so perhaps if one person avoids this mistake after reading this, I'll feel better!


1

Yes, 800 times.

2

Just because you have imposter syndrome, doesn't mean you're not incompetent.

3

It's actually worse than that! Before cleaning up I carefully wrote an email expressing my sincere apologies, and explaining why it had happened and what I would be doing about it. Then I promptly sent it to the wrong person, thanks to a similar name in my email's auto-complete.


TIL

606 Words

2022-07-17 00:00 +0000

comments powered by Disqus