|
| 1 | +GCB CI tools for Magic Modules and Google Providers |
| 2 | +=== |
| 3 | + |
| 4 | +These tools manage the downstream repositories of [magic-modules](https://github.com/GoogleCloudPlatform/magic-modules), and are collectively referred to as "The Magician". |
| 5 | + |
| 6 | +# CI For Downstream/Magic Modules Developers |
| 7 | +If you're interested in developing the repositories that Magic Modules manages, here are the things you'll want to know. |
| 8 | + |
| 9 | +## What The Magician Does |
| 10 | +The Magician takes the PR you write against Magic Modules and creates the downstream (generated) commits that are a result of your changes. It posts the diffs created by your changes as a comment on your PR, to aid in review. When your PR is merged, it updates the downstream repositories with your changes. |
| 11 | + |
| 12 | +## Your Workflow |
| 13 | +You'll write some code and open a PR. The Magician will run the generator and the downstream tests. If either the generator or the tests fail, you'll see "check failed" status on your PR, which will link to the step that failed. Once all the generation steps succeed, the Magician will comment on your PR with links to review the generated diffs. Your reviewer will review those diffs (as well as your code). Once your PR is approved, simply merge it in - the Magician will ensure that the downstream repositories are updated within about 5 minutes. |
| 14 | + |
| 15 | +# CI For CI Developers |
| 16 | +If you're working on enhancing these CI tools, you'll need to know about the structure of the pipeline and about how to develop and test. |
| 17 | + |
| 18 | +## How the pipeline works |
| 19 | + |
| 20 | +### Generation & Diffing |
| 21 | +The generation / diff pipeline generates the downstream at the PR's merge commit and at the left-side parent (`HEAD~`), which guarantees isolation of exclusively the changes made by the PR in question. It creates commits for the downstreams as before-and-after commits. Those commits don't have relevant git history - they're not meant to be applied - but they are used in a [two-dot diff](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-comparing-branches-in-pull-requests#three-dot-and-two-dot-git-diff-comparisons). This means that if there are no changes to the downstream, you'll see an empty diff. These downstream branches are named `auto-pr-$number-old` and `auto-pr-$number` respectively. |
| 22 | + |
| 23 | +### Downstream Pushing |
| 24 | +The Magician maintains a set of tags called `$REPO-sync` that tracks the Magic Modules commit the downstreams are up to date with. |
| 25 | + |
| 26 | +In effect, this means that each downstream commit will correspond 1:1 to an MM commit. If an MM commit had no changes in a downstream, no commit will be created. We are enforcing squash-merges-only in Magic Modules. |
| 27 | + |
| 28 | +Downstream pushing can only take place if the sync tag points to the commit which precedes the commit that is being pushed. It is possible for the pipeline to get stuck if a commit is merged into Magic Modules which cannot be generated - this happens most often when the Gemfile.lock is updated. We expect the failure case to be somewhat uncommon, but when it happens, you need to: |
| 29 | +1. Submit your PR changing Gemfile.lock. The downstream builder will fail in one of the downstream generation steps, whichever one starts first. |
| 30 | +2. Immediately after your downstream builder job is submitted, prevent submissions to the magic-modules repository. |
| 31 | +3. Once the downstream-builder container is regenerated (about 15 minutes) re-enable submissions to magic-modules and press retry on your downstream builder job. |
| 32 | + |
| 33 | +Disabling submissions to magic-modules can be done through the Admin console in GitHub. |
| 34 | + |
| 35 | +It is safe to have more than one downstream-push running at the same time due to this property, in the event of overruns. Each run will either |
| 36 | +a) make no changes to any downstream and fail |
| 37 | +or |
| 38 | +b) atomically update every downstream to a fast-forward state that represents the appropriate HEAD as of the beginning of the run |
| 39 | + |
| 40 | +#### Something went wrong! |
| 41 | +Don't panic - this is all quite safe and we have fixed it before. We store the state of the pusher tasks in tags (the "sync tags"), one per downstream, and it will be easy to get the system back up and running. You can send a message to Nathan if you are anxious about running through these steps. :) |
| 42 | + |
| 43 | +It's possible for a job to be cancelled or fail in the middle of pushing downstreams in a transient way. The sorts of failures that happen at scale - lightning strikes a datacenter (ours or GitHub's!) or some other unlikely misfortune happens. This has a chance to cause a hiccup in the downstream history, but isn't dangerous. If that happens, the sync tags may need to be manually updated to sit at the same commit, just before the commit which needs to be generated, or some failed tasks might need to be run by hand. |
| 44 | + |
| 45 | +First, check their state: `git fetch origin && git rev-parse origin/tpg-sync origin/tpgb-sync origin/tf-oics-sync origin/tgc-sync` will list the commits for each of the sync tags. (If you have changed the name of the `GoogleCloudPlatform/magic-modules` remote from `origin`, substitute that name instead, such as `git fetch upstream && git rev-parse upstream/tpg-sync upstream/tpgb-sync upstream/tf-oics-sync upstream/tgc-sync`) |
| 46 | + |
| 47 | +In normal, steady-state operation, these tags will all be identical. When a failure occurs, some of them may be one commit ahead of the others. It is rare for any of them to be 2 or more commits ahead of any other. If some of them are one commit ahead of the others, and there is no pusher task currently running, this means you need to reset them by hand and rerun the failed jobs. If they diverge by more than one commit, or a pusher task is currently running, you will need to manually run missing tasks. |
| 48 | + |
| 49 | +### Divergence by zero commits |
| 50 | + |
| 51 | +Just click retry on the failed job in Cloud Build. This is fairly rare, as most failures involve a step failing after another has already succeeded. |
| 52 | + |
| 53 | +### Divergence by exactly one commit. |
| 54 | + |
| 55 | +Find which commit caused the error. This will usually be easy - cloud build lists the commit which triggered a build, so you can probably just use that one. You need to set all the sync tags to the parent of that commit. Say the commit which caused the error is `12345abc`. You can find the parent of that commit with `git rev-parse 12345abc~` (note the `~` suffix). Some of the sync tags are likely set to this value already. For the remainder, simply perform a git push. Assuming that the parent commit is `98765fed`, that would be, e.g. `git push -f origin 98765fed:tf-validator-sync`. |
| 56 | + |
| 57 | +If you are unlucky, there may be open PRs - this only happens if the failure occurred during the ~5 second period surrounding the merging of one of the downstreams. Close those PRs before proceeding to the final step. |
| 58 | + |
| 59 | +Finally, click "retry" on the failed job in Cloud Build. Watch the retried job and see if it succeeds - it should! If it does not, the underlying problem may not have been fixed. |
| 60 | + |
| 61 | +### Divergence by more than one commit. |
| 62 | +This situation is interesting. This means that there was a failure but that it went undetected for a little while, or that there was a failure in a PR which was merged at about the same time as another one. Don't worry, it's still easy to fix! To understand this, you should know a little more about the way the pusher tasks work. For each commit merged in Magic Modules, five pusher tasks (part of one pusher job) start. Each pusher task waits for all previous pusher tasks of the same type to finish. That is, the TPGB pusher always waits for all previous TPGB pushers, etc. |
| 63 | + |
| 64 | +When a task fails, all the other tasks which are part of the same job are cancelled. However, if one of those tasks has already succeeded, the next task of the same kind will be free to start. That means that all subsequent jobs have partially succeeded - they pushed TF-OICS, but not TPGB. You'll need to do the work of the failed tasks yourself in order to restore the steady state. |
| 65 | + |
| 66 | +When this happened the first time, Cameron and Nathan wrote this little shell snippet, which should do it for you. You will need to get the Magician's github token, either by generating a new one (be sure to clean up after yourself when done), by decrypting the value in .ci/gcb-push-downstream.yml as cloudbuild does, or by accessing the token in Google's internal secret store. |
| 67 | + |
| 68 | +``` |
| 69 | +SYNC_TAG=tpgb-sync |
| 70 | +REPO=terraform |
| 71 | +VERSION=beta |
| 72 | +git clone https://github.com/GoogleCloudPlatform/magic-modules fix-gcb-run |
| 73 | +pushd fix-gcb-run |
| 74 | +docker pull gcr.io/graphite-docker-images/downstream-builder; |
| 75 | +for commit in $(git log $SYNC_TAG..main --pretty=%H | tac); do |
| 76 | + git checkout $commit && \ |
| 77 | + docker run -v `pwd`:/workspace -w /workspace -e GITHUB_TOKEN=$MAGICIAN_GITHUB_TOKEN -it gcr.io/graphite-docker-images/downstream-builder downstream $REPO $VERSION $commit || \ |
| 78 | + break; |
| 79 | +done |
| 80 | +``` |
| 81 | + |
| 82 | +In the event of a failure, this will stop running. If it succeeds, update the sync tag with `git push origin HEAD:tpg-sync`. |
| 83 | + |
| 84 | +### Downstream build job is not triggered by commits. |
| 85 | +This is rare but we've seen this happened before. In this case, we need to manually trigger a Cloud Build job by running |
| 86 | +``` |
| 87 | +gcloud builds triggers run build-downstreams --project=graphite-docker-images --substitutions=BRANCH_NAME=<BASE_BRANCH_NAME> --sha=<COMMIT_SHA> |
| 88 | +``` |
| 89 | +You'll need to substitute `<COMMIT_SHA>` with the commit sha that you'd like to trigger the build against and `<BASE_BRANCH_NAME>` with base branch that this commit is pushed into, likely `main` but could be feature branches in some cases. |
| 90 | + |
| 91 | +## Deploying the pipeline |
| 92 | +The code on the PR's branch is used to plan actions - no merge is performed. |
| 93 | +If you are making changes to the workflows, your changes will not trigger a workflow run, because of the risk of an untrusted contributor introducing malicious code in this way. You will need to test locally by using the [cloud build local builder](https://cloud.google.com/cloud-build/docs/build-debug-locally). |
| 94 | +If you are making changes to the containers, your changes will not apply until they are merged in and built - this can take up to 15 minutes. If you need to make a breaking change, you will need to pause the pipeline while the build happens. If you are making changes to both the containers and the workflows and those changes need to be coordinated, you will need to pause the build while the containers build and enforce every open PR be rebased on top of our PR. It is probably better to build in backwards-compatibility into your containers. We recommend a 14 day window - 14 days after your change goes in, you can remove the backwards-compatibility. |
| 95 | + |
| 96 | +Pausing the pipeline is done in the cloud console, by setting the downstream-builder trigger to disabled. You can find that trigger [here](https://console.cloud.google.com/cloud-build/triggers/edit/f80a7496-b2f4-4980-a706-c5425a52045b?project=graphite-docker-images) |
| 97 | + |
| 98 | +## Dependency change handbook: |
| 99 | +If someone (often a bot) creates a PR which updates Gemfile or Gemfile.lock, they will not be able to generate diffs. This is because bundler doesn't allow you to run a binary unless your installed gems exactly match the Gemfile.lock, and since we have to run generation before and after the change, there is no possible container that will satisfy all requirements. |
| 100 | + |
| 101 | +The best approach is |
| 102 | +* Build the `downstream-generator` container locally, with the new Gemfile and Gemfile.lock. This will involve hand-modifying the Dockerfile to use the local Gemfile/Gemfile.lock instead of wget from this repo's `main` branch. You don't need to check in those changes. |
| 103 | +* When that container is built, and while nothing else is running in GCB (wait, if you need to), push the container to GCR, and as soon as possible afterwards, merge the dependency-changing PR. |
| 104 | + |
| 105 | +## Changes to cloud build yaml: |
| 106 | +If changes are made to `gcb-contributor-membership-checker.yml` or `gcb-community-checker.yml` they will not be reflected in presubmit runs for existing PRs without a rebase. This is because these build triggers are linked to pull request creation and not pushes to the PR branch. If changes are needed to these build files they will need to be made in a backwards-compatible manner. Note that changes to other files used by these triggers will be immediately reflected in all PRs, leading to a possible disconnect between the yaml files and the rest of the CI code. |
| 107 | + |
| 108 | +## Historical Note: Design choices & tradeoffs |
| 109 | +* The downstream push doesn't wait for checks on its PRs against downstreams. This may inconvenience some existing workflows which rely on the downstream PR checks. This ensures that merge conflicts never come into play, since the downstreams never have dangling PRs, but it requires some up-front work to get those checks into the differ. If a new check is introduced into the downstream Travis, we will need to introduce it into the terraform-tester container. |
| 110 | +* The downstream push is disconnected from the output of the differ (but runs the same code). This means that the diff which is approved isn't guaranteed to be applied *exactly*, if for instance magic modules' behavior changes on main between diff generation and downstream push. This is also intended to avoid merge conflicts by, effectively, rebasing each commit on top of main before final generation is done. |
| 111 | + * Imagine the following situation: PR A and PR B are opened simultaneously. PR A changes the copyright date in each file to 2020. PR B adds a new resource. PR A is merged seconds before PR B, so they are picked up in the same push-downstream run. The commit from PR B will produce a new file with the 2020 copyright date, even though the diff said 2019, since PR A was merged first. |
| 112 | +* We deleted the submodules. They weren't useful to us and they were annoying to update - they're not in use anymore as far as we know - but it's possible there's some long-forgotten workflow that someone is using which will be damaged. So far, we haven't seen any such issue. |
0 commit comments