Skip to content

Commit b8f0a5a

Browse files
jpluscplusmmyitcv
jpluscplusm
authored and
myitcv
committed
content: driving gitlab ci/cd pipelines with CUE
This change adds a new piece of content which shows how to manage a GitLab repo's CI/CD pipeline file in CUE, instead of YAML. Because GitLab still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is pretty lightweight and could do with improving. Part of the problem is that I had to manually write the schema, as GitLab's JSONSchema schema currently confuses `cue import`. I opened cue-lang/cue#2654 to track this. Closes cue-labs#19
1 parent 1552cc5 commit b8f0a5a

File tree

2 files changed

+367
-0
lines changed

2 files changed

+367
-0
lines changed

‎005_gitlab_ci/README.md

+366
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,366 @@
1+
# Driving GitLab CI/CD pipelines with CUE
2+
<sup>by [Jonathan Matthews](https://jonathanmatthews.com)</sup>
3+
4+
This guide explains how to convert a GitLab CI/CD pipeline file from YAML to
5+
CUE, check its contents are valid, and then use CUE's tooling layer to
6+
regenerate YAML.
7+
8+
This is useful because it allows you to switch to CUE as a source of truth for
9+
GitLab pipelines and perform client-side validation, without GitLab needing to
10+
know you're managing your pipelines with CUE.
11+
12+
## Prerequisites
13+
14+
- You have
15+
[CUE installed](https://alpha.cuelang.org/docs/introduction/installation/)
16+
locally. This allows you to run `cue` commands.
17+
- You have a GitLab pipeline file.
18+
- The example shown throughout this guide uses the state of a specific commit
19+
from the
20+
[Flockademic repository](https://gitlab.com/Flockademic/Flockademic/-/blob/8efcea927b10c2773790fe78bb858905a75cf3ef/.gitlab-ci.yml)
21+
on gitlab.com, as linked from
22+
[GitLab's documentation pages](https://docs.gitlab.com/ee/ci/examples/end_to_end_testing_webdriverio/index.html),
23+
but you don't need to use that repository in any way.\
24+
It is used here as it represents a reasonably complex example of a GitLab
25+
pipeline file.
26+
- You have [`git` installed](https://git-scm.com/downloads).
27+
<!-- curl isn't needed until the upstream JSONschema can be imported.
28+
- You have [`curl` installed](https://curl.se/dlwiz/), or can fetch a remote
29+
file some other way.
30+
-->
31+
32+
## Steps
33+
34+
### Convert YAML pipeline to CUE
35+
36+
#### :arrow_right: Begin with a clean git state
37+
38+
Change directory into the root of the repository that contains your GitLab
39+
pipeline file, and ensure you start this process with a clean git state, with
40+
no modified files. For example:
41+
42+
:computer: `terminal`
43+
```sh
44+
cd Flockademic # our example repository
45+
git status # should report "working tree clean"
46+
```
47+
48+
#### :arrow_right: Initialise a CUE module
49+
50+
Initialise a CUE module named after the organisation and repository you're
51+
working with. For example:
52+
53+
:computer: `terminal`
54+
```sh
55+
cue mod init gitlab.com/Flockademic/Flockademic
56+
```
57+
58+
#### :arrow_right: Import YAML pipeline
59+
60+
Use `cue` to import your YAML pipeline file:
61+
62+
:computer: `terminal`
63+
```sh
64+
cue import .gitlab-ci.yml --with-context -p gitlab -f -l pipelines: -l 'strings.TrimSuffix(path.Base(filename),path.Ext(filename))' -o gitlab-ci.cue
65+
```
66+
67+
If your project uses a different name for your pipeline file then use that name
68+
in the above command, and throughout this guide.
69+
70+
Check that a CUE file has been created from your pipeline file. For example:
71+
72+
:computer: `terminal`
73+
```sh
74+
ls {,.}*gitlab-ci*
75+
```
76+
77+
Your output should look similar to this, with a matching YAML and CUE file:
78+
79+
```text
80+
gitlab-ci.cue .gitlab-ci.yml
81+
```
82+
83+
Observe that your file has been imported into the `pipelines` struct at a
84+
location derived from its original file name, by running:
85+
86+
:computer: `terminal`
87+
```sh
88+
head gitlab-ci.cue
89+
```
90+
91+
The output should reflect your pipeline. In our example:
92+
93+
```text
94+
package gitlab
95+
pipelines: ".gitlab-ci": {
96+
image: "node:8.10"
97+
98+
stages: [
99+
"prepare",
100+
"test",
101+
"build-backend",
102+
"deploy-backend",
103+
```
104+
105+
#### :arrow_right: Store CUE pipelines in a dedicated directory
106+
107+
Create a directory called `gitlab` to hold your CUE-based GitLab pipeline
108+
files. For example:
109+
110+
:computer: `terminal`
111+
```sh
112+
mkdir -p internal/ci/gitlab
113+
```
114+
115+
You may change the hierarchy and naming of `gitlab`'s **parent** directories to
116+
suit your repository layout. If you do so, you will need to adapt some commands
117+
and CUE code as you follow this guide.
118+
119+
Move the newly-created CUE pipeline file into its dedicated directory. For example:
120+
121+
:computer: `terminal`
122+
```sh
123+
mv gitlab-ci.cue internal/ci/gitlab
124+
```
125+
126+
### Validate pipeline
127+
128+
#### :arrow_right: Create a pipeline schema
129+
130+
<!-- The upstream schema isn't importable at the moment, cf. https://github.com/cue-lang/cue/issues/2654
131+
132+
Fetch a schema for GitLab pipelines, as defined by the GitLab project, and
133+
place it in the `internal/ci/gitlab` directory:
134+
135+
:computer: `terminal`
136+
```sh
137+
curl -o internal/ci/gitlab/gitlab.cicd.pipeline.schema.json https://gitlab.com/gitlab-org/gitlab/-/raw/d86a7ccc6233aaaf61d9721a537098c3e47fa7c5/app/assets/javascripts/editor/schema/ci.json
138+
```
139+
140+
We use a specific commit from the upstream repository to make sure that this
141+
process is reproducible.
142+
143+
-->
144+
145+
Create a basic CUE schema for GitLab pipelines, adapted from [GitLab's CI/CD
146+
documentation](https://docs.gitlab.com/ee/ci/yaml/index.html), and place it in
147+
the `internal/ci/gitlab` directory:
148+
149+
:floppy_disk: `internal/ci/gitlab/gitlab.cicd.pipeline.schema.cue`
150+
151+
```CUE
152+
package gitlab
153+
154+
_globalKeywords: ["default", "include", "stages", "variables", "workflow"]
155+
_#job: _
156+
#Pipeline: {
157+
default?: {
158+
after_script?: _
159+
artifacts?: _
160+
before_script?: _
161+
cache?: _
162+
hooks?: _
163+
id_tokens?: _
164+
image?: _
165+
interruptible?: _
166+
retry?: _
167+
services?: _
168+
tags?: _
169+
timeout?: _
170+
}
171+
include?: _
172+
stages?: [...string]
173+
variables?: _
174+
workflow?: _
175+
[and([ for _, v in _globalKeywords {!=v}])]: _#job
176+
}
177+
```
178+
179+
| :grey_exclamation: Info :grey_exclamation: |
180+
|:---------------------------------------------- |
181+
| It would be great if we could use [GitLab's authoritative pipeline schema](https://gitlab.com/gitlab-org/gitlab/-/raw/master/app/assets/javascripts/editor/schema/ci.json), here. Unfortunately, CUE's JSONSchema support can't import it currently. This is being tracked in [CUE Issue #2654](https://github.com/cue-lang/cue/issues/2654), and this guide should be updated once the issue is resolved.
182+
183+
#### :arrow_right: Apply the schema
184+
185+
We need to tell CUE to apply the schema to the pipeline.
186+
187+
To do this we'll create a file at `internal/ci/gitlab/pipelines.cue` in our
188+
example.
189+
190+
However, if your earlier pipeline import *already* created a file with that
191+
same path and name, then simply select a different CUE filename that *doesn't*
192+
already exist. Place the file in the `internal/ci/gitlab/` directory.
193+
194+
:floppy_disk: `internal/ci/gitlab/pipelines.cue`
195+
196+
```
197+
package gitlab
198+
199+
// each member of the pipelines struct must be a valid #Pipeline
200+
pipeline: [_]: #Pipeline
201+
```
202+
203+
### Generate YAML from CUE
204+
205+
#### :arrow_right: Create a CUE tool file
206+
207+
Create a CUE "tool" file in `internal/ci/gitlab/` and adapt the
208+
element commented with `TODO`:
209+
210+
:floppy_disk: `internal/ci/gitlab/ci_tool.cue`
211+
```CUE
212+
package gitlab
213+
214+
import (
215+
"path"
216+
"encoding/yaml"
217+
"tool/file"
218+
)
219+
220+
_goos: string @tag(os,var=os)
221+
222+
// Regenerate pipeline files
223+
command: regenerate: {
224+
pipeline_files: {
225+
// TODO: update _toolFile to reflect the directory hierarchy containing this file.
226+
// TODO: update _pipelineDir to reflect the directory containing your pipeline file.
227+
let _toolFile = "internal/ci/gitlab/ci_tool.cue"
228+
let _pipelineDir = path.FromSlash(".", path.Unix)
229+
let _donotedit = "Code generated by \(_toolFile); DO NOT EDIT."
230+
231+
for _pipelineName, _pipelineConfig in pipelines
232+
let _pipelineFile = _pipelineName + ".yml"
233+
let _pipelinePath = path.Join([_pipelineDir, _pipelineFile]) {
234+
let delete = {
235+
"Delete \(_pipelinePath)": file.RemoveAll & {path: _pipelinePath}
236+
}
237+
delete
238+
create: file.Create & {
239+
$after: delete
240+
filename: _pipelinePath
241+
contents: "# \(_donotedit)\n\n\(yaml.Marshal(_pipelineConfig))"
242+
}
243+
}
244+
}
245+
}
246+
```
247+
248+
Make the modifications indicated by the `TODO` comments.
249+
250+
This tool will export your CUE-based pipeline back into its required YAML file,
251+
on demand.
252+
253+
#### :arrow_right: Test the CUE tool file
254+
255+
With the modified `ci_tool.cue` file in place, check that the `regenerate`
256+
command is available **from a shell sitting at the repository root**. For
257+
example:
258+
259+
:computer: `terminal`
260+
```sh
261+
cd $(git rev-parse --show-toplevel) # make sure we're sitting at the repository root
262+
cue help cmd regenerate ./internal/ci/gitlab # the "./" prefix is required
263+
```
264+
265+
The output of the `cue help` command **must** begin with the following:
266+
267+
```text
268+
Regenerate all pipeline files
269+
270+
Usage:
271+
cue cmd regenerate [flags]
272+
[... output continues ...]
273+
```
274+
275+
| :exclamation: WARNING :exclamation: |
276+
|:--------------------------------------- |
277+
| If you *don't* see the usage explanation for the `regenerate` command (or if you receive an error message) then your tool file isn't set up as CUE requires. Double check the contents of the `ci_tool.cue` file and the modifications you made to it, as well as its location in the repository. Ensure the filename is *exactly* `ci_tool.cue`. Make sure you've followed all the steps in this guide, and that you invoked the `cue help` command from the root of the repository.
278+
279+
#### :arrow_right: Regenerate the YAML pipeline file
280+
281+
Run the `regenerate` command to produce a YAML pipeline file from CUE. For
282+
example:
283+
284+
:computer: `terminal`
285+
```sh
286+
cue cmd regenerate ./internal/ci/gitlab # the "./" prefix is required
287+
```
288+
289+
#### :arrow_right: Audit changes to the YAML pipeline file
290+
291+
Check that your YAML pipeline file has a single *material* change from the
292+
original:
293+
294+
:computer: `terminal`
295+
```sh
296+
git diff .gitlab-ci.yml
297+
```
298+
299+
Your output should look similar to the following example:
300+
301+
```diff
302+
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
303+
index d0eaf801..d0a309e3 100644
304+
--- a/.gitlab-ci.yml
305+
+++ b/.gitlab-ci.yml
306+
@@ -1,5 +1,6 @@
307+
-image: node:8.10
308+
+# Code generated by internal/ci/gitlab/ci_tool.cue; DO NOT EDIT.
309+
310+
+image: node:8.10
311+
stages:
312+
- prepare
313+
- test
314+
[ ... output continues ... ]
315+
```
316+
317+
The main change in each YAML file is the addition of a header that warns the
318+
reader not to edit the file directly.
319+
320+
Your diff might also contain some YAML reformatting (with the number of leading
321+
spaces having been changed in nested structures) but this won't make a
322+
difference to the underlying meaning of the file.
323+
324+
Additionally, any comments in the original YAML file will now be found *only*
325+
in the CUE source file - which is important as that's the only file that you'll
326+
be manually changing, from now on.
327+
328+
#### :arrow_right: Add and commit files to git
329+
330+
Add your files to git. For example:
331+
332+
:computer: `terminal`
333+
```sh
334+
git add .gitlab-ci.yml internal/ci/gitlab/ cue.mod/module.cue
335+
```
336+
337+
Make sure to include your slightly modified YAML pipeline file, wherever you
338+
store it, along with all the new files in `internal/ci/gitlab/` and your
339+
`cue.mod/module.cue` file.
340+
341+
Commit your files to git, with an appropriate commit message:
342+
343+
:computer: `terminal`
344+
```sh
345+
git commit -m "ci: create CUE sources for GitLab CI/CD pipelines"
346+
```
347+
348+
## Conclusion
349+
350+
**Well done - your GitLab CI/CD pipeline file has been imported into CUE!**
351+
352+
It can now be managed using CUE, leading to safer and more predictable changes.
353+
The use of a schema to check your pipeline means that you will catch and fix
354+
certain types of mistake earlier than before, without waiting for the slow "git
355+
add/commit/push; check if CI fails" cycle.
356+
357+
From now on, each time you make a change to a CUE pipeline file, immediately
358+
regenerate the YAML files required by GitLab CI/CD, and commit your changes to
359+
all the CUE and YAML files. For example:
360+
361+
:computer: `terminal`
362+
```sh
363+
cue cmd regenerate ./internal/ci/gitlab/ # the "./" prefix is required
364+
git add .gitlab-ci.yml internal/ci/gitlab/
365+
git commit -m "ci: added new release pipeline" # example message
366+
```

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ integrating with 3rd-party tools, services, and systems.
99
- 002: [Writing Terraform plan policies with CUE](002_terraform_plan/README.md)
1010
- 003: [Controlling Kubernetes with CUE](003_kubernetes_tutorial/README.md)
1111
- 004: [Managing Mythic Beasts DNS zones with CUE](004_mythic_beasts_dns/README.md)
12+
- 005: [Driving GitLab CI/CD pipelines with CUE](005_gitlab_ci/README.md)
1213

1314
## Contributing
1415

0 commit comments

Comments
 (0)