Skip to content

Commit 648e82c

Browse files
HoverbearHoverbear
authored
chore(platforms): Migrate x86 gnu target to cross, add new aarch64 gnu target (no sasl) (#3657)
* Add initial cross adoption Signed-off-by: Ana Hobden <[email protected]> * Add archiving Signed-off-by: Ana Hobden <[email protected]> * Clean up the makefile a bit and get tarballs working Signed-off-by: Ana Hobden <[email protected]> * checkpoint Signed-off-by: Ana Hobden <[email protected]> * wip Signed-off-by: Ana Hobden <[email protected]> * Add to CI Signed-off-by: Ana Hobden <[email protected]> * Add autoinstall to CI Signed-off-by: Ana Hobden <[email protected]> * Remove cruft Signed-off-by: Ana Hobden <[email protected]> * Can't test yet Signed-off-by: Ana Hobden <[email protected]> * Autoinstall in e2e Signed-off-by: Ana Hobden <[email protected]> * Readd docker image Signed-off-by: Ana Hobden <[email protected]> * More comments Signed-off-by: Ana Hobden <[email protected]> * Set right CI task Signed-off-by: Ana Hobden <[email protected]> * Add note about TZ issue Signed-off-by: Ana Hobden <[email protected]>
1 parent b28eb18 commit 648e82c

File tree

6 files changed

+152
-49
lines changed

6 files changed

+152
-49
lines changed

‎.github/workflows/e2e.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ on:
4040

4141
env:
4242
VERBOSE: true
43+
AUTOINSTALL: true
4344
RUST_TEST_THREADS: 1
4445
TEST_LOG: vector=debug
4546
RUST_BACKTRACE: full

‎.github/workflows/tests.yml

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ on:
3636

3737
env:
3838
VERBOSE: true
39+
AUTOINSTALL: true
3940
RUST_TEST_THREADS: 1
4041
TEST_LOG: vector=debug
4142
RUST_BACKTRACE: full
@@ -55,7 +56,7 @@ jobs:
5556
access_token: ${{ secrets.GITHUB_TOKEN }}
5657

5758
test-linux:
58-
name: Unit - Linux
59+
name: Unit - x86_64-unknown-linux-gnu
5960
runs-on: ubuntu-20.04
6061
if: |
6162
!contains(github.event.pull_request.labels.*.name, 'ci-condition: skip')
@@ -72,6 +73,46 @@ jobs:
7273
- run: make slim-builds
7374
- run: make test
7475

76+
# TODO: Upgrade to test once the tz issues are solved.
77+
# https://github.com/timberio/vector/issues/3771
78+
build-x86_64-unknown-linux-gnu:
79+
name: Unit - x86_64-unknown-linux-gnu
80+
runs-on: ubuntu-20.04
81+
if: |
82+
!contains(github.event.pull_request.labels.*.name, 'ci-condition: skip')
83+
steps:
84+
- uses: actions/checkout@v2
85+
- uses: actions/cache@v2
86+
with:
87+
path: |
88+
~/.cargo/registry
89+
~/.cargo/git
90+
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
91+
- run: sudo bash scripts/environment/bootstrap-ubuntu-20.04.sh
92+
- run: bash scripts/environment/prepare.sh
93+
- run: make slim-builds
94+
- run: make build-x86_64-unknown-linux-gnu
95+
96+
# TODO: Upgrade to test once the tz issues are solved.
97+
# https://github.com/timberio/vector/issues/3771
98+
build-aarch64-unknown-linux-gnu:
99+
name: Unit - aarch64-unknown-linux-gnu
100+
runs-on: ubuntu-20.04
101+
if: |
102+
!contains(github.event.pull_request.labels.*.name, 'ci-condition: skip')
103+
steps:
104+
- uses: actions/checkout@v2
105+
- uses: actions/cache@v2
106+
with:
107+
path: |
108+
~/.cargo/registry
109+
~/.cargo/git
110+
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
111+
- run: sudo bash scripts/environment/bootstrap-ubuntu-20.04.sh
112+
- run: bash scripts/environment/prepare.sh
113+
- run: make slim-builds
114+
- run: make build-aarch64-unknown-linux-gnu
115+
75116
test-mac:
76117
name: Unit - Mac
77118
# Full CI suites for this platform were only recently introduced.

‎Cargo.toml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ publish = false
1212
[profile.bench]
1313
debug = true
1414

15+
[profile.release]
16+
opt-level = 3
17+
debug = false
18+
lto = "fat"
19+
codegen-units = 1
20+
1521
[package.metadata.deb]
1622
maintainer-scripts = "distribution/debian/scripts/"
1723
conf-files = ["/etc/vector/vector.toml"]
@@ -207,6 +213,12 @@ default-cmake = ["sources", "transforms", "sinks", "vendor-all", "unix", "leveld
207213
# TODO: Enable SASL https://github.com/timberio/vector/pull/3081#issuecomment-659298042
208214
default-msvc = ["sources", "transforms", "sinks", "vendor-openssl", "vendor-libz", "leveldb-cmake", "rdkafka-cmake"]
209215

216+
# Target specific release features.
217+
# The `make` tasks will select this according to the appropriate triple.
218+
# Use this section to turn off or on specific features for specific triples.
219+
target-x86_64-unknown-linux-gnu = ["sources", "transforms", "sinks", "vendor-all", "unix", "leveldb-cmake", "rdkafka-cmake"]
220+
target-aarch64-unknown-linux-gnu = ["sources", "transforms", "sinks", "vendor-openssl", "vendor-libz", "unix", "leveldb-cmake", "rdkafka-cmake"]
221+
210222
# Enables features that work only on systems providing `cfg(unix)`
211223
unix = ["jemallocator"]
212224
# These are **very** useful on Cross compilations!

‎Makefile

Lines changed: 95 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: $(MAKECMDGOALS) all
1+
# .PHONY: $(MAKECMDGOALS) all
22
.DEFAULT_GOAL := help
33
RUN := $(shell realpath $(shell dirname $(firstword $(MAKEFILE_LIST)))/scripts/docker-compose-run.sh)
44

@@ -19,6 +19,8 @@ export SCOPE ?= ""
1919
export AUTOSPAWN ?= true
2020
# Override to control if services are turned off after integration tests.
2121
export AUTODESPAWN ?= ${AUTOSPAWN}
22+
# Override autoinstalling of tools. (Eg `cargo install`)
23+
export AUTOINSTALL ?= false
2224
# Override to true for a bit more log output in your environment building (more coming!)
2325
export VERBOSE ?= false
2426
# Override to set a different Rust toolchain
@@ -57,6 +59,10 @@ FORMATTING_BEGIN_YELLOW = \033[0;33m
5759
FORMATTING_BEGIN_BLUE = \033[36m
5860
FORMATTING_END = \033[0m
5961

62+
# "One weird trick!" https://www.gnu.org/software/make/manual/make.html#Syntax-of-Functions
63+
EMPTY:=
64+
SPACE:= ${EMPTY} ${EMPTY}
65+
6066
help:
6167
@printf -- "${FORMATTING_BEGIN_BLUE} __ __ __${FORMATTING_END}\n"
6268
@printf -- "${FORMATTING_BEGIN_BLUE} \ \ / / / /${FORMATTING_END}\n"
@@ -175,22 +181,83 @@ build-dev: ## Build the project in development mode (Supports `ENVIRONMENT=true`
175181

176182
build-all: build-x86_64-unknown-linux-musl build-aarch64-unknown-linux-musl ## Build the project in release mode for all supported platforms
177183

178-
build-x86_64-unknown-linux-gnu: ## Build dynamically linked binary in release mode for the x86_64 architecture
179-
$(RUN) build-x86_64-unknown-linux-gnu
184+
build-x86_64-unknown-linux-gnu: target/x86_64-unknown-linux-gnu/release/vector ## Build a release binary for the x86_64-unknown-linux-gnu triple.
185+
@echo "Output to ${<}"
186+
187+
build-aarch64-unknown-linux-gnu: target/aarch64-unknown-linux-gnu/release/vector ## Build a release binary for the aarch64-unknown-linux-gnu triple.
188+
@echo "Output to ${<}"
180189

181190
build-x86_64-unknown-linux-musl: ## Build static binary in release mode for the x86_64 architecture
182191
$(RUN) build-x86_64-unknown-linux-musl
183192

184193
build-aarch64-unknown-linux-musl: load-qemu-binfmt ## Build static binary in release mode for the aarch64 architecture
185194
$(RUN) build-aarch64-unknown-linux-musl
186195

196+
##@ Cross Compiling
197+
.PHONY: cross-enable
198+
cross-enable: cargo-install-cross
199+
200+
.PHONY: CARGO_HANDLES_FRESHNESS
201+
CARGO_HANDLES_FRESHNESS:
202+
${EMPTY}
203+
204+
# This is basically a shorthand for folks.
205+
# `cross-anything-triple` will call `cross anything --target triple` with the right features.
206+
.PHONY: cross-%
207+
cross-%: export PAIR =$(subst -, ,$($(strip @):cross-%=%))
208+
cross-%: export COMMAND ?=$(word 1,${PAIR})
209+
cross-%: export TRIPLE ?=$(subst ${SPACE},-,$(wordlist 2,99,${PAIR}))
210+
cross-%: export PROFILE ?= release
211+
cross-%: export RUSTFLAGS += -C link-arg=-s
212+
cross-%: cargo-install-cross
213+
cross ${COMMAND} \
214+
$(if $(findstring release,$(PROFILE)),--release,) \
215+
--target ${TRIPLE} \
216+
--no-default-features \
217+
--features target-${TRIPLE}
218+
219+
target/%/vector: export PAIR =$(subst /, ,$(@:target/%/vector=%))
220+
target/%/vector: export TRIPLE ?=$(word 1,${PAIR})
221+
target/%/vector: export PROFILE ?=$(word 2,${PAIR})
222+
target/%/vector: export RUSTFLAGS += -C link-arg=-s
223+
target/%/vector: cargo-install-cross CARGO_HANDLES_FRESHNESS
224+
cross build \
225+
$(if $(findstring release,$(PROFILE)),--release,) \
226+
--target ${TRIPLE} \
227+
--no-default-features \
228+
--features target-${TRIPLE}
229+
230+
target/%/vector.tar.gz: export PAIR =$(subst /, ,$(@:target/%/vector.tar.gz=%))
231+
target/%/vector.tar.gz: export TRIPLE ?=$(word 1,${PAIR})
232+
target/%/vector.tar.gz: export PROFILE ?=$(word 2,${PAIR})
233+
target/%/vector.tar.gz: target/%/vector CARGO_HANDLES_FRESHNESS
234+
tar --create \
235+
--gzip \
236+
--verbose \
237+
--file target/${TRIPLE}/${PROFILE}/vector.tar.gz \
238+
--transform='s|target/${TRIPLE}/${PROFILE}/|bin/|' \
239+
--transform='s|distribution/|etc/|' \
240+
--transform 's|^|vector-${TRIPLE}/|' \
241+
target/${TRIPLE}/${PROFILE}/vector \
242+
README.md \
243+
LICENSE \
244+
config \
245+
distribution/init.d \
246+
distribution/systemd
247+
187248
##@ Testing (Supports `ENVIRONMENT=true`)
188249

189250
test: ## Run the unit test suite
190251
${MAYBE_ENVIRONMENT_EXEC} cargo test --no-default-features --features ${DEFAULT_FEATURES} ${SCOPE} -- --nocapture
191252

192253
test-all: test test-behavior test-integration ## Runs all tests, unit, behaviorial, and integration.
193254

255+
test-x86_64-unknown-linux-gnu: cross-test-x86_64-unknown-linux-gnu ## Runs unit tests on the x86_64-unknown-linux-gnu triple
256+
${EMPTY}
257+
258+
test-aarch64-unknown-linux-gnu: cross-test-aarch64-unknown-linux-gnu ## Runs unit tests on the aarch64-unknown-linux-gnu triple
259+
${EMPTY}
260+
194261
test-behavior: ## Runs behaviorial test
195262
${MAYBE_ENVIRONMENT_EXEC} cargo run -- test tests/behavior/**/*.toml
196263

@@ -671,21 +738,39 @@ package-x86_64-unknown-linux-gnu-all: package-archive-x86_64-unknown-linux-gnu p
671738
package-aarch64-unknown-linux-musl-all: package-archive-aarch64-unknown-linux-musl package-deb-aarch64 package-rpm-aarch64 # Build all aarch64 MUSL packages
672739

673740
# archives
741+
.PHONY: package-archive
742+
743+
target/artifacts/vector-%.tar.gz: export TRIPLE :=$(@:target/artifacts/vector-%.tar.gz=%)
744+
target/artifacts/vector-%.tar.gz: target/%/release/vector.tar.gz
745+
@echo "Built to ${<}, relocating to ${@}"
746+
@mkdir -p target/artifacts/
747+
@cp -v \
748+
${<} \
749+
${@}
674750

675751
package-archive: build ## Build the Vector archive
676752
${MAYBE_ENVIRONMENT_EXEC} ./scripts/package-archive.sh
677753

754+
.PHONY: package-archive-all
678755
package-archive-all: package-archive-x86_64-unknown-linux-musl package-archive-x86_64-unknown-linux-gnu package-archive-aarch64-unknown-linux-musl ## Build all archives
679756

757+
.PHONY: package-archive-x86_64-unknown-linux-musl
680758
package-archive-x86_64-unknown-linux-musl: build-x86_64-unknown-linux-musl ## Build the x86_64 archive
681759
$(RUN) package-archive-x86_64-unknown-linux-musl
682760

683-
package-archive-x86_64-unknown-linux-gnu: build-x86_64-unknown-linux-gnu ## Build the x86_64 archive
684-
$(RUN) package-archive-x86_64-unknown-linux-gnu
761+
.PHONY: package-archive-x86_64-unknown-linux-gnu
762+
package-archive-x86_64-unknown-linux-gnu: target/artifacts/vector-x86_64-unknown-linux-gnu.tar.gz ## Build an archive of the x86_64-unknown-linux-gnu triple.
763+
@echo "Output to ${<}."
685764

686-
package-archive-aarch64-unknown-linux-musl: build-aarch64-unknown-linux-musl ## Build the aarch64 archive
765+
.PHONY: package-archive-aarch64-unknown-linux-musl
766+
package-archive-aarch64-unknown-linux-musl: build-aarch64-unknown-linux-musl ## Build an archive of the aarch64-unknown-linux-gnu triple.
687767
$(RUN) package-archive-aarch64-unknown-linux-musl
688768

769+
.PHONY: package-archive-aarch64-unknown-linux-gnu
770+
package-archive-aarch64-unknown-linux-gnu: target/artifacts/vector-aarch64-unknown-linux-gnu.tar.gz ## Build the aarch64 archive
771+
@echo "Output to ${<}."
772+
773+
689774
# debs
690775

691776
package-deb: ## Build the deb package
@@ -825,6 +910,10 @@ git-hooks: ## Add Vector-local git hooks for commit sign-off
825910
update-kubernetes-yaml: ## Regenerate the Kubernetes YAML config
826911
${MAYBE_ENVIRONMENT_EXEC} ./scripts/kubernetes-yaml.sh update
827912

913+
cargo-install-%: override TOOL = $(@:cargo-install-%=%)
914+
cargo-install-%:
915+
$(if $(findstring true,$(AUTOINSTALL)),${MAYBE_ENVIRONMENT_EXEC} cargo install ${TOOL} --quiet,)
916+
828917
.PHONY: ensure-has-wasm-toolchain ### Configures a wasm toolchain for test artifact building, if required
829918
ensure-has-wasm-toolchain: target/wasm32-wasi/.obtained
830919
target/wasm32-wasi/.obtained:

‎distribution/rpm/vector.spec

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ URL: %{_url}
2222

2323
%prep
2424
# We are currently in the BUILD dir
25-
tar -xvf %{_sourcedir}/%{_source} --strip-components=2
25+
tar -xvf %{_sourcedir}/%{_source} --strip-components=1
2626
cp -a %{_sourcedir}/systemd/. systemd
2727

2828
%install
@@ -33,7 +33,7 @@ mkdir -p %{buildroot}%{_bindir}
3333
mkdir -p %{buildroot}%{_sysconfdir}/%{_name}
3434
mkdir -p %{buildroot}%{_sharedstatedir}/%{_name}
3535
mkdir -p %{buildroot}%{_unitdir}
36-
cp -a %{_builddir}/bin/. %{buildroot}%{_bindir}
36+
cp -a %{_builddir}/bin/vector %{buildroot}%{_bindir}
3737
cp -a %{_builddir}/config/vector.toml %{buildroot}%{_sysconfdir}/%{_name}/vector.toml
3838
cp -a %{_builddir}/config/vector.spec.toml %{buildroot}%{_sysconfdir}/%{_name}/vector.spec.toml
3939
cp -a %{_builddir}/config/examples/. %{buildroot}%{_sysconfdir}/%{_name}/examples

‎docker-compose.yml

Lines changed: 0 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -22,26 +22,6 @@ services:
2222
user: $USER
2323
command: scripts/build.sh
2424

25-
build-x86_64-unknown-linux-gnu:
26-
build:
27-
context: .
28-
dockerfile: scripts/ci-docker-images/builder-x86_64-unknown-linux-gnu/Dockerfile
29-
args:
30-
USER: $USER
31-
environment:
32-
NATIVE_BUILD: "false"
33-
TARGET: x86_64-unknown-linux-gnu
34-
FEATURES: default
35-
CARGO_TERM_COLOR: always
36-
volumes:
37-
- $PWD:$PWD
38-
- ./target/x86_64-unknown-linux-gnu/cargo/registry:/usr/local/cargo/registry
39-
- ./target/x86_64-unknown-linux-gnu/cargo/git:/usr/local/cargo/git
40-
- ./target/x86_64-unknown-linux-gnu/rustup/tmp:/usr/local/rustup/tmp
41-
working_dir: $PWD
42-
user: $USER
43-
command: scripts/build.sh
44-
4525
build-aarch64-unknown-linux-musl:
4626
build:
4727
context: .
@@ -78,26 +58,6 @@ services:
7858
user: $USER
7959
command: ./scripts/package-archive.sh
8060

81-
package-archive-x86_64-unknown-linux-gnu:
82-
build:
83-
context: .
84-
dockerfile: scripts/ci-docker-images/builder-x86_64-unknown-linux-gnu/Dockerfile
85-
args:
86-
USER: $USER
87-
environment:
88-
NATIVE_BUILD: "false"
89-
TARGET: x86_64-unknown-linux-gnu
90-
FEATURES: ""
91-
CARGO_TERM_COLOR: always
92-
volumes:
93-
- $PWD:$PWD
94-
- ./target/x86_64-unknown-linux-gnu/cargo/registry:/usr/local/cargo/registry
95-
- ./target/x86_64-unknown-linux-gnu/cargo/git:/usr/local/cargo/git
96-
- ./target/x86_64-unknown-linux-gnu/rustup/tmp:/usr/local/rustup/tmp
97-
working_dir: $PWD
98-
user: $USER
99-
command: ./scripts/package-archive.sh
100-
10161
package-archive-aarch64-unknown-linux-musl:
10262
build:
10363
context: .

0 commit comments

Comments
 (0)