Skip to content

Commit 17ac4c3

Browse files
czentgrrobandpdx
authored and
tdcmeehan
committed
[native] Convert CircleCI jobs for PrestoC++ to Github actions
This pull request converts the CircleCI workflows to GitHub actions workflows. The workflows use the standard runners with Ubuntu 22.04. Some modifications were made to allow a build with the limited resources available: |name|build type|comments|expected runtime (no cache)|unit tests|E2E tests| |----|----------|--------|----------------|----------|---------| | prestocpp-linux-build | debug | All adapters enabled | 45m | N | N | | prestocpp-linux-build-and-unit-test | release | JWT, remote function, Parquet enabled | 90m | Y | Y | | prestocpp-macos-build | debug | Setup script plus build | 160m | N | N | | prestocpp-linux-adapters-build | release | Adapters and release build | 120m | N | N | Caching is currently enabled for the prestocpp-linux-build-and-unit-test. If the cache can be used, the time to run tests drops to 50m. Additional workflows are prestocpp-format-check and prestocpp-header-check That run in a few seconds. For the original conversion see: #21545 Co-authored-by: Rob Anderson <[email protected]>
1 parent 901da6d commit 17ac4c3

5 files changed

+597
-0
lines changed

‎.github/workflows/prestocpp-format-and-header-check.yml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: prestocpp-format-and-header-check
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
paths:
7+
- 'presto-native-execution/**'
8+
- '.github/workflows/prestocpp-format-and-header-check.yml'
9+
10+
jobs:
11+
prestocpp-format-and-header-check:
12+
runs-on: ubuntu-latest
13+
container:
14+
image: public.ecr.aws/oss-presto/velox-dev:check
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Fix git permissions
19+
# Usually actions/checkout does this but as we run in a container
20+
# it doesn't work
21+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
22+
23+
- name: Check formatting
24+
run: |
25+
git fetch origin master
26+
cd presto-native-execution
27+
make format-check
28+
29+
- name: Check license headers
30+
run: |
31+
git fetch origin master
32+
cd presto-native-execution
33+
make header-check

‎.github/workflows/prestocpp-linux-adapters-build.yml

+91
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
name: prestocpp-linux-adapters-build
2+
3+
on:
4+
workflow_dispatch:
5+
# Disable the automatic execution on PR because it currently will run out of disk space and
6+
# we will address this subsequently.
7+
# - use smaller image - in the works
8+
# - remove the adapters downloaded files after install - JWT needs fixing because of the cmake files end up
9+
#pull_request:
10+
# paths:
11+
# - 'presto-native-execution/scripts/**'
12+
# - '.github/workflows/prestocpp-linux-adapters-build.yml'
13+
14+
jobs:
15+
prestocpp-linux-adapters-build:
16+
runs-on: ubuntu-22.04
17+
container:
18+
image: prestodb/presto-native-dependency:0.290-20241014120930-e1fc090
19+
env:
20+
CCACHE_DIR: "${{ github.workspace }}/ccache"
21+
steps:
22+
- uses: actions/checkout@v4
23+
24+
- name: Fix git permissions
25+
# Usually actions/checkout does this but as we run in a container
26+
# it doesn't work
27+
run: git config --global --add safe.directory ${GITHUB_WORKSPACE}
28+
29+
- name: Update submodules
30+
run: |
31+
cd presto-native-execution
32+
make submodules
33+
34+
- name: Build all adapter dependencies
35+
run: |
36+
mkdir -p ${GITHUB_WORKSPACE}/adapter-deps/install
37+
mkdir -p ${GITHUB_WORKSPACE}/adapter-deps/download
38+
source /opt/rh/gcc-toolset-12/enable
39+
set -xu
40+
cd presto-native-execution
41+
export DEPENDENCY_DIR=${GITHUB_WORKSPACE}/adapter-deps/download
42+
export INSTALL_PREFIX=${GITHUB_WORKSPACE}/adapter-deps/install
43+
PROMPT_ALWAYS_RESPOND=n ./velox/scripts/setup-adapters.sh
44+
PROMPT_ALWAYS_RESPOND=n ./scripts/setup-adapters.sh
45+
46+
- name: Install Github CLI for using apache/infrastructure-actions/stash
47+
run: |
48+
curl -L https://github.com/cli/cli/releases/download/v2.63.2/gh_2.63.2_linux_amd64.rpm > gh_2.63.2_linux_amd64.rpm
49+
rpm -iv gh_2.63.2_linux_amd64.rpm
50+
51+
- uses: apache/infrastructure-actions/stash/restore@4ab8682fbd4623d2b4fc1c98db38aba5091924c3
52+
with:
53+
path: '${{ env.CCACHE_DIR }}'
54+
key: ccache-prestocpp-linux-adapters-build
55+
56+
- name: Zero ccache statistics
57+
run: ccache -sz
58+
59+
- name: Build engine
60+
run: |
61+
source /opt/rh/gcc-toolset-12/enable
62+
cd presto-native-execution
63+
cmake \
64+
-B _build/release \
65+
-GNinja \
66+
-DTREAT_WARNINGS_AS_ERRORS=1 \
67+
-DENABLE_ALL_WARNINGS=1 \
68+
-DCMAKE_BUILD_TYPE=Release \
69+
-DPRESTO_ENABLE_PARQUET=ON \
70+
-DPRESTO_ENABLE_S3=ON \
71+
-DPRESTO_ENABLE_GCS=ON \
72+
-DPRESTO_ENABLE_ABFS=OFF \
73+
-DPRESTO_ENABLE_HDFS=ON \
74+
-DPRESTO_ENABLE_REMOTE_FUNCTIONS=ON \
75+
-DPRESTO_ENABLE_JWT=ON \
76+
-DPRESTO_STATS_REPORTER_TYPE=PROMETHEUS \
77+
-DPRESTO_MEMORY_CHECKER_TYPE=LINUX_MEMORY_CHECKER \
78+
-DPRESTO_ENABLE_TESTING=OFF \
79+
-DCMAKE_PREFIX_PATH=/usr/local \
80+
-DThrift_ROOT=/usr/local \
81+
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
82+
-DMAX_LINK_JOBS=4
83+
ninja -C _build/release -j 4
84+
85+
- name: Ccache after
86+
run: ccache -s
87+
88+
- uses: apache/infrastructure-actions/stash/save@4ab8682fbd4623d2b4fc1c98db38aba5091924c3
89+
with:
90+
path: '${{ env.CCACHE_DIR }}'
91+
key: ccache-prestocpp-linux-adapters-build

0 commit comments

Comments
 (0)