Skip to content

fix(hermetic-build): obtain gapic-generator-java locally on release branch #2023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 21, 2023
Merged

fix(hermetic-build): obtain gapic-generator-java locally on release branch #2023

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
81535be
fix(hermetic-build): obtain gapic-generator-java locally on release b…
diegomarquezp Sep 21, 2023
2fd17ea
find generator locally first
diegomarquezp Sep 21, 2023
f560f02
copy_from to return value instead of `exit`
diegomarquezp Sep 21, 2023
7f00dc7
remove redundant variable utilization
diegomarquezp Sep 21, 2023
67e1dfb
improve copy_from doc
diegomarquezp Sep 21, 2023
c549795
fix unit tests
diegomarquezp Sep 21, 2023
cb4ea74
improve readability of `copy_from`
diegomarquezp Sep 21, 2023
7ddaddb
invert result of copy_from
diegomarquezp Sep 21, 2023
c415293
improve log messages in download_generator_artifact
diegomarquezp Sep 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions library_generation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ You can find the released version of gapic-generator-java in [maven central](htt

Use `--gapic_generator_version` to specify the value.

Note that you can specify a `SNAPSHOT` version as long as you have build the SNAPSHOT of gapic-generator-java in your maven
local repository.
Note that you can specify any non-published version (e.g. a SNAPSHOT) as long as you have installed it in your maven
local repository. The script will search locally first.

### protobuf_version (optional)
You can find the released version of protobuf in [GitHub](https://github.com/protocolbuffers/protobuf/releases/).
Expand Down
12 changes: 8 additions & 4 deletions library_generation/test/generate_library_unit_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,11 @@ remove_grpc_version_test() {
}

download_generator_success_with_valid_version_test() {
download_generator "2.24.0"
assertFileOrDirectoryExists "gapic-generator-java-2.24.0.jar"
rm "gapic-generator-java-2.24.0.jar"
local version="2.24.0"
local artifact="gapic-generator-java-${version}.jar"
download_generator_artifact "${version}" "${artifact}"
assertFileOrDirectoryExists "${artifact}"
rm "${artifact}"
}

download_generator_failed_with_invalid_version_test() {
Expand All @@ -91,7 +93,9 @@ download_generator_failed_with_invalid_version_test() {
# the other tests can continue executing in the current
# shell.
local res=0
$(download_generator "1.99.0") || res=$?
local version="1.99.0"
local artifact="gapic-generator-java-${version}.jar"
$(download_generator_artifact "${version}" "${artifact}") || res=$?
assertEquals 1 $((res))
}

Expand Down
50 changes: 23 additions & 27 deletions library_generation/utilities.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,19 +75,7 @@ remove_grpc_version() {

download_gapic_generator_pom_parent() {
local gapic_generator_version=$1
if [ ! -f "gapic-generator-java-pom-parent-${gapic_generator_version}.pom" ]; then
if [[ "${gapic_generator_version}" == *"-SNAPSHOT" ]]; then
# copy a SNAPSHOT version from maven local repository.
copy_from "$HOME/.m2/repository/com/google/api/gapic-generator-java-pom-parent/${gapic_generator_version}/gapic-generator-java-pom-parent-${gapic_generator_version}.pom" \
"gapic-generator-java-pom-parent-${gapic_generator_version}.pom"
return
fi
# download gapic-generator-java-pom-parent from Google maven central mirror.
download_from \
"https://maven-central.storage-download.googleapis.com/maven2/com/google/api/gapic-generator-java-pom-parent/${gapic_generator_version}/gapic-generator-java-pom-parent-${gapic_generator_version}.pom" \
"gapic-generator-java-pom-parent-${gapic_generator_version}.pom"
fi
# file exists, do not need to download again.
download_generator_artifact "${gapic_generator_version}" "gapic-generator-java-pom-parent-${gapic_generator_version}.pom" "gapic-generator-java-pom-parent"
}

get_grpc_version() {
Expand All @@ -114,25 +102,31 @@ download_tools() {
local protobuf_version=$2
local grpc_version=$3
local os_architecture=$4
download_generator "${gapic_generator_version}"
download_generator_artifact "${gapic_generator_version}" "gapic-generator-java-${gapic_generator_version}.jar"
download_protobuf "${protobuf_version}" "${os_architecture}"
download_grpc_plugin "${grpc_version}" "${os_architecture}"
popd
}

download_generator() {
download_generator_artifact() {
local gapic_generator_version=$1
local artifact=$2
local project=${3:-"gapic-generator-java"}
if [ ! -f "gapic-generator-java-${gapic_generator_version}.jar" ]; then
if [[ "${gapic_generator_version}" == *"-SNAPSHOT" ]]; then
# copy a SNAPSHOT version from maven local repository.
copy_from "$HOME/.m2/repository/com/google/api/gapic-generator-java/${gapic_generator_version}/gapic-generator-java-${gapic_generator_version}.jar" \
"gapic-generator-java-${gapic_generator_version}.jar"
return
# first, try to fetch the generator locally
local local_fetch_successful=$(copy_from "$HOME/.m2/repository/com/google/api/${project}/${gapic_generator_version}/${artifact}" \
"${artifact}")
if [[ "${local_fetch_successful}" == "false" ]];then
# download gapic-generator-java artifact from Google maven central mirror if not
# found locally
>&2 echo "${artifact} not found locally. Attempting a download from Maven Central"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add logs for successful local fetch or Maven central download? Otherwise it may be misleading that this is the only log we see when we are downloading from Maven Central.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I added a couple success messages for both local and remote fetch

download_from \
"https://maven-central.storage-download.googleapis.com/maven2/com/google/api/${project}/${gapic_generator_version}/${artifact}" \
"${artifact}"
>&2 echo "${artifact} found and downloaded from Maven Central"
else
>&2 echo "${artifact} found copied from local repository (~/.m2)"
fi
# download gapic-generator-java from Google maven central mirror.
download_from \
"https://maven-central.storage-download.googleapis.com/maven2/com/google/api/gapic-generator-java/${gapic_generator_version}/gapic-generator-java-${gapic_generator_version}.jar" \
"gapic-generator-java-${gapic_generator_version}.jar"
fi
}

Expand Down Expand Up @@ -174,17 +168,19 @@ download_from() {
curl -LJ -o "${save_as}" --fail -m 30 --retry 2 "$url" || download_fail "${save_as}" "${repo}"
}

# copies the specified file in $1 to $2
# will return "true" if the copy was successful
copy_from() {
local local_repo=$1
local save_as=$2
cp "${local_repo}" "${save_as}" || \
download_fail "${save_as}" "maven local"
copy_successful=$(cp "${local_repo}" "${save_as}" && echo "true" || echo "false")
echo "${copy_successful}"
}

download_fail() {
local artifact=$1
local repo=${2:-"maven central mirror"}
>&2 echo "Fail to download ${artifact} from ${repo} repository. Please install ${artifact} first if you want to download a SNAPSHOT."
>&2 echo "Fail to download ${artifact} from ${repo} repository. Please install ${artifact} first if you want to use a non-published artifact."
exit 1
}

Expand Down