Refactor: Improve docker builds & Update to java21 (#4811)

* Refactor: Improve docker builds
* Refactor: Improve non-root security
* Add pip3 integration
* Updated base image
* build: update to java 21
* update docs for java 21
* fix: set allowed mismatched to 10ms in snowflake id test

---------

Co-authored-by: DickPostma <(ID)+(username)@users.noreply.github.com>
Co-authored-by: Morris Swertz <[email protected]>
Co-authored-by: YpeZ <[email protected]>
Co-authored-by: Harm Brugge <[email protected]>

Author: 5 people

.circleci/config.yml

@@ -21,7 +21,7 @@ orbs:
 definitions:
   build_config: &build_config
     docker:
-    - image: molgenis/ci-build:1.2.4
+    - image: molgenis/ci-build:1.3.3
     working_directory: ~/repo
     resource_class: large
     environment:
@@ -30,7 +30,7 @@ definitions:
       TERM: dumb
   test_config: &test_config
     docker:
-    - image: molgenis/ci-build:1.2.4
+    - image: molgenis/ci-build:1.3.3
     - image: postgres:15-alpine
       environment:
         POSTGRES_USER: postgres
@@ -44,7 +44,7 @@ definitions:
       TERM: dumb
   release_config: &release_config
     docker:
-      - image: molgenis/ci-build:1.2.4
+      - image: molgenis/ci-build:1.3.3
       - image: postgres:15-alpine
         environment:
           POSTGRES_USER: postgres

Dockerfile

@@ -1,6 +1,12 @@
-FROM eclipse-temurin:17.0.11_9-jdk-jammy
+FROM ubuntu:24.10
+
+RUN apt update && apt -y upgrade
+RUN apt update && apt -y install python3 python3-pip python3-venv openjdk-21-jre-headless
+
 ARG JAR_FILE
 COPY ${JAR_FILE} app.jar
 EXPOSE 8080
-RUN apt-get update && apt-get install python3 python3-venv -y
+RUN useradd -m molgenis
+
+USER molgenis
 ENTRYPOINT ["java","-jar","app.jar"]

azure-pipelines.yml

@@ -36,11 +36,11 @@ steps:
     sudo update-alternatives --display java
   displayName: list available java versions (for debug purposes)
 - script: |
-    export JAVA_HOME=/usr/lib/jvm/temurin-17-jdk-amd64/
+    export JAVA_HOME=/usr/lib/jvm/temurin-21-jdk-amd64/
     ./gradlew -version
     ./gradlew test jacocoMergedReport
   # sonarqube -Dsonar.login=$(SONAR_TOKEN) -Dsonar.organization=molgenis -Dsonar.host.url=https://sonarcloud.io -Dorg.ajoberstar.grgit.auth.username=$(GITHUB_TOKEN) -Dorg.ajoberstar.grgit.auth.password
-  displayName: run test, ensure we are using java 17 JAVA_HOME
+  displayName: run test, ensure we are using java 21 JAVA_HOME
 
   env:
     MOLGENIS_POSTGRES_USER: molgenis

backend/molgenis-emx2-sql/src/test/java/org/molgenis/emx2/sql/SnowflakeIdGeneratorTest.java

@@ -91,7 +91,7 @@ public void testExtractTimestampFromSnowflake() {
     String snowflakeId = generator.generateId();
 
     long snowflakeTimestamp = SnowflakeIdGenerator.extractTimestamp(snowflakeId);
-    // Allow a mismatch of 1ms
-    assertTrue(Math.abs(currentTime - snowflakeTimestamp) <= 1);
+    // Allow a mismatch of 10ms
+    assertTrue(Math.abs(currentTime - snowflakeTimestamp) <= 10);
   }
 }

build.gradle

@@ -8,12 +8,12 @@ plugins {
     id 'maven-publish'
     id 'com.github.johnrengelman.shadow' version '8.1.1'
     id 'application'
-    id 'com.palantir.docker' version '0.35.0'
+    id 'com.palantir.docker' version '0.36.0'
 }
 
 ext {
     javaMainClass = "org.molgenis.emx2.RunMolgenisEmx2"
-    javaVersion = JavaVersion.VERSION_17
+    javaVersion = JavaVersion.VERSION_21
 }
 
 allprojects {

ci-build-images/Dockerfile

@@ -3,23 +3,25 @@
 # docker buildx build -t molgenis/ci-build:1.2.2 --platform linux/amd64 .
 # docker push molgenis/ci-build:1.2.2
 
-FROM gradle:jdk17-jammy
+FROM gradle:jdk21-noble
+RUN useradd -m molgenis
 
-RUN apt-get update && apt-get install python3 python3-venv -y
-RUN apt-get install python3 python3-venv python3-pip -y
-RUN python3 -m pip install --upgrade build twine
+RUN apt-get update && apt-get install vim python3 python3-venv python3-pip python3-setuptools python3-setuptools-whl -y
+RUN python3 -m pip config set global.break-system-packages true
 RUN python3 --version
 
 ENV NODE_VERSION=20.11.0
 RUN apt install -y curl
+USER molgenis 
 RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash
-ENV NVM_DIR=/root/.nvm
+ENV NVM_DIR=/home/molgenis/.nvm
 RUN . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION}
 RUN . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION}
 RUN . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION}
-ENV PATH="/root/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"
+ENV PATH="/home/molgenis/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"
 RUN node --version
 RUN npm --version
+USER root 
 
 
 RUN apt update
@@ -55,3 +57,4 @@ RUN apt-get install postgresql-client -y
 # GemTools & FPM For debian and redhat packages
 RUN apt-get install ruby-dev -y 
 RUN gem install fpm
+USER molgenis

docs/install.md

@@ -1,66 +1,64 @@
 # How to install and run
 
-You can start molgenis-emx2:
+MOLGENIS EMX2 can be started in multiple ways:
+* using Docker Compose
+* using Java and PostgreSQL
+* using Kubernetes
 
-* using docker compose
-* using java commandline + postgresql
-* using kubernetes
+## Using Docker Compose
 
-Details below:
+Prerequisites:
+* Docker Compose
 
-## Using docker compose
-
-* Install [Docker compose](https://docs.docker.com/compose/install/).
+Steps:
 * Download
   molgenis-emx2 <a href="https://raw.githubusercontent.com/mswertz/molgenis-emx2/master/docker-compose.yml" download>
   docker-compose.yml</a> file
-* In directory with docker-compose.yml run:
+* In the directory containing the docker-compose.yml file run: 
 
-```
-docker-compose up
-``` 
+  `docker-compose up`
 
-To update to latest release, run:
+* To update to the latest release, run:
 
-```console
-docker-compose pull
-```
+  `docker-compose pull`
 
-Stop by typing ctrl+c.
 
-N.B.
+Stop the execution of the process by pressing CTRL+C.
 
-* because postgres starts slow, emx2 will restart 2-4 times because of 'ConnectException: Connection refused'. This is
-  normal.
-* the data of postgresql will be stored in 'psql_data' folder. Remove this folder you want a clean start.
-* if you want
-  particular [molgenis-emx2 version](https://hub.docker.com/repository/registry-1.docker.io/mswertz/emx2/tags?page=1)
-  then add version in docker-compose.yml file 'molgenis/molgenis-emx2:version'
+Note:
+* because PostgreSQL starts slowly EMX2 will restart 2-4 times because of 'ConnectException: Connection refused'. This is
+  expected behaviour.
+* the data of the PostgreSQL database will be stored in 'psql_data' folder. Remove this folder you want a clean start.
+* if you want to use a specific [molgenis-emx2 version](https://hub.docker.com/repository/registry-1.docker.io/mswertz/emx2/tags?page=1)
+  add the version number in the docker-compose.yml file with the key `molgenis/molgenis-emx2:version`
 
-## Using java and your own postgresql
+## Using Java and your local PostgreSQL database
 
-* Install java (we use java 17)
-* Download a molgenis-emx2-version-all.jar from [releases](https://github.com/molgenis/molgenis-emx2/releases).
-* Download and install [Postgresql](https://www.postgresql.org/download/) (we use 13)
-* Create postgresql database with name 'molgenis' and with superadmin user/pass 'molgenis'. On Linux/Mac commandline:
+Prerequisites:
+* Java version 21
+* PostgreSQL version 15
+
+Steps:
+* Download a molgenis-emx2-version-all.jar file from [releases](https://github.com/molgenis/molgenis-emx2/releases).
+* Create a PostgreSQL database with name 'molgenis' and with superadmin username/password combination 'molgenis'. 
     ```console
     sudo -u postgres psql
     postgres=# create database molgenis;
     postgres=# create user molgenis with superuser encrypted password 'molgenis';
     postgres=# grant all privileges on database molgenis to molgenis;
     ```
-* Start molgenis-emx2; will run on 8080
-    ```console
-    java -jar molgenis-emx2-<version>-all.jar
-    ```
+* Start molgenis-emx2
+    
+  `java -jar molgenis-emx2-<version>-all.jar`
 
-Optionally, you can change defaults using either java properties or using env variables:
+The process will run on HTTP port 8080.
 
-* MOLGENIS_POSTGRES_URI
-* MOLGENIS_POSTGRES_USER
-* MOLGENIS_POSTGRES_PASS
-* MOLGENIS_HTTP_PORT
-* MOLGENIS_ADMIN_PW
+Optionally, the following default values can be modified using either Java properties or using environment variables:
+* `MOLGENIS_POSTGRES_URI`
+* `MOLGENIS_POSTGRES_USER`
+* `MOLGENIS_POSTGRES_PASS`
+* `MOLGENIS_HTTP_PORT`
+* `MOLGENIS_ADMIN_PW`
 
 For example:
 
@@ -72,7 +70,7 @@ java -DMOLGENIS_POSTGRES_URI=jdbc:postgresql:mydatabase -DMOLGENIS_HTTP_PORT=909
 
 If you have Kubernetes server then you can install using [Helm](https://helm.sh/docs/).
 
-Add helm chart repository (once)
+Add Helm chart repository (once)
 
 ```console
 helm repo add emx2 https://github.com/molgenis/molgenis-ops-helm/tree/master/charts/molgenis-emx2
@@ -84,10 +82,10 @@ Run the latest release (see [Helm docs](https://helm.sh/docs/intro/using_helm/))
 helm install emx2/emx2
 ```
 
-Update helm repository to get newest release
+Refresh the Helm repository to get the latest release
 
 ```console
 helm repo update
 ```
 
-Alternatively, [download latest helm chart](https://github.com/mswertz/molgenis-emx2/tree/master/docs/helm-charts)
+Alternatively, [download the latest version of Helm Chart](https://github.com/mswertz/molgenis-emx2/tree/master/docs/helm-charts)

docs/molgenis/run_java.md

@@ -13,7 +13,7 @@ Steps:
     create user molgenis with login nosuperuser inherit createrole encrypted password 'molgenis';
     grant all privileges on database molgenis to molgenis;
     ```
-* Install java (we use adopt [OpenJDK 17](https://adoptium.net/))
+* Install java (we use adopt [OpenJDK 21](https://adoptium.net/))
 * Optionally, if you want to use [scripts](use_scripts_jobs.md) then also install python3
 * Download molgenis-emx2-version-all.jar from [releases](https://github.com/molgenis/molgenis-emx2/releases).
 * Start molgenis-emx2 using command below (will run on 8080)

docs/molgenis/run_systemd.md

@@ -5,7 +5,7 @@ We support Redhat and Ubuntu based installations.
 
 ## Java 
 
-MOLGENIS EMX2 runs on java > 17
+MOLGENIS EMX2 runs on java > 21
 <!-- tabs:start -->
 
 #### **Ubuntu (apt)**