Creating Kubernetes-Openshift deployment for Adopt+OpenJ9 JDKs

[raguks]

Hi all,
deployment technologies for applications and runtimes have progressed a lot. Kubernetes(K8s) and OpenShift(OCP) based deployments are now common. Do you think, we should add these deployment options to OpenJ9? With JIT Server technology now being available - easy deployment to K8s and OCP allow better consumability for OpenJ9 technologies in various hybrid cloud platforms.

This can include

• Helm Chart for K8s
• Operator for OpenShift4+
• And various platform support in them.

There may be support for other platforms we have to add but K8s powering most hybrid cloud platforms I think it helps to start from there and OpenShift4 is a natural option as well being an open Hybrid Cloud platform.

Please refer to the User Story and Roadmap:
V1
JITServer Deployment User Story.pdf
V2
JITServer Deployment User Storyv2.pdf

[0xdaryl]

Perhaps this is a topic that could be discussed at an upcoming weekly OpenJ9 community call?

[DanHeidinga]

Thanks @raguks for opening this issue. A couple of questions I'll add here and we can either address in this issue or, as Daryl suggested, discuss on the community call.

Are the helm charts / operators for JITServer deployments or intended for a general openj9 deployments?

Operators are a really cool piece of tech that helps with "day 2" concerns when running Java in K8 and there's a lot of value to having an OpenJ9-specific operator. Do you have thoughts on the capabilities of that operator?

Some ideas of potential operator capabilities:

• generate diagnostic dump files (javacore / system core / heap / trace )
• modify gc softmx in a coordinated way for JVMs on the same node
• etc

What role do you see for the Helm charts vs the Operator? My understanding is Helm is more about install while the Operator allows coordination. Do you see the need for both? Would the up-stack frameworks cover the Helm concerns?

In addition to defining the goals of the helm charts & operators, some thought on how to test these artifacts is also required. The project doesn't currently have a K8 cluster and we'd need to get access to one as well as licenses for any particular K8 distribution.

[raguks]

Thank you @DanHeidinga

1. Yes we can tailor the chart to deploy openj9 or JITServer through command line/yaml options. It makes sense to use "official" adopt+openj9 container images
2. While an Operator is capable of lot more, it is also a basic deployment mechanism for OpenShift. We can start with basic deployment to OpenShift as MVP and then iterate to add features you have suggested among others such as support for JDK upgrade etc.

Helm chart while being a common deployment option for all K8 vendors is also an ingredient of the Operator. Helm chart support will give openj9 a vendor neutral way to deploy to any K8 platform.

1. Charts have an inbuilt way to test deployments and we need to update CI/CD pipelines to kick off these tests. @KevinBonilla let me know your thoughts.
2. Yes - we need to find infrastructure to do these tests.

Please let me know if there are questions.

[mstoodle]

@raguk thanks for opening this issue.

Are there other examples of open source projects that provide their own helm charts & operators to facilitate k8s deployment? I wonder if there are some best practices we could learn from others on where to keep them, how to manage (test) them, etc. Is this something that primarily OpenJ9 should do, something primarily AdoptOpenJDK should do, or something we both need to work on?

Should we (also?) consider contributing something to e.g. https://github.com/helm/charts ?

[raguks]

@mstoodle I think you beat me to it - I was preparing this note!
There are two approaches,

1. charts are put in a common public chart repos from where developers draw apps For example, bitnami helm/charts ibm/charts eclipse has one. wildfly for example is in bitnami and does not host on its own repo. open-liberty is on ibm/charts. A lot of apps seem to do that (including prometheus).

2. Some host charts in their own repo (which takes devs couple of extra steps to use the charts) eclipse-che is an example.

I was thinking hosting on openj9 and then a CI/CD process to host on multiple helm repos may be a way to go (helm bitnami) (or just stick to one).

Just to start the conversation on openj9 our charts may look like this (please comment):

/
.
..
buildenv
deploy
          |
          stable
                   |
                    openj9-jitserver
                              |
                              charts
                                       | 
                                       openj9-chart

                              operators
                                             |
                                              openj9-operator                                 

on eclipse-che it looks like this:

and publishing other repos through CI/CD - like
helm/charts/stable/openj9
helm/charts/stable/openj9-operator
bitnami/charts/bitnami/openj9
bitnami/charts/bitnami/openj9-operator

By developing openj9's own charts openj9 can avoid dealing with major changes to other repos but while publishing to them as well to be widely and easily available. This includes adopt-openjdk. We can contribute a more generic version to support all adopt flavours.

[mstoodle]

ok, it sounds like hosting them at openj9 first helps developers to quickly iterate to something that will work well for our users, but once we get things settled, we can then pursue making it easier for users to get them.

I'm still not sure I understand how we'll maintain and test this material though. Can you put a few more words to "Charts have an inbuilt way to test deployments" ?

[raguks]

As part of the Chart validation we can have some tests defined. These test if the Chart deploys openj9 in our case. We can also add actual functional verification of jitserver in particular. To run these tests however, we need to define some Jenkins jobs and need to use openj9 test infrastructure. @chrisc66 @KevinBonilla may be able find out more if @smlambert or someone can point them to CI/CD for openj9.

@mstoodle The name for the Chart is kind of important because its used in many places. Since we need to support openj9 and jitserver what should the Chart be called? openj9-jitserver if we only consider jitserver. openj9-chart openj9-operator if we have to consider openj9 in general.
Appreciate some suggestions here.

[DanHeidinga]

I want to circle back to my earlier question:

Are the helm charts / operators for JITServer deployments or intended for a general openj9 deployments?

My (admittedly limited) knowledge of Helm is that charts are for deploying applications, not runtimes. Is the proposal to add basic charts, really templates, that can easily be adapted by users to deploy their applications on OpenJ9?

It would be good to clarify how this proposal fits with the general Helm guidance.

Additionally, if this is about showing how to deploy applications with Helm charts so they employ jitserver, is it accurate to characterize this effort as providing more of an example chart that highlights the (few?) stanzas of yaml required to enable jitserver?

[DanHeidinga]

For the operator, have you seen the https://github.com/application-stacks/runtime-component-operator project? It provides a set of base capabilities other operators can build on:

This Operator can also be imported into any runtime-specific Operator as library of application capabilities. This architecture ensures compatibility and consistency between all runtime Operators, allowing everyone to benefit from the functionality added in this project.

I ran across this when looking at the Open-Liberty operator which can already control JVM dump facilities.

There may be an opportunity to work with the Liberty community to share code around these capabilities.

[raguks]

@DanHeidinga There are two different questions here. 1) Regarding the nature of openj9 deployment and 2) feature set and libraries.

1. The proposed chart/operator is to deploy openj9 as jitserver only primarily. Because openj9 can be deployed using the same chart as a non jitserver we can provide that option. OpenJ9 deployment works well for java applications that just need a jdk and no web-server. This is because we use adopt-openj9 docker images that are optimized for java(openj9 community is constantly working on it). The work an application developer does is to build an application image based on adopt-openj9 image and use it in the chart to deploy openj9+her app (you can think of that as template in that sense). It is also useful for development and developer advocacy. But the primary deployment is jitserver. How it actually gets used may change based on users preference. Making allowance for it gives openj9 that space. Need to grab a late breakfast! So will get back on the 2nd question a bit after. This is a very good question. Helps us get clarity on these issues.

[raguks]

2. While OpenLiberty operator has been around for a year, use of runtime-component-operator is new (roughly 1.5 months ago). Yes we need to explore its use. However, I think we need to do this in steps once we have a basic helm chart out.

[DanHeidinga]

Thanks @raguks Based on your answers above, I'll update the title of this issue to focus on adding a JITServer Helm chart. That sounds like the primary goal being worked towards here. We can move general discussions about Operators into another issue.

I'd like to hear more about the deployment story for how this Helm chart works with deploying other applications. What's the relationship between the JITSever Helm chart and the user's application Helm chart? Does the user one need to reference the JITServer one? Does the JITServer one replace the user's existing chart? Is the new chart an example where users would only need to copy/paste a "stanza" from it to their existing charts?

If there's a simple picture that shows how the deployment works and the relationships between the JITServer Helm chart and existing charts that would help clarify the goals of this issue.

Can you also list out the steps a user would take to integrate the new Helm chart?

This is a lot of questions but I'm hoping they clarify the direction of this issue and show how the chart would be consumed.

[raguks]

@DanHeidinga Our goal as part of this is to have chart/operator for both openj9 and jitserver. Yes there is material on end to end use of charts/operators. Simply put, users deploy jitserver and update their application to compile on it by using the JAVA_OPTION. There are many possibilities on how they can do it at the application including updating their charts/operators/pipeline.

I can go over the details in one of the community calls, possibly week next (coming week is bit busy for me).