Broken Admin API dependency in 3.x version while 2.x works

[jaswanthikolla]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.x

Current Behavior

Please note that everything works in 2.8.2 version of Kong. But, In 3.x version, It breaks with following error message. Kong is used In DBLess mode with Kong Ingress controller on k8s. As per KIC doc, there are 2 options ( Ingress/CRD and admin API) to configure, and Ingress/CRD is used for configuring kong.

Following is the error log, and it continues with the retry and then go to CrashLoop.

Defaulted container "ingress-controller" out of: ingress-controller, proxy, clear-stale-pid (init)
time="2023-02-21T07:59:49Z" level=info msg="diagnostics server disabled"
time="2023-02-21T07:59:49Z" level=info msg="starting controller manager" commit=7b69f7cbf38999ffd2842344a8d2fe471fa98afd logger=setup release=2.6.0 repo="https://github.com/Kong/kubernetes-ingress-controller.git"
time="2023-02-21T07:59:49Z" level=info msg="getting enabled options and features" logger=setup
time="2023-02-21T07:59:49Z" level=info msg="getting the kubernetes client configuration" logger=setup
W0221 07:59:49.271599       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2023-02-21T07:59:49Z" level=info msg="getting the kong admin api client configuration" logger=setup
time="2023-02-21T07:59:49Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=0/60
time="2023-02-21T07:59:50Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=1/60
time="2023-02-21T07:59:51Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=2/60
time="2023-02-21T07:59:52Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=3/60
time="2023-02-21T07:59:53Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=4/60
time="2023-02-21T07:59:54Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=5/60
time="2023-02-21T07:59:55Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=6/60
time="2023-02-21T07:59:56Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=7/60
time="2023-02-21T07:59:57Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=8/60

Expected Behavior

This works in 2.8.2 version ( or any 2.x) of Kong even with Latest kubernetes-ingress-controller "2.8.1" . Please see steps to reproduce on reproducing working DBLess with 2.8.2 . We do get the retry error for 4 attempts but then it resolves, and doesn't impact.

Steps To Reproduce

This is reproducible on the fresh Kubernetes cluster with Latest Kong verison. Please note that plugins#configMaps: is needed to reproduce, without it everything works just fine.

1. setup kong-plugin-myheader plugin as per this kong plugin doc , But Don't install as per the doc, install with following.
2. Use the following helm chart values to install DBLess Kong

values_kong_3.yaml

image:
  repository: kong/kong-gateway
  tag: "3.1"


manager:
  enabled: false

portal:
  enabled: false

plugins:
  configMaps:              
    - name: kong-plugin-myheader
      pluginName: myheader

env:
  database: "off"

ingressController:
  enabled: true
  image: 
    repository: kong/kubernetes-ingress-controller
    tag: "2.8.1"

proxy:
  annotations: {}
  enabled: true
  http:
    containerPort: 8080
    enabled: true
    hostPort: 80
  ingress:
    enabled: false
  labels:
    enable-metrics: true
  tls:
    containerPort: 8443
    enabled: true
    hostPort: 443
  type: LoadBalancer

  
admin:
  enabled: true
  tls:
    parameters: []
  http:
    enabled: true
    servicePort: 8001
    containerPort: 8001

3. invoke helm command helm install quickstart kong/kong --namespace kong --values values_kong_3.yaml
4. check the pod logs.

Working setup ( everything works with custom plugin) in 2.8.x

1. setup kong-plugin-myheader plugin as per this kong plugin doc , But Don't install kong as per the doc, install with following.
2. Use the following helm chart values to install DBLess Kong. It' exactly same as above YAML file, but with following values instead.
   values_kong_2.yaml:

image:
  repository: kong/kong-gateway
  tag: "2.8.2.4"

4. invoke helm command helm install quickstart kong/kong --namespace kong --values values_kong_2.yaml

Anything else?

1. It works in Kong 3.x without custom plugins.
2. I'm raising this issue on Kong instead of KIC project. Because, latest version of KIC is working with 2.8.2, and while it's not working with Kong 3.x. So, it looks like Kong 3.x is the one with issue

[hbagdi]

cc @Kong/team-k8s

[rainest]

Logs from the proxy container suggest the example plugin needs to be updated for 3.x, though I'm not familiar with this new VERSION requirement offhand:

12:49:50-0800 esenin $ kubectl logs bread-kong-865446c558-s97fp -c proxy
2023/02/21 20:49:37 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /kong_prefix/nginx.conf:6
2023/02/21 20:49:37 [notice] 1#0: [lua] license_helpers.lua:155: read_license_info(): [license-helpers] could not decode license JSON: No license found
2023/02/21 20:49:37 [notice] 1#0: [lua] license_helpers.lua:155: read_license_info(): [license-helpers] could not decode license JSON: No license found
2023/02/21 20:49:37 [warn] 1#0: [kong] [C]:-1 [Penlight 1.13.1] the contents of module 'pl.xml' has been deprecated, please use a more specialized library instead (deprecated after 1.11.0, scheduled for removal in 2.0.0)
nginx: [warn] [kong] [C]:-1 [Penlight 1.13.1] the contents of module 'pl.xml' has been deprecated, please use a more specialized library instead (deprecated after 1.11.0, scheduled for removal in 2.0.0)
2023/02/21 20:49:37 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:676: error loading plugin schemas: on plugin 'myheader': Plugin "myheader" cannot be loaded because its VERSION field does not follow the "x.y.z" format, got: "nil".

stack traceback:
	[C]: in function 'assert'
	/usr/local/share/lua/5.1/kong/init.lua:676: in function 'init'
	init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:676: error loading plugin schemas: on plugin 'myheader': Plugin "myheader" cannot be loaded because its VERSION field does not follow the "x.y.z" format, got: "nil".

stack traceback:
	[C]: in function 'assert'
	/usr/local/share/lua/5.1/kong/init.lua:676: in function 'init'
	init_by_lua:3: in main chunk

No apparent Kubernetes integration issue, just that the guide happens to use it.

[hbagdi]

See #8836.
Perhaps a PR to fix the documentation to add a VERSION field is necessary.

[raoxiaoyan]

There is a PR to fix this issue. close it.