Problem with 'meep-prometheus'

[ruben-bc]

Hello,

The pod meep-prometheus-couchdb continually resets and doesn't allow me to work with the platform. Usually stays running, but without being ready (0/1). Only remain a few minutes 1/1, then go back to 0/1.

Other times, after several attempts, it ends in CrashLoopBackOff.

I have executed meepctl delete core and meepctl delete dep, then the deploy again, but it still doesn't change. I even deleted all and started with the process again. I don't know what's happening.

Thanks in advance.

[dilallkx]

Hi @rubenbc28,

Thank you for reaching out!
I have a few questions regarding the issue you are observing:

• Which OS & AdvantEDGE versions are you using?
  • lsb_release -a
  • uname -a
  • meepctl version all (note: please run this command with dependency pods running)
• Does your system meet the hardware requirements?
  • https://interdigitalinc.github.io/AdvantEDGE/docs/setup/env-hw/
• Is this the first time you install AdvantEDGE on your setup?
• Please provide output from the following commands:
  • kubectl describe node
  • kubectl describe pod <failing pod name>
  • kubectl logs <failing pod name>
  • kubectl logs <failing pod name> -p (note: this command must be run after a pod crash)

This should help debug your issue.
Regards!

[ruben-bc]

Hi again Kevin,

• Prometheus seems to have been started before the k8s node was fully up & running
  • Did you restart your VM with dependency pods already deployed?

No, I always do meepctl delete core and meepctl delete dep before shutdown my VM.

Also, you mention that the pod is no longer crashing. Is it still in running 0/1 state?

It not back in CrashLoopBackOff yet, but still in running 0/1 every so often, remaining some minutes in 1/1 and then other some in 0/1.

• Filesystem size & available space:
  • df /

Filesystem      1K-blocks        Used     Available      Use%    Mounted on
/dev/sda5        40503552    28088608      10327776       74%    /

• Prometheus persistent volume permissions:
  • ls -l ~/.meep/prometheus/server/

total 4
drwxrwxr-x 9 ruben ruben 4096 feb 12 17:15 prometheus-db

• Host/VM uid & gid:
  • id

uid=1000(ruben) gid=1000(ruben) groups=1000(ruben),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),132(lxd),133(sambashare),997(docker)

• NOTE: make sure your host/VM uid & gid are properly configured in the .meepctl-repofg.yaml file before deploying dep & core pods.

I just checked and it's correct:

# permissions
    permissions:
      # user id
      uid: 1000
      # group id
      gid: 1000

Thanks!!

[dilallkx]

Hi @rubenbc28,

Apologies for the delayed response.

From your reply, I don't see why Prometheus would fail to start. You mentioned in a previous response that you did not have the crash logs for the failing pod. Is this because it had not restarted yet? If so, could you please wait for it to crash/restart and then get the output from:
kubectl logs < failing pod name > -p
Hopefully this can provide more insight on the failure.

Other than this your filesystem usage is slightly high but not alarming and did not seem to cause disk pressure in k8s (you can check for this in the kubectl describe node output). Your configuration & permissions seems correct as well, so I am not sure what is causing the failure. Let's see if we can get more information from a crash log.

Regards!

[ruben-bc]

Hi kevin,

It crashed again and I ran the command kubectl logs < failing pod name > -p. This is what I got:

I0217 18:10:49.986858       1 couchdb-exporter.go:77] Starting exporter at '0.0.0.0:9984' to read from CouchDB at 'http://meep-couchdb-svc-couchdb:5984/'
I0217 18:10:53.488051       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:10:58.238476       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:03.349186       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:08.154761       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:14.477552       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:18.952635       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:23.136706       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:29.410652       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:34.021767       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:38.727994       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:43.600690       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:49.250345       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:53.795475       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:55.258882       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:11:58.934907       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:01.185125       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:03.344327       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:05.035812       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:08.677793       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:09.981842       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:13.507823       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:15.188553       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:18.866635       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:19.788838       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:23.360241       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:29.360813       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:33.533715       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:34.898335       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local
I0217 18:12:35.616003       1 couchdb-client.go:82] node[0]: couchdb@meep-couchdb-couchdb-0.meep-couchdb-couchdb.default.svc.cluster.local


  Warning  Unhealthy  24m (x2 over 24m)     kubelet, advantedge  Readiness probe failed: Get "http://10.32.0.67:9984/": dial tcp 10.32.0.67:9984: connect: connection refused

Furthermore, I have search for a solution to the Liveness/Readiness probe failed and found that could due to initial delay, that was so low and the pod didn't be able to connect and up all the elements. So I also ran kubectl edit deploy meep-prometheus-couchdb-exporter -o yaml, changed the initialDelaySeconds from 1s to 60s (to have time to spare), and apparently it works fine, without any restart or crash at least while using the platform.

livenessProbe:
  failureThreshold: 3
  httpGet:
    path: /
    port: http
    scheme: HTTP
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
name: prometheus-couchdb-exporter
ports:
- containerPort: 9984
  name: http
  protocol: TCP
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /
    port: http
    scheme: HTTP
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

I don't understand why this happens if all seems to work fine except this and don't think timers are the only reason. If it happens again or anyone finds a solution, we can reply or tag it in this discussion.

Thank you so much and regards.

[dilallkx]

Thank you for the information @rubenbc28!
Glad to hear you managed to get it working.

I am a bit surprised that you had to change the initialDelaySeconds from 1s to 60s on your system. On my deployment it is set to 60s by default. Can you confirm that the liveness probe initial delay in your cloned repo is 1s or 60s?
~/AdvantEDGE/charts/kube-prometheus-stack/charts/prometheus-couchdb-exporter/templates/deployment.yaml

Do you confirm that on reinstall initialDelaySeconds is set to 1s?
Thanks!

[ruben-bc]

Hi Kevin,

I checked right now these two things, cloning and building all again (in unused VM of course), and the file deployment.yaml is set to 60s. When running the command kubectl edit deploy meep-prometheus-couchdb-exporter -o yaml, it's also 60s, so the data is correct in the repo and after building. Why in my case I had 1s for initialDelaySeconds? I don't know because I never touched any deployment file. Anyways, at the moment there seems to be no more problems.

The delay part in deployment.yaml:

ports:
  - name: http
    containerPort: 9984
  livenessProbe:
  httpGet:
    path: /
    port: http
  initialDelaySeconds: 60
  readinessProbe:
  httpGet:
    path: /
    port: http
  initialDelaySeconds: 60
  resources:

Thanks to you for all the support!!