[Bug] 使用deploy/kubernetes进行部署时，提示ingress.yml相关错误 #360

djun · 2022-10-10T16:42:04Z

Search before asking

I had searched in the issues and found no similar issues.

Environment

Linux (self-host)

CPU Architecture

amd64

laf version

0.8.x

What happened

环境是国外VPS，Ubuntu 18.04，自己部署的单节点 k8s v1.22.1 + kubesphere v3.2.0；

已修改config.yml、ingress.yml中与域名相关的为自己域名的nip.io解析。

执行 kubectl apply -f . 时，提示如下错误内容：

Error from server (BadRequest): error when creating "ingress.yml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
-------------------------------------------------------------------------------
Error: exit status 1

日志较长，详见下方debug logs区内容。

谢谢大佬的支持！

How to reproduce

直接git clone，然后只对config.yml、ingress.yml中与域名相关的为自己域名的nip.io解析进行修改，没有做其他任何改动，然后apply，此时会出现该问题。

Debug logs

namespace/laf created
namespace/laf-apps created
clusterrolebinding.rbac.authorization.k8s.io/clusterrolebinding created
service/app-console created
deployment.apps/app-console created
configmap/db-config created
configmap/oss-config created
configmap/root-account created
configmap/secret-salt created
configmap/deploy-domain created
configmap/app-service created
ingress.networking.k8s.io/ingress-oss-console created
ingress.networking.k8s.io/ingress-oss-endpoint created
ingress.networking.k8s.io/ingress-oss-subdomain created
ingress.networking.k8s.io/ingress-sys-console created
ingress.networking.k8s.io/ingress-app-console created
ingress.networking.k8s.io/ingress-sys-api created
ingress.networking.k8s.io/ingress-sys-extension-api created
service/instance-controller created
deployment.apps/instance-controller created
service/mongo created
statefulset.apps/mongo created
service/oss created
statefulset.apps/oss created
service/sys-console created
deployment.apps/sys-console created
service/system-server created
deployment.apps/system-server created
Error from server (BadRequest): error when creating "ingress.yml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:

Error: exit status 1
2022/10/10 16:41:03 [warn] 1553#1553: the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx/nginx-cfg3713638685:144
nginx: [warn] the "http2_max_field_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx/nginx-cfg3713638685:144
2022/10/10 16:41:03 [warn] 1553#1553: the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx/nginx-cfg3713638685:145
nginx: [warn] the "http2_max_header_size" directive is obsolete, use the "large_client_header_buffers" directive instead in /tmp/nginx/nginx-cfg3713638685:145
2022/10/10 16:41:03 [warn] 1553#1553: the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /tmp/nginx/nginx-cfg3713638685:146
nginx: [warn] the "http2_max_requests" directive is obsolete, use the "keepalive_requests" directive instead in /tmp/nginx/nginx-cfg3713638685:146
2022/10/10 16:41:03 [emerg] 1553#1553: host not found in resolver "kube-dns.kube-system.svc.cluster.local" in /tmp/nginx/nginx-cfg3713638685:294
nginx: [emerg] host not found in resolver "kube-dns.kube-system.svc.cluster.local" in /tmp/nginx/nginx-cfg3713638685:294
nginx: configuration file /tmp/nginx/nginx-cfg3713638685 test failed

Are you willing to submit PR?

Yes I am willing to submit a PR!

The text was updated successfully, but these errors were encountered:

maslow · 2022-10-11T02:43:24Z

Error from server (BadRequest): error when creating "ingress.yml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:

根据报错信息，是 ingesss.yml 配置有错。

2022/10/10 16:41:03 [emerg] 1553#1553: host not found in resolver "kube-dns.kube-system.svc.cluster.local" in /tmp/nginx/nginx-cfg3713638685:294

根据此错误日志，知道 host not found in resolver，查一下你的环境或者 kubesphere 的 resolver 到底是什么，然后在这里更新为正确的 resolver 就好了。

另建议使用 sealos 安装 k8s，更简单快捷。未来 laf 的版本是 all in k8s 的，并且和 sealos 超融合。

djun · 2022-10-12T16:32:53Z

以下做一个问题排查处理的记录。

根据大佬指引的方向，辗转看了好几个其他开源项目的issue，最终摸查到了官方的这篇文档：调试DNS问题

https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/dns-debugging-resolution/

进行排查（在我自己VPS上的执行结果）：

root@node1:~# kubectl exec -ti dnsutils -- cat /etc/resolv.conf
nameserver 169.254.25.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

root@node1:~# kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:		169.254.25.10
Address:	169.254.25.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.233.0.1

root@node1:~# kubectl get svc --namespace=kube-system
NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
coredns                       ClusterIP   10.233.0.3   <none>        53/UDP,53/TCP,9153/TCP         100d
kube-controller-manager-svc   ClusterIP   None         <none>        10257/TCP                      100d
kube-scheduler-svc            ClusterIP   None         <none>        10259/TCP                      100d
kubelet                       ClusterIP   None         <none>        10250/TCP,10255/TCP,4194/TCP   100d

root@node1:~# kubectl get endpoints coredns --namespace=kube-system
NAME      ENDPOINTS                                                        AGE
coredns   10.233.90.142:53,10.233.90.147:53,10.233.90.142:53 + 3 more...   100d

root@node1:~# kubectl exec -i -t dnsutils -- nslookup app-console.laf
Server:		169.254.25.10
Address:	169.254.25.10#53

Name:	app-console.laf.svc.cluster.local
Address: 10.233.18.127

root@node1:~# kubectl exec -i -t dnsutils -- nslookup sys-console.laf
Server:		169.254.25.10
Address:	169.254.25.10#53

Name:	sys-console.laf.svc.cluster.local
Address: 10.233.15.70

root@node1:~# kubectl exec -i -t dnsutils -- nslookup system-server.laf
Server:		169.254.25.10
Address:	169.254.25.10#53

Name:	system-server.laf.svc.cluster.local
Address: 10.233.21.142

root@node1:~# kubectl exec -i -t dnsutils -- nslookup instance-controller.laf
Server:		169.254.25.10
Address:	169.254.25.10#53

Name:	instance-controller.laf.svc.cluster.local
Address: 10.233.38.80

root@node1:~# kubectl exec -i -t dnsutils -- nslookup kube-dns.kube-system.svc.cluster.local
Server:		169.254.25.10
Address:	169.254.25.10#53

** server can't find kube-dns.kube-system.svc.cluster.local: NXDOMAIN

command terminated with exit code 1

虽然对k8s的确有不太熟悉之处，但其实查到这里的时候突然有些明白了，我的dns服务这里用的是coredns，跟kube-dns不是同一个，laf原配ingress.yaml里面用的是kube-dns的域名“kube-dns.kube-system.svc.cluster.local”而不是“kubernetes.default.svc.cluster.local”这个（似乎后者这个更为通用，可以使用kube-dns和coredns？）。遂将ingress.yaml中的resolver修改了，马上进行测试。

最终apply结果如下：

root@node1:~/laf/deploy/kubernetes# kubectl apply -f .
namespace/laf unchanged
namespace/laf-apps unchanged
clusterrolebinding.rbac.authorization.k8s.io/clusterrolebinding unchanged
service/app-console unchanged
deployment.apps/app-console configured
configmap/db-config unchanged
configmap/oss-config unchanged
configmap/root-account unchanged
configmap/secret-salt unchanged
configmap/deploy-domain unchanged
configmap/app-service unchanged
ingress.networking.k8s.io/ingress-oss-console unchanged
ingress.networking.k8s.io/ingress-oss-endpoint unchanged
ingress.networking.k8s.io/ingress-oss-subdomain unchanged
ingress.networking.k8s.io/ingress-sys-console unchanged
ingress.networking.k8s.io/ingress-app-console unchanged
ingress.networking.k8s.io/ingress-sys-api unchanged
ingress.networking.k8s.io/ingress-sys-extension-api unchanged
ingress.networking.k8s.io/ingress-apps created
service/instance-controller unchanged
deployment.apps/instance-controller configured
service/mongo unchanged
statefulset.apps/mongo configured
service/oss unchanged
statefulset.apps/oss configured
service/sys-console unchanged
deployment.apps/sys-console configured
service/system-server unchanged
deployment.apps/system-server configured

部署问题似乎解决了。请求重开此问题，想恳请大佬关注一下resolver这里，是否可以按如上提到的进行修改：laf官方k8s部署文件中ingress.yaml的resolver修改为 kubernetes.default.svc.cluster.local 。谢谢大佬的关注！

maslow · 2022-10-12T17:38:40Z

👍可以直接pr 改成你建议的那样

djun · 2022-10-13T02:14:24Z

👍可以直接pr 改成你建议的那样

请大佬审核，谢谢： #364

…364 ) * Update deploy/kubernetes/README.md and deploy/kubernetes/ingress.yml. * Fix ingress.yml

maslow · 2022-10-13T02:39:08Z

在你实际部署的过程中，如果有其它配置需要更新的，也可以一同更新上来。
毕竟 k8s 的部署脚本，后续我没有亲自部署过，不知道还跟不跟得上最新版。

maslow closed this as completed Oct 11, 2022

maslow added user-error setup issue labels Oct 11, 2022

maslow pushed a commit that referenced this issue Oct 13, 2022

fix(deploy): fix #360 k8s deployment nginx ingress default resovler (#…

ae4a26a

…364 ) * Update deploy/kubernetes/README.md and deploy/kubernetes/ingress.yml. * Fix ingress.yml

maslow reopened this Oct 13, 2022

maslow closed this as completed Oct 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] 使用deploy/kubernetes进行部署时，提示ingress.yml相关错误 #360

[Bug] 使用deploy/kubernetes进行部署时，提示ingress.yml相关错误 #360

djun commented Oct 10, 2022

maslow commented Oct 11, 2022

djun commented Oct 12, 2022 •

edited

Loading

maslow commented Oct 12, 2022

djun commented Oct 13, 2022

maslow commented Oct 13, 2022

[Bug] 使用deploy/kubernetes进行部署时，提示ingress.yml相关错误 #360

[Bug] 使用deploy/kubernetes进行部署时，提示ingress.yml相关错误 #360

Comments

djun commented Oct 10, 2022

Search before asking

Environment

CPU Architecture

laf version

What happened

How to reproduce

Debug logs

Are you willing to submit PR?

maslow commented Oct 11, 2022

djun commented Oct 12, 2022 • edited Loading

maslow commented Oct 12, 2022

djun commented Oct 13, 2022

maslow commented Oct 13, 2022

djun commented Oct 12, 2022 •

edited

Loading