Skip to content

HTTPRoute not working with error 503 due to cointainer image cloudflare/cloudflared:2024.11.1 #202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pipoe2h opened this issue Dec 21, 2024 · 6 comments
Open

HTTPRoute not working with error 503 due to cointainer image cloudflare/cloudflared:2024.11.1 #202

pipoe2h opened this issue Dec 21, 2024 · 6 comments

Comments

@pipoe2h
Copy link

pipoe2h commented Dec 21, 2024

The current cloudflared image in the controller makes HTTPRoutes to not work. After changing the container image to cloudflare/cloudflared:2024.12.2, HTTPRoutes start to work

cloudflare-kubernetes-gateway/config/manager/manager.yaml

Lines 60 to 62 in 2d2113a

- name: GATEWAY_IMAGE
# renovate: datasource=docker depName=cloudflare/cloudflared
value: "cloudflare/cloudflared:2024.11.1"

With cloudflared:2024.11.1:

2024-12-21T14:43:19Z INF Registered tunnel connection connIndex=3 connection=dc537efb-cb6a-4f33-975b-2783792a775b event=0 ip=198.41.200.43 location=ams01 protocol=quic
2024-12-21T14:45:42Z WRN No ingress rules were defined in provided config (if any) nor from the cli, cloudflared will return 503 for all incoming HTTP requests

With cloudflared:2024.12.2 (cloudflare/cloudflared@sha256:cb38f3f30910a7d51545118a179b8516eb7066eac61855d62ce6ed733c54ce70):

2024-12-21T14:49:28Z INF Registered tunnel connection connIndex=2 connection=84ed7958-cca3-47a3-a9d0-4320ca10525a event=0 ip=198.41.200.23 location=ams01 protocol=quic
2024-12-21T14:49:28Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"ex01.example.com\",\"path\":\"/\",\"service\":\"http://whoami.default:80\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=1
@pipoe2h pipoe2h mentioned this issue Dec 21, 2024
Merged
1 task
@pl4nty
Copy link
Owner

pl4nty commented Dec 21, 2024

Can you get the operator logs please? cloudflared just receives config from the operator via cloudflare's API, so its version shouldn't impact config

@pipoe2h
Copy link
Author

pipoe2h commented Dec 22, 2024

@pl4nty, you are right; I think I've identified the error now. If I apply a single YAML with a gateway and a httproute like in the example, I get the 503 error when accessing the httproute. If I delete the gateway pod, then after it gets recreated, I can see the message in the logs with Updated to new configuration, and the 503 error is gone.

It could be that if the httproute gets created before the gateway exists, the gateway isn't able to reconcile the configuration and update.

2024-12-22T09:39:26Z	INFO	starting server	{"name": "health probe", "addr": "[::]:8081"}
I1222 09:39:26.636594       1 leaderelection.go:254] attempting to acquire leader lease cloudflare-gateway/8c89aab9.cfargotunnel.com...
2024-12-22T09:39:26Z	INFO	controller-runtime.metrics	Starting metrics server
2024-12-22T09:39:26Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
I1222 09:39:53.637374       1 leaderelection.go:268] successfully acquired lease cloudflare-gateway/8c89aab9.cfargotunnel.com
2024-12-22T09:39:53Z	DEBUG	events	cloudflare-controller-manager-685d77c75d-2xbww_b88ed0ff-1117-4805-8049-af7c772e9038 became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"cloudflare-gateway","name":"8c89aab9.cfargotunnel.com","uid":"e92f182d-f967-4d4e-965a-05c895a86866","apiVersion":"coordination.k8s.io/v1","resourceVersion":"198397"}, "reason": "LeaderElection"}
2024-12-22T09:39:53Z	INFO	Starting EventSource	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "source": "kind source: *v1.HTTPRoute"}
2024-12-22T09:39:53Z	INFO	Starting Controller	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute"}
2024-12-22T09:39:53Z	INFO	Starting EventSource	{"controller": "gatewayclass", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "GatewayClass", "source": "kind source: *v1.GatewayClass"}
2024-12-22T09:39:53Z	INFO	Starting Controller	{"controller": "gatewayclass", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "GatewayClass"}
2024-12-22T09:39:53Z	INFO	Starting EventSource	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.Gateway"}
2024-12-22T09:39:53Z	INFO	Starting EventSource	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "source": "kind source: *v1.Deployment"}
2024-12-22T09:39:53Z	INFO	Starting Controller	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway"}
2024-12-22T09:39:53Z	INFO	Starting workers	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "worker count": 1}
2024-12-22T09:39:53Z	INFO	Starting workers	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "worker count": 1}
2024-12-22T09:39:53Z	INFO	Starting workers	{"controller": "gatewayclass", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "GatewayClass", "worker count": 1}
2024-12-22T09:39:53Z	INFO	Updating Gateway listeners	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "2939754b-6d20-4265-9a3a-5da544378d11", "AttachedRoutes": 2}
2024-12-22T09:39:55Z	INFO	Updated Tunnel configuration	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "2939754b-6d20-4265-9a3a-5da544378d11", "ingress": [{"hostname":"ex01.example.com","path":"/","service":"http://whoami.default:80"},{"service":"http_status:404"}]}
2024-12-22T09:39:58Z	INFO	Updated DNS records	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "2939754b-6d20-4265-9a3a-5da544378d11", "hostnames": ["ex01.example.com"]}
2024-12-22T09:44:11Z	INFO	Updating Gateway listeners	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "c1464c65-0423-4566-996c-cbfd52ab752d", "AttachedRoutes": 1}
2024-12-22T09:44:13Z	INFO	Updated Tunnel configuration	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "c1464c65-0423-4566-996c-cbfd52ab752d", "ingress": [{"service":"http_status:404"}]}
2024-12-22T09:44:13Z	INFO	Updated DNS records	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "c1464c65-0423-4566-996c-cbfd52ab752d", "hostnames": []}
2024-12-22T09:44:47Z	INFO	Updating Gateway listeners	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "e81b068f-2904-4401-a6bc-7fd7a2dec12d", "AttachedRoutes": 3}
2024-12-22T09:44:49Z	INFO	Updated Tunnel configuration	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "e81b068f-2904-4401-a6bc-7fd7a2dec12d", "ingress": [{"hostname":"gapi.example.com","path":"/","service":"http://whoami.default:80"},{"hostname":"new.example.com","path":"/","service":"http://whoami.default:80"},{"service":"http_status:404"}]}
2024-12-22T09:44:54Z	INFO	Updated DNS records	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "e81b068f-2904-4401-a6bc-7fd7a2dec12d", "hostnames": ["gapi.example.com","new.example.com"]}
2024-12-22T09:46:31Z	INFO	Updating Gateway listeners	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "dd1eb6f4-641a-4e7c-bd67-81a0e4e75401", "AttachedRoutes": 1}
2024-12-22T09:46:33Z	INFO	Updated Tunnel configuration	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "dd1eb6f4-641a-4e7c-bd67-81a0e4e75401", "ingress": [{"service":"http_status:404"}]}
2024-12-22T09:46:33Z	INFO	Updated DNS records	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "dd1eb6f4-641a-4e7c-bd67-81a0e4e75401", "hostnames": []}
2024-12-22T09:47:00Z	INFO	Performing Finalizer Operations for Gateway before delete CR	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "7b569f8c-2147-4fa7-986c-6056ec6bd2ae"}
2024-12-22T09:47:00Z	ERROR	Failed to update Gateway finalizer status	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "7b569f8c-2147-4fa7-986c-6056ec6bd2ae", "error": "Operation cannot be fulfilled on gateways.gateway.networking.k8s.io \"gateway\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/pl4nty/cloudflare-kubernetes-gateway/internal/controller.(*GatewayReconciler).Reconcile
	/home/runner/work/cloudflare-kubernetes-gateway/cloudflare-kubernetes-gateway/internal/controller/gateway_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224
2024-12-22T09:47:00Z	ERROR	Reconciler error	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "7b569f8c-2147-4fa7-986c-6056ec6bd2ae", "error": "Operation cannot be fulfilled on gateways.gateway.networking.k8s.io \"gateway\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224
2024-12-22T09:47:00Z	INFO	Performing Finalizer Operations for Gateway before delete CR	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "868a0409-51a8-484d-910a-c6e0cd206d34"}
2024-12-22T09:47:01Z	INFO	Deleting Tunnel	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "868a0409-51a8-484d-910a-c6e0cd206d34"}
2024-12-22T09:47:06Z	DEBUG	events	Gateway gateway is being deleted from the namespace cloudflare-gateway	{"type": "Warning", "object": {"kind":"Gateway","namespace":"cloudflare-gateway","name":"gateway","uid":"d01a05c2-5464-427e-930c-a865dec0c27e","apiVersion":"gateway.networking.k8s.io/v1","resourceVersion":"200435"}, "reason": "Deleting"}
2024-12-22T09:47:06Z	INFO	Removing Finalizer for Gateway after successfully perform the operations	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "868a0409-51a8-484d-910a-c6e0cd206d34"}
2024-12-22T09:47:06Z	INFO	gateway resource not found. Ignoring since object must be deleted	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "bc6db5be-b905-4f86-b37e-6b6c03685260"}
2024-12-22T09:47:06Z	INFO	gateway resource not found. Ignoring since object must be deleted	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "b0a01d66-ad94-4dec-a58a-f5b3b825fe76"}
2024-12-22T09:48:21Z	INFO	Adding Finalizer for Gateway	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "58cea0d3-d989-480b-a9ee-48c37b831eaa"}
2024-12-22T09:48:21Z	INFO	Updating Gateway listeners	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "c7efab43-bc7b-4e3b-a9f4-87e57e4649ab", "AttachedRoutes": 3}
2024-12-22T09:48:22Z	INFO	Creating tunnel	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "58cea0d3-d989-480b-a9ee-48c37b831eaa"}
2024-12-22T09:48:22Z	INFO	Tunnel doesn't exist yet, probably waiting for the Gateway controller. Retrying in 1 minute	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "c7efab43-bc7b-4e3b-a9f4-87e57e4649ab", "gateway": "gateway"}
2024-12-22T09:48:24Z	INFO	Creating a new Deployment	{"controller": "gateway", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "Gateway", "Gateway": {"name":"gateway","namespace":"cloudflare-gateway"}, "namespace": "cloudflare-gateway", "name": "gateway", "reconcileID": "58cea0d3-d989-480b-a9ee-48c37b831eaa", "Deployment.Namespace": "cloudflare-gateway", "Deployment.Name": "gateway"}
2024-12-22T09:49:22Z	INFO	Updating Gateway listeners	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "5458a924-cada-4be2-9cc5-e65b887ebf71", "AttachedRoutes": 3}
2024-12-22T09:49:24Z	INFO	Updated Tunnel configuration	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "5458a924-cada-4be2-9cc5-e65b887ebf71", "ingress": [{"hostname":"ex02.example.com","path":"/","service":"http://whoami.default:80"},{"hostname":"ex03.example.com","path":"/","service":"http://whoami.default:80"},{"service":"http_status:404"}]}
2024-12-22T09:49:30Z	INFO	Updated DNS records	{"controller": "httproute", "controllerGroup": "gateway.networking.k8s.io", "controllerKind": "HTTPRoute", "HTTPRoute": {"name":"example-route","namespace":"default"}, "namespace": "default", "name": "example-route", "reconcileID": "5458a924-cada-4be2-9cc5-e65b887ebf71", "hostnames": ["ex02.example.com","ex03.example.com"]}

@jseely
Copy link

jseely commented Jan 3, 2025

I've run into a similar issue, it appears that there's a race condition between the creation of the HttpRoute resource and the configuration being applied to the tunnel.

I believe the issue @pipoe2h ran into is that if you create both the Gateway and the HttpRoute at the same time then the configuration doesn't exist on first launch of the Gateway but then doesn't ever receive the configuration update with the HttpRoute configuration.

Tangentially it appears that old hostname routes aren't cleaned up in Cloudflare after they've been removed. I'll open another issue for that. Happy to look into fixes over the next week!

@pl4nty
Copy link
Owner

pl4nty commented Jan 3, 2025

@pipoe2h do you have the logs from cloudflared? those controller logs indicate tunnel config was sent to the API, but may not be received by cloudflared

@jseely if the gateway and route are created simultaneously, the route controller will wait 1 minute for the tunnel to exist in Cloudflare's API. after that, it should send config to the API which cloudflared will download and use

pl4nty added a commit that referenced this issue Jan 4, 2025
@pl4nty
fix: README split example yaml to avoid race condition, warn about or…
50e9027 
…phaned hostnames

#202 suggests the 1-minute race condition delay isn't great UX. let's split the Gateway and HttpRoute example yaml, so if the user is applying manually, the Gateway should be ready by the time they apply the HttpRoute

#206 should be documented, pending a fix
@GoodLucky777
Copy link

I had the similar issue with v0.8.1. It seems the Cloudflared doesn't update config even the controller update the config. It wasn't fixed automatically after 1 minute so I had to delete Cloudflared pod to fix it.

@RealDyllon
Copy link

As a temporary fix for anyone reading, deleting the gateway Pod was required to solve this race condition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jseely @pipoe2h @pl4nty @RealDyllon @GoodLucky777