Skip to content

v0.21.0 can't connect to RDS db when running on AWS ECS #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dom-monaco opened this issue Oct 8, 2020 · 7 comments
Closed

v0.21.0 can't connect to RDS db when running on AWS ECS #118

dom-monaco opened this issue Oct 8, 2020 · 7 comments

Comments

@dom-monaco
Copy link

dom-monaco commented Oct 8, 2020

background:
Running postgres on an RDS instance
Trying to run terraboard on ECS (EC2 instance)

When I try to run terraboard's docker image on AWS ECS I get an error on startup that the RDS DB instance is busy

level=info msg="Terraboard v0.21.0 (built for Terraform v0.12.28) is starting..."
level=info msg="Using AWS (S3+DynamoDB) as the state/locks provider"
level=fatal msg="dial tcp: lookup terraboard.abcd1234.us-east-1.rds.amazonaws.com: device or resource busy"

I confirmed all my configs/permissions are correct by running the container locally with no issues, also confirmed that the ECS instance the container runs on is able to connect to the RDS instance directly so that end is setup correctly as well.

@raphink
Copy link
Contributor

raphink commented Jan 18, 2021

This looks to me like a DNS issue, maybe due to the statically built binary.

@dom-monaco
Copy link
Author

This looks to me like a DNS issue, maybe due to the statically built binary.

What statically built binary?
Not sure how that could be a DNS issue, I already stated that the instance the container runs on is able to connect to RDS.

@raphink
Copy link
Contributor

raphink commented Jan 20, 2021

How did you run Terraboard? From the provided binary, from the provided Docker image, or did you build it yourself?

@dom-monaco
Copy link
Author

docker image v0.21.0

@raphink
Copy link
Contributor

raphink commented Jan 21, 2021

Then you're using a static binary, since Terraboard is built statically in the Docker image, in order to use a scratch image as template.

Several go binaries have reports of DNS request failures when built statically, though usually this seems to happen only with names ending in .local (see containers/skopeo#636 for ex)

I wonder if you're hitting a similar bug here.

@mihaiplesa
Copy link
Contributor

Just pass GODEBUG: netdns=go to the Terraboard container as per https://golang.org/pkg/net/#hdr-Name_Resolution

@raphink
Copy link
Contributor

raphink commented Feb 17, 2021

Should be fixed in #135

@raphink raphink closed this as completed Feb 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants