terraform-provider-google_v1.13.0_x4 fails to create services

[pdaborowski-playtech]

Hi,
When creating new resource google_project_services in google provider version published 14 hours ago, project ID seems to be hardcoded to 764086051850 in every new project we're creating.

* google_project_services.service: Error creating services: googleapi: Error 403: Service Usage API has not been used in project 764086051850 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=764086051850 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

The problem was introduced this night, yesterday everything worked fine, and works fine if we enforce provider version 1.12.0

Terraform Version

• terraform v0.11.7
• provider.google v1.13.0

Affected Resource(s)

Please list the resources as a list, for example:

• google_project_services

[paddycarver]

Hey @pdaborowski-playtech, sorry for the issue! As called out in the changelog, this is a change in how the services resource works. Clicking the link in that error message and hitting the button should make the issue go away. Apologies again for the error.

[camechis]

Also seeing this error. Seems kind of like a chicken and the egg when generating a project with terraform. I would have to generate the project only, then click the api, then continue on with terraform.

[ronlipke]

This also broke existing projects that are already being managed with terraform. Now we have to use the console to enable this api outside of our terraform ci pipeline and version control.

[TaysirTayyab]

@paddycarver Can you describe how this change fits into pipelines? The change essentially makes it to so that you need the api to use the resource, but you need the resource to enable the API.

Are end-users expected to just enable the API manually each time?

[mlauter]

I agree with the commenters above about the issues this change introduces. In addition to that, I have enabled the service usage api by hand for one of my projects, and I'm still getting this error. The error message says If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry, however, it's been about an hour now.

@paddycarver is project_id=764086051850 an internal project that also needs this api enabled? That's not a project in my org, and I see it's been referenced in other issue threads. e.g. #974

[camechis]

I did just notice one thing that may help. I Believe this api has to be enabled on the project executing terraform. For instance I have a terraform project setup that has an account to be able to create other projects and the tfstate is stored in this project. It seemed like this was the account that needed the api. Not the one being created.

[mlauter]

yeah @camechis i just realized that too. In that light, this seems alright. Still a bit annoying to have to do this manually for the project we're running tf from, but not a big deal.

[paddycarver]

Apologies for an unclear answer, everyone. Here's how this should be working (and works in my tests):

You have a service account you're authenticating with, and it belongs to a project. Let's call that project A.

You have a project that is being created, presumably by a google_project resource, and having services enabled on it. Let's call this project B.

My reproductions and experiments show only project A needs the API turned on, project B does not. So it's still perfectly valid to create a project in a terraform config, and enable a service on it, all without clicking in the console for every project. You just need to click once, for the project the service account belongs to.

If this is not the case for anyone, or there's a workflow out there that this still breaks, we definitely want to hear about it, and will find a path forward to unblock that workflow again.

Finally, I believe the number (764086051850) being specified as the project is just the number ID assigned to it at creation; if you click the link, you should see either project A or project B.

Sorry for all the confusion and messiness here. I'll see if we can't get the docs updated today to better explain some of this.

[camechis]

@paddycarver Thats exactly how its working for me. This is less of an issue now that this is cleared up (at least for me ).

[mlauter]

Thanks for the follow up @paddycarver this makes a lot more sense now! The one thing I'll say is clicking the link directly didn't work for me. It didn't lead to my project (but i just had to select my project, so not a huge deal).

[TaysirTayyab]

@paddycarver That makes more sense. Thank you so much!

[shamil]

@paddycarver, something definitely wrong here...

I get this error too with exactly same project id 764086051850. Clicking the link obviously not working, because I don't have any project with that id. Not sure where the id come from.

I have even more weird behavior. I get this error when creating redis resource. Even thought I get this error and terraform exits the redis resource actually still being created.

[morgante]

For some context, 764086051850 is a Google project which is part of the gcloud sdk. When you generate application default credentials (ex. by executing gcloud auth application-default login) the generated credentials are actually tied to that project. That project does not have the serviceusage API enabled, which #1522 depends upon.

I think rolling back #1522 should be considered. It's a very major change and will break several workflows in Terraform. Specifically, it depends on the service-usage API being enabled in the Terraform execution project. This is doable (if annoying) for people using service accounts, but it completely breaks for people using application default credentials from gcloud to execute Terraform. At a minimum, we should try to get the serviceusage API activated in the gcloud default project before keeping this.

/cc @danawillow

[shamil]

thanks for clarification about the id @morgante.
I do use application default credentials, so this affects me pretty heavily. So I vote up for rolling back #1522 unless there is a workaround...

[danawillow]

Hey all, sorry about this change- it was not our intent to break all of you, and I recognize that it sucks a lot to think that you're getting an "upgrade" and then have everything stop working.

The team that owns this API at Google is aware of the issue and working on a fix. In the meantime, if you're currently using user credentials (i.e. gcloud credentials), try using a service account instead. That sometimes fixes it. If that doesn't work, you'll need to keep using v1.12 for the next day or two while we get this sorted. Thanks for your patience!

[danawillow]

All right, I checked with the team and the fix has been rolled out- if you're still seeing this particular problem, please post a gist of your debug logs so we can see if there's anything else going on.

(leaving the issue open for the time being just in case- I'll close it once we're confident the issue's been solved)

[shamil]

@danawillow I'm still getting error when trying to create google_redis_instance resource.

Some notes:

1. I use application default credentials.
2. 764086051850 is the project id that owned by gcloud sdk as mentioned by @morgante.

I've the following config:

resource "google_redis_instance" "redis" {
  name           = "redis"
  memory_size_gb = 4
  tier           = "BASIC"
}

I get the following error when applying above configuration, which complains about 764086051850 project id:

Error: Error applying plan:

1 error(s) occurred:

* google_redis_instance.redis: 1 error(s) occurred:

* google_redis_instance.redis: Error waiting to create Instance: Error waiting for Creating Instance: googleapi: Error 403: Cloud Memorystore for Redis API has not been used in project 764086051850 before or it is disabled. Enable it byvisiting https://console.developers.google.com/apis/api/redis.googleapis.com/overview?project=764086051850 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

Maybe, the team at Google should enable Memorystore in 764086051850 project?

[danawillow]

Ah yeah, that's a different one- I'll reach out to them and see what I can do.

[danawillow]

@shamil and others who are using application default credentials: do you have a use case that requires you to do so, or was it just easier than downloading a service account key? I'm still talking to people internally about this (it's more complicated than it seems), but in the meantime using a service account will work.

[a13x212]

I'm running into the same problem.

Error waiting for api to disable: googleapi: Error 403: Service Usage API has not been used in project 104878349458 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=104878349458 then retry.

[morgante]

@a13x212 The fix for that is to enable the Service Usage API in your project (using the link in that message).

[shamil]

@shamil and others who are using application default credentials: do you have a use case that requires you to do so, or was it just easier than downloading a service account key? I'm still talking to people internally about this (it's more complicated than it seems), but in the meantime using a service account will work.

@danawillow this is how our team was told to configure it. This is much easier and doesn't require each user to generate the credentials keys then download them and keep them safe. They just run gcloud auth application-default login and they are done.

So basically you say that using default application credentials is no more recommended. If so I will accept that and migrate to using service account key, but if this something that is going to be fixed soon, I'll prefer to wait for the fix.

Btw, I don't really experience any issues with the latest version besides this issue with redis_instance, which by the way even if terraform throwing the above error, the instance actually was created. The problem is that terraform doesn't save its state, due to the error.

[paddycarver]

Hey @shamil,

So, just as a background on what's going on here, when you use Application Default Credentials, you're really using OAuth2 credentials for a project that Google controls. Historically, most services have been enabled on that project, but for complicated reasons, the Service Usage and Redis APIs were not, and only Google could fix that.

So you're still generating credentials, and you still need to keep them safe. The difference is that you have one command to generate those credentials conveniently.

The Googlers that work on this provider are working on this issue, and we definitely want to make sure there's a reasonable story for everyone that uses the provider today for how they're going to use it in the future, whether they use Application Default Credentials or a service account.

However, to your point of whether it's recommended, I don't believe it has ever been recommended (by the maintainers of the provider, can't really speak to anyone else) that Application Default Credentials be used. We allow for them, sure, but because they need to be tied to a user account, we don't run our tests against them nightly like we do with service account credentials. And they work in a way that is a bit different, and a bit weird. And you can't lock down which permissions they have. So we really recommend the use of service accounts as the best way to work with the provider.

To be clear though, even though you're encountering the error through Terraform, anything that uses Application Default Credentials to interact with these APIs will have the same problem.

I am, sincerely, very sorry for the way this release rolled out, and we take our responsibility to learn from it and prevent it from recurring very seriously. I do believe, however, we are out of action items for this issue, and that everyone's issues should be resolved. If you continue to experience issues, please feel free to comment below and we'll reopen the issue, or feel free to start a new issue.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!