Config Server - PCF Auth breaks after 24 hours

[kamalakarp]

Describe the bug
I have deployed Spring Cloud Config Server application in PCF with HashiCorp Vault as backend. I'm using PCF cert Auth and it break after 1 day and it seems mainly due to PCF instance certificates are valid only for a day since PCF Diego Certification manager rotates instance certificate everyday. Some how config server is not reading the updated certificates and still using the old certificates using the environment variables - $CF_INSTANCE_KEY and $CF_INSTANCE_CERT.

Error message :

error:org.springframework.vault.authentication.VaultLoginException: Cannot login using PCF: x509: certificate has expired or is not yet valid; nested exception is org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: [{"errors":["x509: certificate has expired or is not yet valid"]} ]

Sample

Config Server App application.yaml

          kv-version: 2
          backend: kv_qas
          order: 1
          authentication: PCF
          pcf:
            role: aicshd-cit-qas-vault-config-server
            pcf-path: cf_sr1_aiccoreqasorg_aicshd-cit-qas

Vault Registration :

vault auth enable -path=cf_vdc02 cf
 
# Write Vault Config
vault write auth/cf_vdc02/config identity_ca_certificates=@ca.crt cf_api_addr=https://api.sausvdc02.com cf_username="<user_name>" cf_password="<password>" cf_api_trusted_certificates=@api.crt
 
# Bind the PCF Org and Space ID's
vault write auth/cf_vdc02/roles/<Name of the Config Server> bound_organization_ids=@vdc02_org_guid.txt bound_space_ids=@vdc02_space_guid.txt policies=<use-readonly-policy> disable_ip_matching=true ttl=86400s max_ttl=86400s period=86400s

I tried specifying the instance-key and cert manually in application.yaml but it's reading variables and failing to start.

            instance-key: '/etc/cf-instance-credentials/instance.key'
            instance-certificate: '/etc/cf-instance-credentials/instance.crt'

@mp911de / @spencergibb - did you hear about this issue ?

[devgig]

I am experiencing a similar issue.

[spencergibb]

@mp911de and idea?

[mp911de]

CF certificates are cached to avoid File I/O as we avoid I/O by default when accessing details that are required to build the authentication request. This is specifically for Spring Vault usage important when using Vault with reactive programming.

For Spring Config Server, this restriction doesn't seem necessary. The workaround is to construct PcfAuthenticationOptions with a non-cached ResourceCredentialSupplier(…).

[kamalakarp]

@mp911de - Thank you Mark for quickly finding it out. Not only for Spring Config Server, even for regular app this restriction is not needed, because PCF rotates the certificates everyday. Are we checking the validity of the instance certificate anywhere ?

[mp911de]

No, the certificates are taken as-is and sent to Vault. Essentially, they are opaque to us. Not sure what Vault is doing.

Regarding caching, that's a limitation imposed by configuration infrastructure in combination with reactive runtimes. Simply disabling caching will break other arrangements. What we actually need is a way to fetch certificates without blocking. I think @nebhale created a component that reloads the certificates in a keystore. In any case, that's a discussion for Spring Vault.

[kamalakarp]

ok. Can i add boolean variable called for caching and making it true by default and from when calling from Spring Config Server, we can make it as false. With this, we don't break the functionality but provide an option to turn on/off. Do you think it's a good idea ?

[spencergibb]

@mp911de is saying that there needs to be some changes in https://github.com/spring-projects/spring-vault/. Can you file an issue there and reference this one?

[scottfrederick]

I think @nebhale created a component that reloads the certificates in a keystore.

That's part of the Java buildpack for CF: https://github.com/cloudfoundry/java-buildpack-security-provider

[mp911de]

@mp911de is saying that there needs to be some changes in https://github.com/spring-projects/spring-vault/. Can you file an issue there and reference this one?

For the time being, it would make sense to apply the change in Spring Cloud Config server. Adding a non-blocking/declarative I/O abstraction in Spring Vault is a bigger change which will take a fair bit to change to put it in place.