Configuration property to export environment configuration properties to .properties or .yaml file

[GrahamHannington]

Problem

Spring Boot's (impressively!) flexible externalization of properties, with multiple property sources, can raise questions such as, "Did the application merge the properties from the multiple sources as expected?" And, more specifically, "Did the properties from one source get overridden as expected by the properties from another source?"

Note: I acknowledge that the questions "Did the application get properties from all of the expected sources?" and, more specifically, "Did the application load all of the expected configuration files?" are already answered by the configuration property logging.level.org.springframework.boot.context.config: trace, as described in answers to the Stack Overflow question "Is there a way to get list of loaded properties file in SpringBoot application?".

Proposed solution (enhancement)

A new configuration property to export the merged set of properties.

For example:

spring.config.export: ./exported-application.yaml

spring.config.export: ./exported-application.properties

where the export format is inferred from the file extension, as opposed to, say, specified by a separate spring.config.export.format property.

Example use case: specifying properties in multiple files

Suppose:

• You have a single, monolithic application.yaml file that specifies properties for forwarding data to multiple locations
• The locations are managed by different teams of people
• The application.yaml file is under source control management (SCM)

To avoid merging issues when different teams update that single file, it would be useful to split the file into multiple files, one per location.

Spring Boot's flexible externalization of properties means that there are multiple different ways to do that; multiple possible solutions.

The proposed spring.config.export property offers an easy way to compare different solutions and verify that they produce the same configuration as the original single file.

Related to configuration validation, but different

Questions about whether the configuration is as expected are closely related to, but distinct from, the question "Is the configuration valid"?

A set of configuration properties can be both (a) valid according to a set of rules that specify allowed structures and values and (b) not what you expect or intend.

Hence, I've created this separate issue, rather than adding a comment to issue #10030.

Why not use the existing configprops and env endpoints?

Reasons:

• I want to review properties before the application uses them to do whatever it does. I want the application to merge properties from the various sources (including setDefaultProperties() in the application main()), export them, and then exit. I don't want the application to start a web server.
• The endpoints publish properties in JSON format. I want to review the properties in the same format that I specify them in external configuration files. In my case, YAML.
• The env endpoint shows all property values from all sources, with property keys reported using the same dot notation as .properties files, but it doesn't report the merged ("last wins") property value. (Except, in my testing, for the value of server.port, which the endpoint reports under the property name local.server.port. I'm guessing this property is a special case; perhaps there are others.)
• The configprops endpoint does report the merged values, but parsing that structure into the equivalent YAML structure of an input configuration file seems like unnecessarily hard work. For example, given the single property management.endpoints.web.exposure.include: health,configprops,env in an input YAML configuration file, the configprops endpoint contains contexts.application.beans.management ... WebEndpointProperties.inputs.exposure.include, where include is an array of three identical objects, where each object contains the same key/value "value": "health,configprops,env". For now, I've decided to look away.

What exactly do I mean by "merge, export, and then exit"?

More specifically, I mean:

• Component scanning that uses a filter to only include some classes, such as those marked with @Configuration.
• spring.main.web-application-type=none (don't start a web server).
• A handler for the context refresh event that exports (writes) configuration properties to a file. For example, see the various answers to the Stack Overflow question "How to log the active configuration in a Spring Boot application?"; the only difference is writing to a separate file versus the log.
• Exits, because there are no active threads (such as a web server, or anything else the application normally does).

What I'm doing now: exporting to .properties, and then converting to .yaml

Thanks to more experienced colleagues, I'm already able to do most of this in experimental code. I code in other languages, but I'm a Java beginner.

I'm exporting to a .properties file, but not yet to a .yaml file.

I haven't yet invested time in, if you'll excuse the expression, "joining the dots" between the flattened, "compound" dot notation of the in-memory Spring Boot property keys and the nested structure of "atomic" property keys (including arrays, which might themselves contain nested structures) that I could pass to the SnakeYAML dump() method to write a .yaml file.

Instead, I'm using existing external tools to convert the .properties file to YAML. For example, the Environment Variable Generator website (Input: Spring Boot Properties; Output: Spring Boot YAML).

Notes:

• Environment Variable Generator website output nit: it wraps numeric property values in quotes, explicitly characterizing them as strings, not numbers. I'm using a JSON schema in Microsoft Visual Code editor to validate the YAML. If the schema defines a property as numeric, the quoted value gets reported as a validation error. I mention this because I'm hoping for a solution that is more type-aware: that exports values of numeric properties to YAML without enclosing quotes.

• I'd like to use the VS Code Spring Boot Tools extension "Convert .properties to .yaml", but I currently can't get that to work.

• Acknowledging that I'm being opportunistic, and that Stack Overflow is probably a better place to ask: if you feel like replying with Java code that converts in-memory Spring Boot configuration properties into an object that can be passed to the SnakeYAML dump() method to produce YAML with "atomic" (as opposed to "compound"; dotted) keys, please do! I'm aware of the GitHub project anubhavshukla/properties-to-yaml-converter, but I'm hoping for more concise code that begins with in-memory Spring Boot configuration properties and uses SnakeYAML for YAML serialization.

Implementation details: possible additional sub-properties

Sanitization

Example: spring.config.export.show-values: never|always

From the Spring Boot docs topic "Endpoints":

Sanitize Sensitive Values

Information returned by the /env, /configprops [...] can be sensitive

The same is true of this proposed new property.

Perhaps this new property should also have a corresponding show-values (sub-)property; although, perhaps the value when-authorized doesn't apply in this context.

And perhaps the same SanitizingFunction should apply. I'm watching issue #39243 with interest.

Only export properties from files

Example: spring.config.export.only-properties-from-files: true|false

Exporting all properties can result in too much information.

As mentioned, one of the primary use cases for this new property is for users to confirm that the properties that they specified in multiple configuration files have been merged as they expected.

In that case, users are primarily interested in properties sourced from configuration files; properties from other sources, such as environment variables, are effectively noise.

only-properties-from-files: true would only export properties from files; that is, property sources that match instanceof OriginTrackedMapPropertySource.

I'm applying this filter now in my experimental code, and the results are far more concise, and more useful for my purposes, than a comprehensive export of all properties, regardless of source.

A sub-property to filter exported property keys by regex patterns has occurred to me, but I think that's a lower priority.

Risk of exported .properties or .yaml file being inadvertently used as input?

Yes, it's a possibility. Maybe mention this in the docs for this property; for instance, recommend an export file name (and path) that Spring Boot won't automatically find and load.

The flipside: this property also offers an opportunity for condensing properties from multiple configuration files (back) into a single, functionally equivalent configuration file. That might be useful in some cases (to encapsulate environment variables, Java system properties, command-line arguments, multiple files, into a single resource).

[GrahamHannington]

(Risible?) ad hoc workaround: reporting property values in a custom banner

Possibly too much info (possibly also an easy target for derisive eyeball rolls, if that's your thing): here is an example of what I'm doing now in the absence of something like spring.config.export, and in situations where the application JAR doesn't contain the custom code that I described in an earlier comment to export that information.

Sometimes, when I want to check the "last wins" (or "did it get found?") values of a few specific application properties (which might be spread across multiple property key prefixes), and I don't feel like enabling the env endpoint, or finding in the JSON-format output of that endpoint each of the properties that I'm interested in, I'll temporarily set the application banner to an external file that refers to those properties.

e.g. spring.banner.location: file:./banner.txt, where banner.txt contains references such as:

my.custom-property.a: ${my.custom-property.a}
my.custom-property.a: ${my.custom-property.b}

Using this workaround makes me half-wish for a read-only "property" (set by Spring Boot itself) that "contains" (when referenced, returns) a serialization of all (or, since I'm dreaming 😉, selected-by-wildcarded-pattern) environment property keys and their values (e.g. ${spring.config.export.serialized.yaml}), but I don't want to go off piste here, it's not really what I'm asking for in this issue: as described in my initial comment, what I'd really like is a property that exports (or, if you prefer, "dumps") that content to an external file.

[GrahamHannington]

The proposed solution is intended primarily for application end users

To dispel any possible confusion caused by my mention of Spring Boot Tools, I thought it was worth clarifying that the primary intended users of the proposed new spring.config.export property are application end users.

While I imagine application developers and testers will also find such a property useful, my point is: the people who need this information don't necessarily have a Java IDE. They almost certainly won't have the application Java source.

So, with apologies if I'm stating the bleeding obvious, and the possibility had occurred to nobody but me: the solution isn't adding more smarts to a Java IDE extension such as Spring Boot Tools.

[philwebb]

We discussed this today on our call and unfortunately we don't think it's feasible to implement the functionality your looking for in a way that will work with every application. We're also not convinced that there will be that much demand for such a feature (vs the amount of time it would take for us to implement it).

You may be able to develop what you're looking for yourself in a way that will work with your own applications, if you accept certain limitations.

Specifically, Spring's Environment abstraction is backed by one or more org.springframework.core.env.PropertySource implementations and only a subset are org.springframework.core.env.EnumerablePropertySource. This means, you may be able to ask an Environment to resolve a property such as server.port, but you can't necessarily list all the server.port properties that are contained in the Environment.

The other problem, as you've already identified, is that we don't have a way to convert Properties into YAML. This isn't something we've needed to do and it's not really within the remit of Spring Boot.

I think your best option is to look at the existing code for the /configprops and /env actuator endpoints and see if you can adapt them for your needs. If you limit your application to only use EnumerablePropertySources, and you're happy writing out YAML that may not exactly line up with the original input, you might be able to get something to work.

One more thing to be aware of, you can system environment variables and command line flags to change properties. So each run of the application may result in different exported properties. E.g. java -jar myapp.jar --server.port=1234 will override server.port in application.yml.

[GrahamHannington]

Thanks for at least considering this issue ("for: team-meeting"). I understand that it's been declined.

I've converted the JSON from the configprops endpoint to the YAML I want

Although I would have preferred a different response 🙂, this prompt and unequivocal "not planned" response is helpful to me, because it means that I can pursue a homegrown solution knowing that it's unlikely to be replaced any time soon by a new (and, very likely, better!) out-of-the-box feature in Spring Boot.

To that end, I've written a script that gets the JSON from the configprops endpoint and then uses yq to convert it into YAML that is effectively identical to merging the multiple input YAML configuration files (in the "last wins" order defined by Spring Boot). Creating the yq pipeline to do that took me a few hours, but it was interesting work. The bean-centric nature of the data at that endpoint is not ideal for my purpose, but, in part thanks due to the particular binding choices in the application I'm working on, I've got the output I want.

[GrahamHannington]

Hi @philwebb,

Sorry, I stupidly missed your recent comment (thanks!) when I wrote my previous comment. (Maybe I forgot to refresh the page, or just didn't scroll up.)

certain limitations

Yeah, I didn't mention it in my previous comments, but I've been iterating over the EnumerablePropertySource sources to get what I need.

we don't have a way to convert Properties into YAML

Understood. I was planning to (a) convert the Spring Boot configuration properties into an object with "atomic" (not composite, dotted) keys that can be passed to the SnakeYAML dump() method, and then (b) call that method to do the conversion. But I'm currently baulking at writing the Java to do (a), due to limited time and my limited Java skills. Nor is it really within my remit. 😉

your best option is to look at the existing code for the /configprops and /env actuator

Thanks for the tip. I'm already—by iterating over the property keys in the various property sources, omitting duplicate keys, and then getting the "active"/"last wins" property value from the Spring environment—getting the properties I need in .properties format that I can then convert to YAML. But I'll still look at that code.

you can [set/use] system environment variables and command line flags to change properties

Understood. I really appreciate how Spring Boot loads properties from multiple property sources such as environment variables, Java system properties, and command-line arguments, not just multiple .properties and .yaml files. Thank you so much for such rich external configuration features!

We're also not convinced that there will be that much demand for such a feature

I can't speak about "much"; I defer to your judgement on that, no question. However, there's "some" (not just me): the nearly-11-year-old Stack Overflow question "How to log the active configuration in a Spring Boot application?" has been viewed 83k times, and there are 11 answers. The page for that question also shows multiple related questions, most of which are asking essentially the same question.

Thanks again for considering this issue and for your advice. And thanks for Spring Boot!