[FR] Add an option to use config file over env vars.

[loomlike]

Willingness to contribute

Yes. I can contribute a fix for this bug independently.

Feature Request Proposal

Give options to read config variables either from env or yaml file.

Motivation

Currently, feathr client always use env vars regardless of the configured variables in the feathr yaml file.
This behavior is sometimes confusing since the yaml config is explicitly written while the env vars are hidden and implicit.

I think some of customers already get used to use env vars over the config file and thus changing the priority of reading config variables may cause problems to the existing workflows and pipelines.

So I'm proposing introducing a flag to FeathrClient to indicate which variables (env or config yaml file) the feathr client should read.

Details

No response

What component(s) does this feature request affect?

• Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
• Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
• Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
• Feature Registry Web UI: The Web UI for feature registry. Written in React