Follow pureconfig not json load/loads

[dwsmith1983]

With load and loads, we can only load from string config or file config. However, if we have a config tree, we would have to call __parse. I know Python doesn't really have private functions, but if we used the pureconfig style, we would have

fromString
fromFile
fromConfig

Why would we want a fromConfig? In a pipeline, generally, there will be a master config for the framework configuration and user configs for their code.

pipeline {
  framework {}
  usercode_1 {}
  usercode_2 {}
  frameowrk_2 {}
}

In the above example, we wouldn't know or care about the dataclasses of the user so we couldn't build the nested dataclass config parsing. We would just want to call by key their tree and send it to their code for them to handle. In the orchestration code, we would read in the file config and then pass config trees to the user code.

config = load()
framework = config[framework]        # but want to parse this into a dataclass but we have a config tree
call_user_code(config[user_code_1])  # same idea here

Or maybe just do load and loadOrThrow and behind the scenes we determine from string, from file, from config tree.

https://pureconfig.github.io/docs/loading-a-config.html

Here is an example from some pipeline code in Scala I have. I get the config from as key in the larger config from the framework and I parse a config object since it is no long a string or a file.

object DeequDataFlow extends ConfigurableInstance {

  case class RulesTable(query: String)

  case class InstanceConfig(
                             inputSource: InputType,
                             metricsRepoPath: Option[String],
                             tags: Option[Map[String, String]],
                             checkLevel: String,
                             description: String,
                             dqQuery: RulesTable
                           )

  override def createFromConfig(conf: Config): DeequDataFlow =
    new DeequDataFlow(pureconfig.ConfigSource.fromConfig(conf).loadOrThrow[InstanceConfig])
}

@zifeo thoughts?

[zifeo]

@dwsmith1983 thanks for the proposal. The load/loads comes from usual json/parsing library in python. I would favour having a new a generic method loadOrThrow as you suggest.

[dwsmith1983]

@zifeo what is the best way to work together communication wise? I feel PR comments and discussion seem to be slow. Is there a better way we can communicate to plan on the code?

[dwsmith1983]

@zifeo how do you want to approach refactoring to allow for from file, string, config what about url? Here is the trait in pureconfig for executing this plus the loadOrThrow.

https://github.com/pureconfig/pureconfig/blob/master/core/src/main/scala/pureconfig/ConfigSource.scala

[zifeo]

@dwsmith1983 I would add a generic loadOrThrow and alias the current load/loads with dataconf.file(), dataconf.string(), dataconf.config(). If you have better, feel free to suggest?

[dwsmith1983]

@zifeo can you create a dev branch I can have access to? This way we can collaborate on a branch without needing to PR to main and work out the code there.

[zifeo]

Added you as a colab, feel free to add the dev branch.

[dwsmith1983]

@zifeo thanks, I created the dev branch. Haven't started but pushed it up

[dwsmith1983]

@zifeo we are writing a blog post at my work about our usage of your library. It replicates a lot of conveniences we have in Scala. I also shared it on the Python group on LinkedIn. We will try to help the library get more traction in usage.

It isn't up yet but will be available at tech.trueanalytics.ai

[zifeo]

@dwsmith1983 Great! You can add the blog post in at the bottom of the readme!

[dwsmith1983]

@zifeo I created secondary methods for load and loads (from_file, from_string) and updated __parse to from_config and gave path="" as a default param to make usage easy; that is, the user can skip it when parsing a config.

I added deprecation warnings for load and loads. How should we roll it out? In pieces or all at once?

1. Update now to get the deprecation out and figure out the more complex relationship of loadOrThrow and wrapping up the parsing options from_config, from_string, from_file.
2. Wait until everything is good to go

Any thoughts of changes from the small changes so far? Do we need that path parameter in __parse (now from_config)?

[zifeo]

@dwsmith1983 Deprecating load and loads is not that necessary as it eases new user used to similar pythonic api. Regarding the roll out, I would go for the second approach.

[dwsmith1983]

@zifeo the one caveat to number two is the uncertainty on how easy or tedious it may be. It could take some time to complete all in one.

[dwsmith1983]

@zifeo this should be done I believe with your update--at lest most of it.

[zifeo]

@dwsmith1983 Yes!