You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Here are my thoughts about the mara config interface:
Phase 1
A config is done by a mara_config.py file. This file is placed in the root folder of the project or the path is specified via environment variable MARA_CONFIG. You can either provide a path to a directory or to the python file. When the directory is passed, the config should search for mara_config.py in the folder.
For now, I'd say the config should work as of today importing mara packages and patching them (and by that activating them). The logic could be duplicated in mara-cli and mara-app as they are currently the only projects invoking the config. But it is be possible as well to reactivate the mara-config package and placing the logic to import mara_config there.
Future ideas
For the future, I personally would like to get rid of all this monkey patching of configs which just return a scalar or dict value. I like the way how superset does it, see superset/config.py. One can still do fancy stuff in the config module, but by default, you just set global variables which are then used in the code. I am not 100% sure about this idea, because it still keeps us in the situation that one has to write the config manually.
For non-tech users I was thinking to add a mara init cli function which guides through setting up the config. For such a scenario a format which supports serialization would make more sense. E.g. yaml, toml or json.
One other idea I had in mind was that we could support a mapping from dbt config to mara which would make executioning of a dbt project throgh mara possible (e.g. via cli mara dbt run -s <model>). It would be powered by an integration with mara-catalog which has an integration to read dbt manifest files. But that's just a random idea into the blue...
Activity
leo-schick commentedon Dec 3, 2023
Here are my thoughts about the mara config interface:
Phase 1
A config is done by a
mara_config.pyfile. This file is placed in the root folder of the project or the path is specified via environment variableMARA_CONFIG. You can either provide a path to a directory or to the python file. When the directory is passed, the config should search formara_config.pyin the folder.For now, I'd say the config should work as of today importing mara packages and patching them (and by that activating them). The logic could be duplicated in mara-cli and mara-app as they are currently the only projects invoking the config. But it is be possible as well to reactivate the mara-config package and placing the logic to import mara_config there.
Future ideas
For the future, I personally would like to get rid of all this monkey patching of configs which just return a scalar or dict value. I like the way how superset does it, see superset/config.py. One can still do fancy stuff in the config module, but by default, you just set global variables which are then used in the code. I am not 100% sure about this idea, because it still keeps us in the situation that one has to write the config manually.
For non-tech users I was thinking to add a
mara initcli function which guides through setting up the config. For such a scenario a format which supports serialization would make more sense. E.g. yaml, toml or json.One other idea I had in mind was that we could support a mapping from dbt config to mara which would make executioning of a dbt project throgh mara possible (e.g. via cli
mara dbt run -s <model>). It would be powered by an integration withmara-catalogwhich has an integration to read dbt manifest files. But that's just a random idea into the blue...leo-schick commentedon Dec 3, 2023
@jankatins