2 years ago
#366682
Greg Wilson
how to pass optional flags to pre-tasks using Python invoke?
I am using Invoke and have two tasks: one cleans a raw data file to produce a clean data file, and the other produces several plots from the clean data file:
RAW_FILE = "raw.csv"
CLEAN_FILE = "clean.csv"
PLOT_FILE = "plot.svg"
@task(optional=["logging"])
def clean_data(c, logging=None):
    """Produce cleaned-up dataset."""
    print("CLEAN", logging)
    _configure_logging(logging)
    df = clean_raw_data(RAW_FILE)
    df.to_csv(CLEAN_FILE, index=False)
@task(pre=[clean_data], optional=["logging"])
def plot_data(c, logging=None):
    """Create plots of data."""
    print("PLOT", logging)
    _configure_logging(logging)
    make_plot(CLEAN_FILE, PLOT_FILE)
def _configure_logging(log_level):
    """Initialize logging."""
    if log_level is not None:
        print("SETTING LOGGING TO", log_level)
        CONFIG["LOGGING_LEVEL"] = log_level.upper()
If I run:
$ invoke clean-data --logging info
then logging is set to INFO and I get a message from inside clean_raw_data. However, if I run:
$ invoke plot-data --logging info
then:
- clean_datais invoked with- logging=None, so no log message appears.
- plot_datais then invoked with- logging="info", so its log message appears.
My expectation was that command-line flags would be passed down to dependent tasks. I tried doing this manually:
@task(pre=[call(clean_data, logging=logging)], optional=["logging"])
def plot_data(c, logging=None):
    ...as before...
but this produces an error message because logging isn't defined at the point the @task decorator is invoked.
Is there a way to chain optional arguments in the desired fashioned?
python
invoke
0 Answers
Your Answer