Skip to contents

Run a pipeline

Source: R/pipeline_run.R
pipeline_run.Rd

pipeline_run() manages the running of a user defined pipeline simplifying object caching and configuration management across different scenarios.

Usage

pipeline_run(
  x,
  ...,
  scenario = NULL,
  scenario_default = getOption("pipeline.scenario_default", "default"),
  pipeline_dir = ".",
  relative_config_file = getOption("pipeline.config_file", "config.R"),
  relative_function_dir = getOption("pipeline.function_dir", "R"),
  relative_output_dir = getOption("pipeline.output_dir", "output"),
  force = FALSE,
  saveRDS_args = list(),
  readRDS_args = list(),
  return = TRUE
)

Arguments

x

[expression]

R Expression of pipeline assignments. Normally this will involve multiple assignments and will need to be embraced to represent a single expression.

...

Not currently used.

scenario

If NULL, any configuration file (see config_file) will be read in as is.

Character input will be treated as configurations you wish to loop over. In this case the configuration file will first be parsed looking for a named-list entry corresponding to the value of the scenario_default argument. The chosen scenarios are then layered on top of this default using utils::modifyList().

scenario_default

[character]

The default scenario to consider.

If scenario is NULL this will represent the folder under the output_dir where file output and cached objects are stored.

If scenario is not NULL, this represents the default scenario on which the specified scenarios are layered.

pipeline_dir

[character]

The directory you wish to run the the pipeline relative to.

relative_config_file

[character]

The configuration file.

Must be a none-nested and given relative to pipeline_dir.

relative_function_dir

[character]

The directory to look for user defined pipeline functions.

Must be a none-nested and given relative to pipeline_dir.

relative_output_dir

[character]

The directory for output.

Must be a none-nested and given relative to pipeline_dir.

force

[bool]

Do you want to force a run of the pipeline.

If TRUE, then cached objects are removed and the pipeline is (re)run.

If a character vector then the corresponding object(s) are removed from the cache and the pipeline is rerun.

saveRDS_args

[list]

List of additional arguments passed to saveRDS().

This argument allows you to pass additional arguments to that function (e.g. list(compress = "zstd"))

readRDS_args

[list]

List of additional arguments passed to readRDS().

return

[bool]

Should the output be returned.

Defaults to TRUE, but when running across multiple scenarios and with outputs that use a large amount of memory it can be useful to set FALSE.

If FALSE, a successful run will return NULL invisibly.

Value

A named list of outputs for each configuration.