Automated documentation update for PR #119 [skip ci]

docs/_build/dagrunner.config.md

@@ -0,0 +1,139 @@
+# module: `dagrunner.config`
+
+[Source](../../dagrunner/config.py#L0)
+
+This module handles the run-time configuration of the dagrunner library.
+
+Certain hooks are present in the library for providing detailed control over
+dagrunner run-time.
+
+The configuration of dagrunner follows a first-in first-out approach on parsing
+configuration options and is handled by :class:`dagrunner.config.GlobalConfiguration`.
+This means that any number of configuration files can be parsed.  On import,
+`dagrunner.cfg` is parsed when present in the dagrunner root folder.  Each successive
+configuration file parsing will override existing parameter values.
+
+see [class: dagrunner.utils.Singleton](dagrunner.utils.md#class-singleton)
+
+## GlobalConfiguration: `CONFIG`
+
+## class: `GlobalConfiguration`
+
+[Source](../../dagrunner/config.py#L28)
+
+### Call Signature:
+
+```python
+GlobalConfiguration(*args, **kwargs)
+```
+
+The global configuration class handles any number of configuration files,
+where subsequent configuration entries act to override previous entries
+parsed.
+
+All group names parsed are prefixed with "dagrunner" and all entries are then
+parsed strictly.  Those groups not with this prefix are silently ignored.
+
+The following represents a description of the runtime configuration
+options::
+
+    # Graph visualisation
+    [dagrunner_visualisation]
+    enabled
+    title
+    collapse_properties
+    backend
+    output_filepath
+    group_by
+    label_by
+
+    [dagrunner_runtime]
+    # cache disable/enable: None/False/True.
+    # None implies enabled only if cache_dir is set.
+    cache_enabled
+    # if not specified and cache enabled, uses temp directory 'dagrunner_cache'
+    # in temp folder
+    cache_dir
+
+    # Logging
+    [dagrunner_logging]
+    enabled
+    host
+    port
+
+### function: `__getitem__`
+
+[Source](../../dagrunner/config.py#L172)
+
+#### Call Signature:
+
+```python
+__getitem__(self, key)
+```
+
+### function: `__init__`
+
+[Source](../../dagrunner/config.py#L83)
+
+#### Call Signature:
+
+```python
+__init__(self)
+```
+
+Initialize self.  See help(type(self)) for accurate signature.
+
+### function: `__repr__`
+
+[Source](../../dagrunner/config.py#L90)
+
+#### Call Signature:
+
+```python
+__repr__(self)
+```
+
+Return repr(self).
+
+### function: `__setitem__`
+
+[Source](../../dagrunner/config.py#L175)
+
+#### Call Signature:
+
+```python
+__setitem__(self, key, value)
+```
+
+### function: `__str__`
+
+[Source](../../dagrunner/config.py#L87)
+
+#### Call Signature:
+
+```python
+__str__(self)
+```
+
+Return str(self).
+
+### function: `parse_configuration`
+
+[Source](../../dagrunner/config.py#L146)
+
+#### Call Signature:
+
+```python
+parse_configuration(self, filename)
+```
+
+Parses a new configuration file.
+
+Entries in 'filename' override existing entries in the configuration,
+while entries not set remain unchanged from the previous state.
+
+Parameters
+----------
+filename : str
+    Name of the configuration file to read.
+

docs/_build/dagrunner.events.md

@@ -0,0 +1,64 @@
+# module: `dagrunner.events`
+
+[Source](../../dagrunner/events.py#L0)
+
+## Overview
+DAGrunner defines two special singleton events that plugins can return to control the
+execution flow of their graph.
+
+- event.IGNORE - removes a particular input from further processing.
+- event.SKIP -  aborts the execution of a plugin (and all downstream nodes) when any
+  input carries this event.
+
+
+### event.IGNORE
+When a plugin returns `event.IGNORE`, the immediate descendant node filters out that
+input.
+The remaining inputs of that plugin are utilised by that node as normal.
+If all inputs to a node are `event.IGNORE`, the node's execution is ignored, and a
+`event.IGNORE` event is returned instead, ignoring execution through all descendants.
+
+```mermaid
+---
+title: event.IGNORE (filtering some)
+---
+graph
+    cycle1{cycleX}
+    cycle1 --> Input1 -- filepath --> Load1 --cube1 --> Proc
+    cycle1 --> Input2 -- filepath --> Load2 -- event.IGNORE --> Proc
+    cycle1 --> Input3 -- filepath --> Load3 -- cube3 --> Proc
+    cycle1 --> Input4 -- filepath --> Load4 -- event.IGNORE --> Proc
+    Proc -- cube --> Save
+    Proc["Proc (cube1, cube3)"]
+```
+Here, Input2 and Input4 return an IGNORE event, likely due to there being missing data.
+Only the non-ignored cubes (`cube1` and `cube3`) reach `Proc`; the ignored inputs are
+dropped.
+
+### `event.IGNORE`
+The SKIP event differs from the IGNORE event in that if **any** input to a plugin is a
+SKIP event, node execution is skipped and it instead propagates this skip event so that
+all dependent nodes and their descendants aren't executed.
+
+```mermaid
+---
+title: event.SKIP
+---
+graph
+    cycle1{cycleX}
+    cycle1 --> Input1 -- filepath --> Load1 -- cube --> Proc
+    cycle1 --> Input2 -- filepath --> Load2 -- event.SKIP --> Proc
+    cycle1 --> Input3 -- filepath --> Load3 -- cube --> Proc
+    cycle1 --> Input4 -- filepath --> Load4 -- event.IGNORE --> Proc
+    Proc["Proc<br><sup>(not called)"] -- event.SKIP --> Save["Save<br><sup>(not called)"]
+```
+Because `Input2` returns a SKIP event, the Proc node and everything that follows aren't
+executed and neither is the Save node since the skip is propagated along the execution
+graph.
+
+see [class: dagrunner.utils.Singleton](dagrunner.utils.md#class-singleton)
+
+## _IgnoreEvent: `IGNORE_EVENT`
+
+## _SkipEvent: `SKIP_EVENT`
+

docs/_build/dagrunner.execute_graph.md

@@ -0,0 +1,171 @@
+# module: `dagrunner.execute_graph`
+
+[Source](../../dagrunner/execute_graph.py#L0)
+
+see [GlobalConfiguration: dagrunner.config.CONFIG](dagrunner.config.md#globalconfiguration-config)
+
+see [class: dagrunner.utils.CaptureProcMemory](dagrunner.utils.md#class-captureprocmemory)
+
+see [class: dagrunner.plugin_framework.NodeAwarePlugin](dagrunner.plugin_framework.md#class-nodeawareplugin)
+
+see [class: dagrunner.utils.TimeIt](dagrunner.utils.md#class-timeit)
+
+see [function: dagrunner.utils.as_iterable](dagrunner.utils.md#function-as_iterable)
+
+see [module: dagrunner.events](dagrunner.events.md#module-dagrunnerevents)
+
+see [function: dagrunner.utils.function_to_argparse_parse_args](dagrunner.utils.md#function-function_to_argparse_parse_args)
+
+see [function: dagrunner.utils.get_object_dot_module_path](dagrunner.utils.md#function-get_object_dot_module_path)
+
+see [module: dagrunner.utils.logger](dagrunner.utils.logger.md#module-dagrunnerutilslogger)
+
+see [function: dagrunner.utils.networkx.visualise_graph](dagrunner.utils.networkx.md#function-visualise_graph)
+
+## class: `ExecuteGraph`
+
+[Source](../../dagrunner/execute_graph.py#L318)
+
+### Call Signature:
+
+```python
+ExecuteGraph(networkx_graph: str, networkx_graph_kwargs: dict = None, <function plugin_executor>, scheduler: str = 'multiprocessing', num_workers: int = 1, profiler_filepath: str = None, config_filepath: str = None, dry_run: bool = False, verbose: bool = False, **kwargs)
+```
+
+### function: `__call__`
+
+[Source](../../dagrunner/execute_graph.py#L443)
+
+#### Call Signature:
+
+```python
+__call__(self)
+```
+
+Call self as a function.
+
+### function: `__init__`
+
+[Source](../../dagrunner/execute_graph.py#L319)
+
+#### Call Signature:
+
+```python
+__init__(self, networkx_graph: str, networkx_graph_kwargs: dict = None, <function plugin_executor>, scheduler: str = 'multiprocessing', num_workers: int = 1, profiler_filepath: str = None, config_filepath: str = None, dry_run: bool = False, verbose: bool = False, **kwargs)
+```
+
+Execute a networkx graph using a chosen scheduler.
+
+Args:
+- `networkx_graph` (networkx.DiGraph, callable or str):
+  Python dot path to a `networkx.DiGraph` or tuple(edges, settings) object, or
+  callable that returns one.  When called via the library, we support passing
+  the `networkx.DiGraph` or `tuple(edges, settings)` objects directly.  Note
+  that 'settings' represent a mapping (dictionary) between node and the node
+  attributes.  When provided, DAGrunner will attempt to convert this tuple into
+  a networkx through the following pseudo-code:
+    1. Copy node identity properties into the node attributes dictionary
+      and remove any attributes that are 'None' ('settings' from the tuple
+      provided).
+    2. Construct an empty networkx.DiGraph object.
+    3. Add edges to this graph ('edges' from the tuple provided).
+    4. Add node to attributes lookup to this graph ('settings' from the tuple
+      provided).
+    It is recommended that the user instead provide the networkx graph directly
+    rather than relying on DAGrunner to decide how to construct it.
+- `networkx_graph_kwargs` (dict):
+  Keyword arguments to pass to the `networkx_graph` when it represents a
+  callable.  Optional.
+- `plugin_executor` (callable):
+  A callable object that executes a plugin function or method with the provided
+  arguments and keyword arguments.  By default, uses the `plugin_executor`
+  function.  Optional.
+- `scheduler` (str):
+  Accepted values include "ray", "multiprocessing" and those recognised
+  by dask: "threads", "processes" and "single-threaded" (useful for debugging)
+  and "distributed".  See https://docs.dask.org/en/latest/scheduling.html.
+  Optional.
+- `num_workers` (int):
+  Number of processes or threads to use.  Optional.
+- `config_filepath` (str):
+  Path to the configuration file.  See [dagrunner.config](dagrunner.config.md).
+  Optional.
+- `dry_run` (bool):
+  Print executed commands but don't actually run them.  Optional.
+- `profiler_filepath` (str):
+  Output html profile filepath if supported by the chosen scheduler.
+  See https://docs.dask.org/en/latest/diagnostics-local.html
+  Optional.
+- `verbose` (bool):
+  Print executed commands.  Optional.
+- `**kwargs`:
+  Optional global keyword arguments to apply to all applicable plugins.
+
+### function: `visualise`
+
+[Source](../../dagrunner/execute_graph.py#L440)
+
+#### Call Signature:
+
+```python
+visualise(self, **kwargs)
+```
+
+## dict: `SCHEDULERS`
+
+## function: `main`
+
+[Source](../../dagrunner/execute_graph.py#L454)
+
+### Call Signature:
+
+```python
+main()
+```
+
+Entry point of the program.
+Parses command line arguments and executes the graph using the ExecuteGraph class.
+
+## function: `plugin_executor`
+
+[Source](../../dagrunner/execute_graph.py#L83)
+
+### Call Signature:
+
+```python
+plugin_executor(*args, call=None, verbose=False, dry_run=False, common_kwargs=None, node_id=None, **node_properties)
+```
+
+Executes a plugin callable with the provided arguments and keyword arguments.
+
+Plugins can be functions or classes.  If a class, it is instantiated with the
+keyword arguments provided in the `call` tuple.  The plugin callable is then
+executed with the positional arguments provided in `args` and the keyword arguments
+provided in the `call` tuple.  A plugin call is skipped if 1 or more of the `args`
+is the `SKIP_EVENT` object.
+
+Args:
+- `*args`: Positional arguments to be passed to the plugin callable.
+- `call`: A tuple containing the callable object (plugin) or python dot path to one
+  and optionally keyword arguments on instantiating and calling to that plugin:
+  - `(CallableClass, kwargs_init, kwargs_call)` -> `CallableClass(**kwargs_init)(*args, **kwargs_call)`
+  - `(CallableClass, {}, kwargs_call)` -> `CallableClass()(*args, **kwargs_call)`
+  - `(CallableClass)` - `CallableClass()(*args)`
+  - `(callable, kwargs)` -> `callable(*args, **kwargs)`
+  - `(callable)` -> `callable(*args)`
+- `verbose`: A boolean indicating whether to print verbose output.
+- `dry_run`: A boolean indicating whether to perform a dry run without executing
+  the plugin.
+- `common_kwargs`: A dictionary of optional keyword arguments to apply to all
+  applicable plugins.  That is, being passed to the plugin initialisation and or
+  call if such keywords are expected from the plugin.  This is a useful alternative
+  to global or environment variable usage.
+- `**node_properties`: Node properties.  These will be passed to 'node-aware'
+  plugins.
+
+Returns:
+- The result of executing the plugin function or method.
+
+Raises:
+- ValueError: If the `call` argument is not provided.
+

docs/_build/dagrunner.md

@@ -0,0 +1,20 @@
+# module: `dagrunner`
+
+[Source](../../dagrunner/__init__.py#L0)
+
+see [class: plugin_framework.DataPolling](dagrunner.plugin_framework.md#class-datapolling)
+
+see [class: plugin_framework.Input](dagrunner.plugin_framework.md#class-input)
+
+see [class: plugin_framework.NodeAwarePlugin](dagrunner.plugin_framework.md#class-nodeawareplugin)
+
+see [class: plugin_framework.Plugin](dagrunner.plugin_framework.md#class-plugin)
+
+see [class: plugin_framework.Shell](dagrunner.plugin_framework.md#class-shell)
+
+see [module: events](dagrunner.events.md#module-dagrunnerevents)
+
+see [module: plugin_framework](dagrunner.plugin_framework.md#module-dagrunnerplugin_framework)
+
+see [module: utils](dagrunner.utils.md#module-dagrunnerutils)
+