Version:

salvus.flow.executors.task_chain.task_chain_controller

salvus.flow.executors.task_chain.task_chain_controller salvus flow executors task_chain task_chain_controller
The task chain controller implementation.

Classes

StdOutErr

The stdout and stderr of a single task chain.
Both output could be None in case they don't exist.
SIGNATURE
class StdOutErr(builtins.object):
    def __init__(
        self, stdout: dict | None, stderr: dict | None, timings: dict | None
    ) -> None: ...
ARGUMENTS
Required
stdout
Type:dict | None
Description:
The stdout.
Required
stderr
Type:dict | None
Description:
The stderr.
Required
timings
Type:dict | None
Description:
The timings for the given task chain.

Methods

StdOutErr.get_diagnosis_string()
Get a hopefully useful diagnostic string from the task chain output.
SIGNATURE
def get_diagnosis_string(self) -> str: ...

TaskChainController

The TaskChainController is responsible for distributing TaskChainRunners across resources.
SIGNATURE
class TaskChainController(builtins.object):
    def __init__(
        self,
        task_chain_tasks: dict[str, list],
        sites: dict[str, TaskChainSiteConfig],
        task_chain_runner_context: typing.Callable | None = None,
    ): ...
ARGUMENTS
Required
task_chain_tasks
Type:dict[str, list]
Description:
A dictionary mapping an arbitrary job key (name) to a list of tasks, e.g. tasks that happen within one task chain.
Required
sites
Type:dict[str, TaskChainSiteConfig]
Description:
A dictionary of identifier + site config for sites to run the tasks on. The identifier does not have to be the site name.
Optional
task_chain_runner_context
Type:typing.Callable | None
Default value:None
Description:
A optional function returning a context manager that will be used in each TaskChainRunner prior to running the actual task chains.

Methods

TaskChainController.check_status_of_task_chain_runners()
Check the status of the individual task chain runners. Will return nothing if all finished, otherwise an exception with more information will be thrown.
SIGNATURE
def check_status_of_task_chain_runners(
    self, raise_on_error: bool = True
) -> None: ...
ARGUMENTS
Optional
raise_on_error
Type:bool
Default value:True
Description:
If True, an exception will be raised if one of the child local task chain runner fail (not the individual task chains). If False, a warning will be logged.
TaskChainController.delete_remote_files()
Delete all remote files for the task chain controller - in most cases, nothing should remain after this method has been called.
SIGNATURE
def delete_remote_files(self, verbosity: int = 1) -> None: ...
ARGUMENTS
Optional
verbosity
Type:int
Default value:1
Description:
The verbosity.
TaskChainController.get_site_and_remote_path_for_task_chain()
Get the site + remote path for a given task chain.
SIGNATURE
def get_site_and_remote_path_for_task_chain(
    self, identifier: str
) -> tuple[BaseExecutor, _RemotePathType]: ...
ARGUMENTS
Required
identifier
Type:str
Description:
Identifier of the task chain to get.
TaskChainController.get_status_for_all_task_chains()
Get a dictionary with the status of every individual task chain.
This is most useful after the controller finished running.
The status for a task chain might be None, if it has not been set.
SIGNATURE
def get_status_for_all_task_chains(self) -> dict[str, JobStatus | None]: ...
TaskChainController.get_stdouterr_for_task_chain()
Get the stdout and stderr for a given task chain.
SIGNATURE
def get_stdouterr_for_task_chain(self, identifier: str) -> StdOutErr: ...
ARGUMENTS
Required
identifier
Type:str
Description:
Identifier of the task chain to get.
TaskChainController.launch()
Launch the remote task chain runners.
SIGNATURE
def launch(self) -> None: ...
TaskChainController.run()
Run the task chains across the individual sites by dispatching to one task chain runner per site.
The task chain runner will take care of the parallelization per site.
SIGNATURE
def run(self, assert_no_failed_runners: bool = True) -> None: ...
ARGUMENTS
Optional
assert_no_failed_runners
Type:bool
Default value:True
Description:
If True, an exception will be raised if one of the child local task chain runner fails (not the individual task chains). If False, a warning will be logged. This is useful for error resilient workflows but will require error handling elsewhere.
TaskChainController.update_status()
Update the status of all task chain runners.
SIGNATURE
def update_status(self) -> dict[str, JobStatus]: ...
TaskChainController.wait()
Wait until all task chains runners have finished or failed.
SIGNATURE
def wait(self) -> None: ...
PAGE CONTENTS