Mondaic
This API reference is not for the latest stable Salvus version.

salvus.flow.executors.implementations.local_hpc

Local versions of all the HPC executors.

They still interact with the job queueing system. They do not use SSH to communicate but directly execute the commands.

Classes

LocalGridEngineExecutor

class LocalGridEngineExecutor(
    salvus.flow.executors.implementations.grid_engine.GridEngineExecutor,
    salvus.flow.executors.implementations.local.LocalExecutor,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ): ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir types

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

os_type salvus.flow.executors.remote_io_executor.OSType

Return the type of operating system.

sftp_client salvus.flow.executors.remote_io_executor.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.executors.remote_io_executor.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None: ...

Cancel the GridEngine job by calling qdel.

Parameters
  • job_dir str — The job directory.
Returns None
execute_command()
def execute_command(
    self,
    command: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]: ...

Execute the command.

Parameters
  • command str — The command to run as a string.
  • assert_ok bool — Raise if the command does not exist with a code of 0.
  • environment Optional[Dict[str, str]] — Environment variables to be set for this command.
Returns Tuple[int, List[str], List[str]]
get_job_status()
def get_job_status(self, job_dir: str) -> Union[
    salvus.flow.executors.types.JobStatus,
    List[salvus.flow.executors.types.JobStatus],
]: ...

Returns the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns Union[salvus.flow.executors.types.JobStatus, List[salvus.flow.executors.types.JobStatus]]
is_folder_writeable()
def is_folder_writeable(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Return True if the user has write permissions in the chosen folder.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path on the remote machine.
Returns bool
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.executors.types.JobStatus]: ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be set for GridEngine sites.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.executors.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.executors.executor_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.executors.types.JobStatus: ...

Launch a job on a GridEngine cluster using MPI.

Returns the initial status - always “pending” here.

Parameters
  • commands List[salvus.flow.executors.executor_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds Optional[int] — Wall-time in seconds. Must be set for GridEngine sites.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
Returns salvus.flow.executors.types.JobStatus
pretty_print()
def pretty_print(self) -> None: ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Check if a remote file or directory exists.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    localpath: Union[_io.BytesIO, pathlib.Path],
    show_progressbar: bool = False,
) -> None: ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_listdir()
def remote_listdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> List[str]: ...

List the contents of a remote directory.

Just dispatches to the underlying representation.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    mode: int = 511,
) -> None: ...

Create a directory on the remote or local machine.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    show_progressbar: bool = False,
) -> None: ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Where to copy to.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    verbosity: int = 1,
) -> None: ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    allow_failure: bool,
    verbosity: int = 1,
) -> None: ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None

LocalLSFExecutor

class LocalLSFExecutor(
    salvus.flow.executors.implementations.lsf.LSFExecutor,
    salvus.flow.executors.implementations.local.LocalExecutor,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ): ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir types

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

os_type salvus.flow.executors.remote_io_executor.OSType

Return the type of operating system.

sftp_client salvus.flow.executors.remote_io_executor.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.executors.remote_io_executor.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None: ...

Cancel the LSF job by calling bkill.

Parameters
  • job_dir str — Directory where job is running.
Returns None
execute_command()
def execute_command(
    self,
    command: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]: ...

Execute the command.

Parameters
  • command str — The command to run as a string.
  • assert_ok bool — Raise if the command does not exist with a code of 0.
  • environment Optional[Dict[str, str]] — Environment variables to be set for this command.
Returns Tuple[int, List[str], List[str]]
get_job_status()
def get_job_status(self, job_dir: str) -> Union[
    salvus.flow.executors.types.JobStatus,
    List[salvus.flow.executors.types.JobStatus],
]: ...

Return the status of the job or job array.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — Directory where job is running.
Returns Union[salvus.flow.executors.types.JobStatus, List[salvus.flow.executors.types.JobStatus]] — Either a single job status or a list of Job statuses if this is a job array.
is_folder_writeable()
def is_folder_writeable(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Return True if the user has write permissions in the chosen folder.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path on the remote machine.
Returns bool
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.executors.types.JobStatus]: ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be given for an LSF site.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.executors.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.executors.executor_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.executors.types.JobStatus: ...

Launch a job on LSF cluster using MPI.

Parameters
  • commands List[salvus.flow.executors.executor_utils.RemoteCommand] — List of commands to run.
  • job_dir str — Root directory of job, where all files will be copied.
  • working_dir str — Run directory of job, where Salvus will be executed.
  • stdout_path str — Path to the Salvus stdout file.
  • stderr_path str — Path to the Salvus stderr file.
  • wall_time_in_seconds Optional[int] — Max wall time in seconds for the job.
  • is_debug_job bool — Should we run this job in the debug queue, by default False
Returns salvus.flow.executors.types.JobStatus — Status of the job directly after submission.
pretty_print()
def pretty_print(self) -> None: ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Check if a remote file or directory exists.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    localpath: Union[_io.BytesIO, pathlib.Path],
    show_progressbar: bool = False,
) -> None: ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_listdir()
def remote_listdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> List[str]: ...

List the contents of a remote directory.

Just dispatches to the underlying representation.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    mode: int = 511,
) -> None: ...

Create a directory on the remote or local machine.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    show_progressbar: bool = False,
) -> None: ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Where to copy to.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    verbosity: int = 1,
) -> None: ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    allow_failure: bool,
    verbosity: int = 1,
) -> None: ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None

LocalPbsExecutor

class LocalPbsExecutor(
    salvus.flow.executors.implementations.pbs.PbsExecutor,
    salvus.flow.executors.implementations.local.LocalExecutor,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ): ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir types

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

os_type salvus.flow.executors.remote_io_executor.OSType

Return the type of operating system.

sftp_client salvus.flow.executors.remote_io_executor.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.executors.remote_io_executor.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None: ...

Cancel the PBS job by calling qdel.

Parameters
  • job_dir str — The job directory.
Returns None
execute_command()
def execute_command(
    self,
    command: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]: ...

Execute the command.

Parameters
  • command str — The command to run as a string.
  • assert_ok bool — Raise if the command does not exist with a code of 0.
  • environment Optional[Dict[str, str]] — Environment variables to be set for this command.
Returns Tuple[int, List[str], List[str]]
get_job_status()
def get_job_status(self, job_dir: str) -> Union[
    salvus.flow.executors.types.JobStatus,
    List[salvus.flow.executors.types.JobStatus],
]: ...

Returns the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns Union[salvus.flow.executors.types.JobStatus, List[salvus.flow.executors.types.JobStatus]]
is_folder_writeable()
def is_folder_writeable(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Return True if the user has write permissions in the chosen folder.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path on the remote machine.
Returns bool
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.executors.types.JobStatus]: ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be set for PBS sites.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.executors.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.executors.executor_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.executors.types.JobStatus: ...

Launch a job on a PBS cluster using MPI.

Returns the initial status - always “pending” here.

Parameters
  • commands List[salvus.flow.executors.executor_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds Optional[int] — Wall-time in seconds. Must be set for PBS sites.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
Returns salvus.flow.executors.types.JobStatus
pretty_print()
def pretty_print(self) -> None: ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Check if a remote file or directory exists.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    localpath: Union[_io.BytesIO, pathlib.Path],
    show_progressbar: bool = False,
) -> None: ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_listdir()
def remote_listdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> List[str]: ...

List the contents of a remote directory.

Just dispatches to the underlying representation.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    mode: int = 511,
) -> None: ...

Create a directory on the remote or local machine.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    show_progressbar: bool = False,
) -> None: ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Where to copy to.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    verbosity: int = 1,
) -> None: ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    allow_failure: bool,
    verbosity: int = 1,
) -> None: ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None

LocalSlurmExecutor

class LocalSlurmExecutor(
    salvus.flow.executors.implementations.slurm.SlurmExecutor,
    salvus.flow.executors.implementations.local.LocalExecutor,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ): ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir types

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

os_type salvus.flow.executors.remote_io_executor.OSType

Return the type of operating system.

sftp_client salvus.flow.executors.remote_io_executor.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.executors.remote_io_executor.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None: ...

Cancels the job.

Parameters
  • job_dir str — The job directory.
Returns None
execute_command()
def execute_command(
    self,
    command: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]: ...

Execute the command.

Parameters
  • command str — The command to run as a string.
  • assert_ok bool — Raise if the command does not exist with a code of 0.
  • environment Optional[Dict[str, str]] — Environment variables to be set for this command.
Returns Tuple[int, List[str], List[str]]
get_job_status()
def get_job_status(self, job_dir: str) -> Union[
    salvus.flow.executors.types.JobStatus,
    List[salvus.flow.executors.types.JobStatus],
]: ...

Returns the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns Union[salvus.flow.executors.types.JobStatus, List[salvus.flow.executors.types.JobStatus]]
is_folder_writeable()
def is_folder_writeable(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Return True if the user has write permissions in the chosen folder.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path on the remote machine.
Returns bool
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.executors.types.JobStatus]: ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be given for slurm sites.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.executors.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.executors.executor_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.executors.types.JobStatus: ...

Run an MPI job.

Returns the initial status. Always “pending” for slurm jobs.

Parameters
  • commands List[salvus.flow.executors.executor_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds Optional[int] — Wall-time in seconds. Must be given for slurm sites.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
Returns salvus.flow.executors.types.JobStatus
pretty_print()
def pretty_print(self) -> None: ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Check if a remote file or directory exists.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    localpath: Union[_io.BytesIO, pathlib.Path],
    show_progressbar: bool = False,
) -> None: ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_listdir()
def remote_listdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> List[str]: ...

List the contents of a remote directory.

Just dispatches to the underlying representation.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    mode: int = 511,
) -> None: ...

Create a directory on the remote or local machine.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    show_progressbar: bool = False,
) -> None: ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Where to copy to.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    verbosity: int = 1,
) -> None: ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    allow_failure: bool,
    verbosity: int = 1,
) -> None: ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None