Mondaic
This API reference is not for the latest stable Salvus version.

salvus.flow.executors.implementations.ssh

Classes

SSHExecutor

class SSHExecutor(salvus.flow.executors.base_executor.BaseExecutor):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ): ...

Site that will directly execute a job over SSH.

It attempts to be UNIX and Windows compatible.

There are a few assumptions about MPI.

On Unix it must be installed relative to the Salvus binary:

/bin
    mpirun
    salvus
/lib
    mpi_libs

On Windows MS-MPI must be installed and available via mpiexec.exe.

Parameters
  • name str — Name of the site using the executor.
  • configuration Dict — Configuration for the site.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir types

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

os_type salvus.flow.executors.remote_io_executor.OSType

Returns the type of the operating system.

Subclasses can overwrite this.

sftp_client Union[paramiko.sftp_client.SFTPClient, salvus.flow.executors.remote_io_executor.LocalSFTPClient]

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client Union[paramiko.client.SSHClient, salvus.flow.executors.remote_io_executor.LocalSSHClient]

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None: ...

Cancel the job.

Parameters
  • job_dir str — The job directory.
Returns None
execute_command()
def execute_command(
    self,
    command: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]: ...

Execute the command.

Parameters
  • command str — The command to run as a string.
  • assert_ok bool — Raise if the command does not exist with a code of 0.
  • environment Optional[Dict[str, str]] — Environment variables to be set for this command.
Returns Tuple[int, List[str], List[str]]
get_job_status()
def get_job_status(self, job_dir: str) -> JobStatus: ...

Return the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns JobStatus
is_folder_writeable()
def is_folder_writeable(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Return True if the user has write permissions in the chosen folder.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path on the remote machine.
Returns bool
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: typing.Dict[int, typing.Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: typing.Optional[int],
    license_tokens: typing.Optional[typing.List[str]] = None,
) -> typing.List[JobStatus]: ...

Launch many MPI jobs at once.

Parameters
  • job_commands typing.Dict[int, typing.Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job typing.Optional[int] — Wall-time in seconds per job. Not necessary for the SSH site.
  • license_tokens typing.Optional[typing.List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns typing.List[JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: typing.List[executor_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: typing.Optional[int],
    is_debug_job: bool = False,
    interactive_tty: bool = False,
) -> JobStatus: ...

Run an MPI job.

Must return the initial status. Should either be “running” or “pending”, depending on the system.

Parameters
  • commands typing.List[executor_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds typing.Optional[int] — Wall-time in seconds. Might be None, depending on the site.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
  • interactive_tty bool — Launch the job in an interactive terminal. This only works for local sites, which are based on this SSH site.
Returns JobStatus
pretty_print()
def pretty_print(self) -> None: ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> bool: ...

Check if a remote file or directory exists.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    localpath: Union[_io.BytesIO, pathlib.Path],
    show_progressbar: bool = False,
) -> None: ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_listdir()
def remote_listdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
) -> List[str]: ...

List the contents of a remote directory.

Just dispatches to the underlying representation.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    mode: int = 511,
) -> None: ...

Create a directory on the remote or local machine.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    show_progressbar: bool = False,
) -> None: ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Where to copy to.
  • show_progressbar bool — Show a progressbar or not.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    verbosity: int = 1,
) -> None: ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[
        str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath
    ],
    allow_failure: bool,
    verbosity: int = 1,
) -> None: ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath, pathlib.PureWindowsPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None