Mondaic
This API reference is not for the latest stable Salvus version.

salvus.flow.sites.concrete_sites.local_hpc

Local versions of all the HPC sites.

They still interact with the job queueing system. They do not use SSH to communicate but directly execute the commands.

Classes

LocalGridEngineSite

class LocalGridEngineSite(
    salvus.flow.sites.concrete_sites.grid_engine.GridEngineSite,
    salvus.flow.sites.concrete_sites.local.LocalSite,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        keyring_settings: Optional[Dict],
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ):
        ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • keyring_settings Optional[Dict] — Keyring settings, if any.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir pathlib.PurePosixPath

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

sftp_client salvus.flow.sites.remote_io_site.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.sites.remote_io_site.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None:
    ...

Cancel the GridEngine job by calling qdel.

Parameters
  • job_dir str — The job directory.
Returns None
check_if_folder_can_be_written_to()
def check_if_folder_can_be_written_to(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Checks if the folder can be written to.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Path on the remote machine.
Returns bool
get_job_status()
def get_job_status(
    self, job_dir: str
) -> Union[
    salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]
]:
    ...

Returns the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns Union[salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]]
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.sites.types.JobStatus]:
    ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be set for GridEngine sites.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.sites.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.sites.site_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.sites.types.JobStatus:
    ...

Launch a job on a GridEngine cluster using MPI.

Returns the initial status - always “pending” here.

Parameters
  • commands List[salvus.flow.sites.site_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds Optional[int] — Wall-time in seconds. Must be set for GridEngine sites.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
Returns salvus.flow.sites.types.JobStatus
pretty_print()
def pretty_print(self) -> None:
    ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Check if the remote path exists or not.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    localpath: Union[_io.BytesIO, pathlib.Path],
    progressbar: bool = False,
) -> None:
    ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • progressbar bool — Show progressbar.
Returns None
remote_listdir()
def remote_listdir(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> List[str]:
    ...

List the contents of a remote directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    mode: int = 511,
) -> None:
    ...

Remotely create a directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    progressbar: bool = False,
) -> None:
    ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Where to copy to.
  • progressbar bool — Show a progressbar.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    verbosity: int = 1,
) -> None:
    ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
run_ssh_command()
def run_ssh_command(
    self,
    cmd: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]:
    ...

Run the “ssh command” on this machine without using SSH.

Parameters
  • cmd str — Command to run.
  • assert_ok bool — Assert that it exits with code zero, otherwise raise a remote execution error with more information.
  • environment Optional[Dict[str, str]] — A dict of shell environment variables to be merged into the default environment that the remote command executes with.
Returns Tuple[int, List[str], List[str]]
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    allow_failure: bool,
    verbosity: int = 1,
) -> None:
    ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None

LocalLSFSite

class LocalLSFSite(
    salvus.flow.sites.concrete_sites.lsf.LSFSite,
    salvus.flow.sites.concrete_sites.local.LocalSite,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        keyring_settings: Optional[Dict],
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ):
        ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • keyring_settings Optional[Dict] — Keyring settings, if any.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir pathlib.PurePosixPath

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

sftp_client salvus.flow.sites.remote_io_site.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.sites.remote_io_site.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None:
    ...

Cancel the LSF job by calling bkill.

Parameters
  • job_dir str — Directory where job is running.
Returns None
check_if_folder_can_be_written_to()
def check_if_folder_can_be_written_to(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Checks if the folder can be written to.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Path on the remote machine.
Returns bool
get_job_status()
def get_job_status(
    self, job_dir: str
) -> Union[
    salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]
]:
    ...

Return the status of the job or job array.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — Directory where job is running.
Returns Union[salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]] — Either a single job status or a list of Job statuses if this is a job array.
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.sites.types.JobStatus]:
    ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be given for an LSF site.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.sites.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.sites.site_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.sites.types.JobStatus:
    ...

Launch a job on LSF cluster using MPI.

Parameters
  • commands List[salvus.flow.sites.site_utils.RemoteCommand] — List of commands to run.
  • job_dir str — Root directory of job, where all files will be copied.
  • working_dir str — Run directory of job, where Salvus will be executed.
  • stdout_path str — Path to the Salvus stdout file.
  • stderr_path str — Path to the Salvus stderr file.
  • wall_time_in_seconds Optional[int] — Max wall time in seconds for the job.
  • is_debug_job bool — Should we run this job in the debug queue, by default False
Returns salvus.flow.sites.types.JobStatus — Status of the job directly after submission.
pretty_print()
def pretty_print(self) -> None:
    ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Check if the remote path exists or not.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    localpath: Union[_io.BytesIO, pathlib.Path],
    progressbar: bool = False,
) -> None:
    ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • progressbar bool — Show progressbar.
Returns None
remote_listdir()
def remote_listdir(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> List[str]:
    ...

List the contents of a remote directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    mode: int = 511,
) -> None:
    ...

Remotely create a directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    progressbar: bool = False,
) -> None:
    ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Where to copy to.
  • progressbar bool — Show a progressbar.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    verbosity: int = 1,
) -> None:
    ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
run_ssh_command()
def run_ssh_command(
    self,
    cmd: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]:
    ...

Run the “ssh command” on this machine without using SSH.

Parameters
  • cmd str — Command to run.
  • assert_ok bool — Assert that it exits with code zero, otherwise raise a remote execution error with more information.
  • environment Optional[Dict[str, str]] — A dict of shell environment variables to be merged into the default environment that the remote command executes with.
Returns Tuple[int, List[str], List[str]]
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    allow_failure: bool,
    verbosity: int = 1,
) -> None:
    ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None

LocalPbsSite

class LocalPbsSite(
    salvus.flow.sites.concrete_sites.pbs.PbsSite,
    salvus.flow.sites.concrete_sites.local.LocalSite,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        keyring_settings: Optional[Dict],
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ):
        ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • keyring_settings Optional[Dict] — Keyring settings, if any.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir pathlib.PurePosixPath

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

sftp_client salvus.flow.sites.remote_io_site.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.sites.remote_io_site.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None:
    ...

Cancel the PBS job by calling qdel.

Parameters
  • job_dir str — The job directory.
Returns None
check_if_folder_can_be_written_to()
def check_if_folder_can_be_written_to(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Checks if the folder can be written to.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Path on the remote machine.
Returns bool
get_job_status()
def get_job_status(
    self, job_dir: str
) -> Union[
    salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]
]:
    ...

Returns the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns Union[salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]]
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.sites.types.JobStatus]:
    ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be set for PBS sites.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.sites.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.sites.site_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.sites.types.JobStatus:
    ...

Launch a job on a PBS cluster using MPI.

Returns the initial status - always “pending” here.

Parameters
  • commands List[salvus.flow.sites.site_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds Optional[int] — Wall-time in seconds. Must be set for PBS sites.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
Returns salvus.flow.sites.types.JobStatus
pretty_print()
def pretty_print(self) -> None:
    ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Check if the remote path exists or not.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    localpath: Union[_io.BytesIO, pathlib.Path],
    progressbar: bool = False,
) -> None:
    ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • progressbar bool — Show progressbar.
Returns None
remote_listdir()
def remote_listdir(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> List[str]:
    ...

List the contents of a remote directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    mode: int = 511,
) -> None:
    ...

Remotely create a directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    progressbar: bool = False,
) -> None:
    ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Where to copy to.
  • progressbar bool — Show a progressbar.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    verbosity: int = 1,
) -> None:
    ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
run_ssh_command()
def run_ssh_command(
    self,
    cmd: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]:
    ...

Run the “ssh command” on this machine without using SSH.

Parameters
  • cmd str — Command to run.
  • assert_ok bool — Assert that it exits with code zero, otherwise raise a remote execution error with more information.
  • environment Optional[Dict[str, str]] — A dict of shell environment variables to be merged into the default environment that the remote command executes with.
Returns Tuple[int, List[str], List[str]]
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    allow_failure: bool,
    verbosity: int = 1,
) -> None:
    ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None

LocalSlurmSite

class LocalSlurmSite(
    salvus.flow.sites.concrete_sites.slurm.SlurmSite,
    salvus.flow.sites.concrete_sites.local.LocalSite,
):
    def __init__(
        self,
        name: str,
        configuration: Dict,
        keyring_settings: Optional[Dict],
        db: Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ):
        ...
Parameters
  • name str — Site name.
  • configuration Dict — Configuration for the site.
  • keyring_settings Optional[Dict] — Keyring settings, if any.
  • db Any — Connection to the internal database.
  • verbosity int — Verbosity level.
  • skip_version_number_check bool — Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.
Attributes
file_path_type Union[Type[pathlib.PurePosixPath], Type[pathlib.PureWindowsPath]]

Returns the type of file paths for the given site.

home_dir pathlib.PurePosixPath

The home directory.

is_initialized bool

True if the site has been synchronized/initialized with the remote site.

sftp_client salvus.flow.sites.remote_io_site.LocalSFTPClient

Get the sftp client instance.

Will be initialized upon the first access.

ssh_client salvus.flow.sites.remote_io_site.LocalSSHClient

Get the ssh client instance.

Will be initialized upon the first access.

update_interval_in_seconds float

A site’s actual update interval.

use_cuda_capable_gpus bool

True if a site uses license tokens.

use_license_tokens float

True if a site uses license tokens.

Methods
cancel_job()
def cancel_job(self, job_dir: str) -> None:
    ...

Cancels the job.

Parameters
  • job_dir str — The job directory.
Returns None
check_if_folder_can_be_written_to()
def check_if_folder_can_be_written_to(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Checks if the folder can be written to.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Path on the remote machine.
Returns bool
get_job_status()
def get_job_status(
    self, job_dir: str
) -> Union[
    salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]
]:
    ...

Returns the status of the job or job array to the best knowledge of the remote site.

If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.

This is a single method as, in practice, it turns out to be more convenient to implement.

Parameters
  • job_dir str — The job directory.
Returns Union[salvus.flow.sites.types.JobStatus, List[salvus.flow.sites.types.JobStatus]]
launch_many_mpi_jobs()
def launch_many_mpi_jobs(
    self,
    job_commands: Dict[int, Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: Optional[int],
    license_tokens: Optional[List[str]] = None,
) -> List[salvus.flow.sites.types.JobStatus]:
    ...

Launch many MPI jobs at once.

Parameters
  • job_commands Dict[int, Any] — Dictionary of a list of commands, one per list per job.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds_per_job Optional[int] — Wall-time in seconds per job. Must be given for slurm sites.
  • license_tokens Optional[List[str]] — If the job array uses license tokens for the license enforcement they are passed here.
Returns List[salvus.flow.sites.types.JobStatus]
launch_mpi_job()
def launch_mpi_job(
    self,
    commands: List[salvus.flow.sites.site_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: Optional[int],
    is_debug_job: bool = False,
) -> salvus.flow.sites.types.JobStatus:
    ...

Run an MPI job.

Returns the initial status. Always “pending” for slurm jobs.

Parameters
  • commands List[salvus.flow.sites.site_utils.RemoteCommand] — List of commands. One will be run after the other, but all will be submitted at the same time.
  • job_dir str — The remote directory where all the job files are stored.
  • working_dir str — The remote working directory for the execution.
  • stdout_path str — Path to pipe stdout to.
  • stderr_path str — Path to pipe stderr to.
  • wall_time_in_seconds Optional[int] — Wall-time in seconds. Must be given for slurm sites.
  • is_debug_job bool — True if this is supposed to be a debug job. Some job management systems have debug or other “fast” queues. Use these special queues if this is set to true.
Returns salvus.flow.sites.types.JobStatus
pretty_print()
def pretty_print(self) -> None:
    ...

Pretty and colorful representation of the site.

Returns None
remote_exists()
def remote_exists(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> bool:
    ...

Check if the remote path exists or not.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to check.
Returns bool
remote_get()
def remote_get(
    self,
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    localpath: Union[_io.BytesIO, pathlib.Path],
    progressbar: bool = False,
) -> None:
    ...

Get a remote file.

Parameters
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote file to get.
  • localpath Union[_io.BytesIO, pathlib.Path] — Local path to save to Can be a file or a file like object.
  • progressbar bool — Show progressbar.
Returns None
remote_listdir()
def remote_listdir(
    self, path: Union[str, pathlib.Path, pathlib.PurePosixPath]
) -> List[str]:
    ...

List the contents of a remote directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path whose contents to list.
Returns List[str]
remote_mkdir()
def remote_mkdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    mode: int = 511,
) -> None:
    ...

Remotely create a directory.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Directory to create.
  • mode int — Permissions for the new directory.
Returns None
remote_put()
def remote_put(
    self,
    localpath: Union[_io.BytesIO, pathlib.Path],
    remotepath: Union[str, pathlib.Path, pathlib.PurePosixPath],
    progressbar: bool = False,
) -> None:
    ...

Copy a local file to a remote path.

Parameters
  • localpath Union[_io.BytesIO, pathlib.Path] — Local file to copy. Can be a file like object.
  • remotepath Union[str, pathlib.Path, pathlib.PurePosixPath] — Where to copy to.
  • progressbar bool — Show a progressbar.
Returns None
remote_rmdir()
def remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    verbosity: int = 1,
) -> None:
    ...

Remotely and recursively delete a remote folder and its contents.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • verbosity int — Verbosity level.
Returns None
run_ssh_command()
def run_ssh_command(
    self,
    cmd: str,
    assert_ok: bool = True,
    environment: Optional[Dict[str, str]] = None,
) -> Tuple[int, List[str], List[str]]:
    ...

Run the “ssh command” on this machine without using SSH.

Parameters
  • cmd str — Command to run.
  • assert_ok bool — Assert that it exits with code zero, otherwise raise a remote execution error with more information.
  • environment Optional[Dict[str, str]] — A dict of shell environment variables to be merged into the default environment that the remote command executes with.
Returns Tuple[int, List[str], List[str]]
safe_remote_rmdir()
def safe_remote_rmdir(
    self,
    path: Union[str, pathlib.Path, pathlib.PurePosixPath],
    allow_failure: bool,
    verbosity: int = 1,
) -> None:
    ...

Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.

Parameters
  • path Union[str, pathlib.Path, pathlib.PurePosixPath] — Remote path to delete.
  • allow_failure bool — Whether to allow failure or not.
  • verbosity int — Verbosity level.
Returns None