Mondaic Documentation
www.mondaic.com
Mondaic
Version:
No results found.
python_api
/
salvus
/
flow
/
executors
/
implementations
/
grid_engine

salvus.flow.executors.implementations.grid_engine

salvus.flow.executors.implementations.grid_engine salvus flow executors implementations grid_engine
The GridEngine executor.

Classes

GridEngineExecutor

Base class for a an executor.
An executor is a set of instructions to run a Salvus job or other commands on a type of remote machine.
SIGNATURE
class GridEngineExecutor(salvus.flow.executors.base_executor.BaseExecutor):
    def __init__(
        self,
        name: str,
        configuration: dict,
        db: typing.Any,
        verbosity: int = 1,
        skip_version_number_check: bool = False,
    ): ...
ARGUMENTS
Required
name
Type:str
Description:
Name of the site using the executor.
Required
configuration
Type:dict
Description:
Configuration for the site.
Required
db
Type:typing.Any
Description:
Connection to the internal database.
Optional
verbosity
Type:int
Default value:1
Description:
Verbosity level.
Optional
skip_version_number_check
Type:bool
Default value:False
Description:
Skip the version number check of the local Python version vs the remote site. Useful for initializing and updating sites.

Properties

  • file_path_type(type[pathlib.PurePosixPath | pathlib.PureWindowsPath])-Returns the type of file paths for the given site.
  • home_dir(pathlib.PurePosixPath | pathlib.PureWindowsPath)-The home directory.
  • is_initialized(bool)-True if the site has been synchronized/initialized with the remote site.
  • os_type(OSType)-Returns the type of the operating system. Subclasses can overwrite this.
  • sftp_client(paramiko.sftp_client.SFTPClient | LocalSFTPClient)-Get the sftp client instance. Will be initialized upon the first access.
  • ssh_client(paramiko.client.SSHClient | LocalSSHClient)-Get the ssh client instance. Will be initialized upon the first access.
  • update_interval_in_seconds(float)-A site's actual update interval.
  • use_cuda_capable_gpus(bool)-True if a site uses license tokens.
  • use_license_tokens(float)-True if a site uses license tokens.

Methods

GridEngineExecutor.cancel_job()
Cancel the GridEngine job by calling qdel.
SIGNATURE
def cancel_job(self, job_dir: str) -> None: ...
ARGUMENTS
Required
job_dir
Type:str
Description:
The job directory.
GridEngineExecutor.execute_command()
Execute the command.
SIGNATURE
def execute_command(
    self,
    command: str,
    assert_ok: bool = True,
    environment: dict[str, str] | None = None,
) -> tuple[int, list[str], list[str]]: ...
ARGUMENTS
Required
command
Type:str
Description:
The command to run as a string.
Optional
assert_ok
Type:bool
Default value:True
Description:
Raise if the command does not exist with a code of 0.
Optional
environment
Type:dict[str, str] | None
Default value:None
Description:
Environment variables to be set for this command.
GridEngineExecutor.get_job_status()
Returns the status of the job or job array to the best knowledge of the remote site.
If the job is a job array, return the status of every job in a list if possible, otherwise just the total status of the whole job.
This is a single method as, in practice, it turns out to be more convenient to implement.
SIGNATURE
def get_job_status(self, job_dir: str) -> JobStatus | list[JobStatus]: ...
ARGUMENTS
Required
job_dir
Type:str
Description:
The job directory.
GridEngineExecutor.is_folder_writeable()
Return True if the user has write permissions in the chosen folder.
SIGNATURE
def is_folder_writeable(self, path: _RemotePathInputType) -> bool: ...
ARGUMENTS
Required
path
Type:_RemotePathInputType
Description:
Path on the remote machine.
GridEngineExecutor.launch_many_mpi_jobs()
Launch many MPI jobs at once.
SIGNATURE
def launch_many_mpi_jobs(
    self,
    job_commands: dict[int, typing.Any],
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds_per_job: int | None,
    license_tokens: list[str] | None = None,
) -> list[JobStatus]: ...
ARGUMENTS
Required
job_commands
Type:dict[int, typing.Any]
Description:
Dictionary of a list of commands, one per list per job.
Required
working_dir
Type:str
Description:
The remote working directory for the execution.
Required
stdout_path
Type:str
Description:
Path to pipe stdout to.
Required
stderr_path
Type:str
Description:
Path to pipe stderr to.
Required
wall_time_in_seconds_per_job
Type:int | None
Description:
Wall-time in seconds per job. Must be set for GridEngine sites.
Optional
license_tokens
Type:list[str] | None
Default value:None
Description:
If the job array uses license tokens for the license enforcement they are passed here.
GridEngineExecutor.launch_mpi_job()
Launch a job on a GridEngine cluster using MPI.
Returns the initial status - always "pending" here.
SIGNATURE
def launch_mpi_job(
    self,
    commands: list[executor_utils.RemoteCommand],
    job_dir: str,
    working_dir: str,
    stdout_path: str,
    stderr_path: str,
    wall_time_in_seconds: int | None,
    is_debug_job: bool = False,
) -> JobStatus: ...
ARGUMENTS
Required
commands
Type:list[executor_utils.RemoteCommand]
Description:
List of commands. One will be run after the other, but all will be submitted at the same time.
Required
job_dir
Type:str
Description:
The remote directory where all the job files are stored.
Required
working_dir
Type:str
Description:
The remote working directory for the execution.
Required
stdout_path
Type:str
Description:
Path to pipe stdout to.
Required
stderr_path
Type:str
Description:
Path to pipe stderr to.
Required
wall_time_in_seconds
Type:int | None
Description:
Wall-time in seconds. Must be set for GridEngine sites.
Optional
is_debug_job
Type:bool
Default value:False
Description:
True if this is supposed to be a debug job. Some job management systems have debug or other "fast" queues. Use these special queues if this is set to true.
GridEngineExecutor.pretty_print()
Pretty and colorful representation of the site.
SIGNATURE
def pretty_print(self) -> None: ...
GridEngineExecutor.remote_exists()
Check if a remote file or directory exists.
SIGNATURE
def remote_exists(self, path: _RemotePathInputType) -> bool: ...
ARGUMENTS
Required
path
Type:_RemotePathInputType
Description:
Path to check.
GridEngineExecutor.remote_get()
Get a remote file.
SIGNATURE
def remote_get(
    self,
    remotepath: _RemotePathInputType,
    localpath: io.BytesIO | pathlib.Path,
    show_progressbar: bool = False,
) -> None: ...
ARGUMENTS
Required
remotepath
Type:_RemotePathInputType
Description:
Remote file to get.
Required
localpath
Type:io.BytesIO | pathlib.Path
Description:
Local path to save to Can be a file or a file like object.
Optional
show_progressbar
Type:bool
Default value:False
Description:
Show a progressbar or not.
GridEngineExecutor.remote_listdir()
List the contents of a remote directory.
Just dispatches to the underlying representation.
SIGNATURE
def remote_listdir(self, path: _RemotePathInputType) -> list[str]: ...
ARGUMENTS
Required
path
Type:_RemotePathInputType
Description:
Remote path whose contents to list.
GridEngineExecutor.remote_mkdir()
Create a directory on the remote or local machine.
SIGNATURE
def remote_mkdir(
    self, path: _RemotePathInputType, mode: int = 511
) -> None: ...
ARGUMENTS
Required
path
Type:_RemotePathInputType
Description:
Directory to create.
Optional
mode
Type:int
Default value:511
Description:
Permissions for the new directory.
GridEngineExecutor.remote_put()
Copy a local file to a remote path.
SIGNATURE
def remote_put(
    self,
    localpath: io.BytesIO | pathlib.Path,
    remotepath: _RemotePathInputType,
    show_progressbar: bool = False,
) -> None: ...
ARGUMENTS
Required
localpath
Type:io.BytesIO | pathlib.Path
Description:
Local file to copy. Can be a file like object.
Required
remotepath
Type:_RemotePathInputType
Description:
Where to copy to.
Optional
show_progressbar
Type:bool
Default value:False
Description:
Show a progressbar or not.
GridEngineExecutor.remote_rmdir()
Remotely and recursively delete a remote folder and its contents.
SIGNATURE
def remote_rmdir(
    self, path: _RemotePathInputType, verbosity: int = 1
) -> None: ...
ARGUMENTS
Required
path
Type:_RemotePathInputType
Description:
Remote path to delete.
Optional
verbosity
Type:int
Default value:1
Description:
Verbosity level.
GridEngineExecutor.safe_remote_rmdir()
Convenience wrapper that optionally allows the remote_rmdir() command to fail. Useful for some networked file systems.
SIGNATURE
def safe_remote_rmdir(
    self, path: _RemotePathInputType, allow_failure: bool, verbosity: int = 1
) -> None: ...
ARGUMENTS
Required
path
Type:_RemotePathInputType
Description:
Remote path to delete.
Required
allow_failure
Type:bool
Description:
Whether to allow failure or not.
Optional
verbosity
Type:int
Default value:1
Description:
Verbosity level.
Previous
implementations
Next
local
We use cookies to analyze our traffic, but won't track you until explicit consent is given. Using our contact form will create relevant cookies and transmit data to third party services to keep track of your request. Read more about how we use personal information in our Privacy Policy.
PAGE CONTENTS