Mondaic
This API reference is not for the latest stable Salvus version.

salvus.flow.executors.salvus_job_array

Classes

SalvusJobArray

class SalvusJobArray(builtins.object):
    def __init__(
        self,
        site: salvus.flow.executors.base_executor.BaseExecutor,
        input_files: Iterable[
            Union[
                str,
                pathlib.Path,
                Dict,
                salvus.flow.utils.deep_setters._DeepSetter,
            ]
        ],
        ranks_per_job: Optional[int, numpy.int32, numpy.int64] = None,
        wall_time_in_seconds_per_job: Optional[
            int, numpy.int32, numpy.int64, float, numpy.float32, numpy.float64
        ] = None,
        job_groups: Optional[List[Union[str, salvus.flow.db.JobGroup]]] = None,
        job_array_name: Optional[str] = None,
        verbosity: int = 1,
        initialize_on_site: bool = True,
    ): ...

A Salvus job array collecting many similar runs.

Parameters
  • site salvus.flow.executors.base_executor.BaseExecutor — The site this job is to be run at.
  • input_files Iterable[Union[str, pathlib.Path, Dict, salvus.flow.utils.deep_setters._DeepSetter]] — Salvus input files. One for each job.
  • ranks_per_job Optional[int, numpy.int32, numpy.int64] — Number of ranks to use for each job.
  • wall_time_in_seconds_per_job Optional[int, numpy.int32, numpy.int64, float, numpy.float32, numpy.float64] — Wall time in seconds for each job. Not required for jobs on all sites.
  • job_groups Optional[List[Union[str, salvus.flow.db.JobGroup]]] — List of job groups names this job should be part of.
  • job_array_name Optional[str] — Name of the job array. A random one will be created if none is given.
  • verbosity int — Verbosity level.
  • initialize_on_site bool — Initialize on site. Should be true in most cases. False is useful for creating a Job object from the database.
Attributes
db salvus.flow.db.DB

THe job’s currently used database.

ranks_per_job int

THe number of ranks per job.

stderr str

Return the stderr of the job array as a string.

stderr_path Union[pathlib.PurePosixPath, pathlib.PureWindowsPath]

Stderr path of the job.

stdout str

Return the stdout of the job array as a string.

stdout_path Union[pathlib.PurePosixPath, pathlib.PureWindowsPath]

Stdout path of the job.

Methods
cancel()
def cancel(self, verbosity: int = 1) -> None: ...

Cancel job array.

Parameters
  • verbosity int — Verbosity level.
Returns None
copy_output()
def copy_output(
    self,
    destination: Union[str, pathlib.Path],
    get_all: bool = False,
    verbosity: int = 1,
    allow_existing_destination_folder: bool = False,
    copy_partial_results: bool = True,
) -> List[Tuple[Dict[pathlib.Path, int], bool]]: ...

Copy the output files of the job array to a local folder.

Will create a separate folder named job_0000, job_0001, and so on, one for each job in the array.

Parameters
  • destination Union[str, pathlib.Path] — The destination folder. Must not yet exist.
  • get_all bool — Also get the large output files, if any.
  • verbosity int — Verbosity level.
  • allow_existing_destination_folder bool — Allow that the destination already exists. Files might be overwritten.
  • copy_partial_results bool — If False and some jobs in the array are not finished, it will raise. Otherwise (default) it will copy the finished jobs and not touch the others.
Returns List[Tuple[Dict[pathlib.Path, int], bool]]
delete()
def delete(
    self,
    verbosity: int = 1,
    allow_missing_jobs: bool = False,
    allow_directory_deletion_failure: bool = False,
) -> None: ...

Delete all data of a job array.

Will delete all remote files and remove all jobs and the job array from the internal database.

Parameters
  • verbosity int — Verbosity level.
  • allow_missing_jobs bool — If True, it will not raise if less jobs are available on disc and in the DB than indicated by the job array. This can happen if individual sub jobs have been deleted manually.
  • allow_directory_deletion_failure bool — If True this function will still return successfully (but raise a warning) if the deletion of the remote Salvus managed files failed for some reason. This can easily happen on for example networked file systems.
Returns None
get_diagnosis_string()
def get_diagnosis_string(self) -> str: ...

Get a diagnostic string with stdout/stderr from the first job in the array.

Useful for trying to figure out why a job array fails.

Returns str
get_status_from_db()
def get_status_from_db(self) -> salvus.flow.executors.types.JobStatus: ...

Returns the current status of the job array from the database. Please note that this might not be up-to-date.

Returns salvus.flow.executors.types.JobStatus
launch()
def launch(self, verbosity: int = 1) -> None: ...

Launch the job array.

Parameters
  • verbosity int — The verbosity level.
Returns None
set_job_status_in_db()
def set_job_status_in_db(
    self, status: List[salvus.flow.executors.types.JobStatus]
) -> None: ...

Sets the job array status in the database.

Parameters
  • status List[salvus.flow.executors.types.JobStatus] — One status for each job.
Returns None
update_status()
def update_status(
    self,
    progress_callback: Optional[Callable[[Dict], NoneType]] = None,
    force_update: bool = False,
) -> List[salvus.flow.executors.types.JobStatus]: ...

Update the status of all jobs in the job array.

Parameters
  • progress_callback Optional[Callable[[Dict], NoneType]] — Callback function that is called whenever something happens.
  • force_update bool — Job arrays that are not running and not pending will by default only query the local job database and not check the remote site. In most cases this is correct as the jobs are done with and will not change again. Setting this to True will force an update nonetheless. Uses for this are for example if a job update fails for example due to a temporary network problem.
Returns List[salvus.flow.executors.types.JobStatus]
wait()
def wait(
    self,
    poll_interval_in_seconds: Optional[
        int, numpy.int32, numpy.int64, float, numpy.float32, numpy.float64
    ] = None,
    timeout_in_seconds: Optional[float] = None,
    verbosity: int = 1,
    progress_callback: Optional[Callable[[Dict], NoneType]] = None,
) -> List[salvus.flow.executors.types.JobStatus]: ...

Wait for the job to finish.

Parameters
  • poll_interval_in_seconds Optional[int, numpy.int32, numpy.int64, float, numpy.float32, numpy.float64] — If given, the site will be queried with this interval. Defaults to the site’s default.
  • timeout_in_seconds Optional[float] — Time in seconds after which this function will return, independent of the status.
  • verbosity int — Verbosity level.
  • progress_callback Optional[Callable[[Dict], NoneType]] — Optional progress callback function. Will be called if there are any status updates.
Returns List[salvus.flow.executors.types.JobStatus] — The status for each job.