Mondaic - Full Waveform Solutions

Copy

%matplotlib inline

Copy

import numpy as np
import os
import salvus.namespace as sn

Copy

SALVUS_FLOW_SITE_NAME = os.environ.get("SITE_NAME", "local")

Copy

p = sn.Project(path="project")

The inverse problem is a similar entity like all the other configuration objects. We want to be able to keep track of all the individual steps, and avoid unnecessary repetitions of the same task. Fortunately, SalvusProject takes care of data management and book keeping.

We need to specify the prior, which is just the SimulationConfiguration object of the initial model we created above. Furthermore, we need to specify all possible events that we might consider during the inversion. This could be a subset of events defined in the project, and we could add more events later on. Together with the events, we need to pass the observed data. Because we created it synthetically, this is also just a SimulationConfiguration object. The remaining parameters specify which parameters to invert for (VP and RHO), what misfit functional to use, preconditioner and descent method, and where to run the simulations.

Copy

p.add_to_project(
    sn.InverseProblemConfiguration(
        name="my_inversion",
        prior_model="initial_model",
        events=p.events.list(),
        mapping=sn.Mapping(
            scaling="absolute", inversion_parameters=["VP", "RHO"]
        ),
        preconditioner=sn.ConstantSmoothing({"VP": 0.01, "RHO": 0.01}),
        method=sn.TrustRegion(initial_trust_region_linf=10.0),
        misfit_configuration="L2",
        wavefield_compression=sn.WavefieldCompression(
            forward_wavefield_sampling_interval=10
        ),
        job_submission=sn.SiteConfig(
            site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=4
        ),
    )
)

Instead of monolithic inversions and a linear flow of iteration, SalvusOpt uses a tree-based framework of iteration. At any point during the inversion, you can branch off, modify the settings, and run one or more streams simultaneously.

Ready for our first iteration? Without specifying any additional information, all parameters will be inherited from the InverseProblemConfiguration.

Copy

p.inversions.add_iteration(inverse_problem_configuration="my_inversion")

[2025-07-23 21:54:03,662] INFO: Adding new iteration #0.

True

Many steps within a single iteration involve expensive simulations, e.g., for computing misfits or adjoint simulations to compute gradients. In order to be able to closely monitor the progress, SalvusOpt steps through an iteration, and automatically dispatches simulations whenever necessary. The function resume will return whenever SalvusOpt is waiting for other tasks to finish first. Calling it several time, will step through the iteration in sequence.

Copy

p.inversions.resume(
    inverse_problem_configuration="my_inversion",
)

[2025-07-23 21:54:03,680] INFO: Resuming iteration #0.
Current stage: initialize
[2025-07-23 21:54:03,682] INFO: 1 new tasks have been issued.
[2025-07-23 21:54:03,684] INFO: Processing task `misfit_and_gradient`
[2025-07-23 21:54:03,685] INFO: -> Mapping between inversion and simulation space: 0 ms
[2025-07-23 21:54:03,829] INFO: While looking for checkpoints, the following error occured: Remote input file `/builds/Mondaic/Code/integrations/TutorialsAndIntegrationTests/SALVUS_INSTALL/SalvusFlow/tmp/job_array_2507232153022909_de307a2039/job_0_of_job_array_2507232153022909_de307a2039/output.h5` does not exist.. The simulations will be resubmitted.
[2025-07-23 21:54:03,830] INFO: While looking for checkpoints, the following error occured: Remote input file `/builds/Mondaic/Code/integrations/TutorialsAndIntegrationTests/SALVUS_INSTALL/SalvusFlow/tmp/job_array_2507232153022909_de307a2039/job_1_of_job_array_2507232153022909_de307a2039/output.h5` does not exist.. The simulations will be resubmitted.
[2025-07-23 21:54:03,830] INFO: While looking for checkpoints, the following error occured: Remote input file `/builds/Mondaic/Code/integrations/TutorialsAndIntegrationTests/SALVUS_INSTALL/SalvusFlow/tmp/job_array_2507232153022909_de307a2039/job_2_of_job_array_2507232153022909_de307a2039/output.h5` does not exist.. The simulations will be resubmitted.
[2025-07-23 21:54:03,831] INFO: While looking for checkpoints, the following error occured: Remote input file `/builds/Mondaic/Code/integrations/TutorialsAndIntegrationTests/SALVUS_INSTALL/SalvusFlow/tmp/job_array_2507232153022909_de307a2039/job_3_of_job_array_2507232153022909_de307a2039/output.h5` does not exist.. The simulations will be resubmitted.
[2025-07-23 21:54:03,832] INFO: While looking for checkpoints, the following error occured: Remote input file `/builds/Mondaic/Code/integrations/TutorialsAndIntegrationTests/SALVUS_INSTALL/SalvusFlow/tmp/job_array_2507232153022909_de307a2039/job_4_of_job_array_2507232153022909_de307a2039/output.h5` does not exist.. The simulations will be resubmitted.
[2025-07-23 21:54:03,832] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0000', 'event_0001', 'event_0002', 'event_0003', 'event_0004']
[2025-07-23 21:54:03,855] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:54:04,021] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:54:04,022] INFO: Some tasks of iteration #0 are still running. Please check again later.

True

Copy

p.inversions.resume(
    inverse_problem_configuration="my_inversion",
)

[2025-07-23 21:54:04,026] INFO: Resuming iteration #0.
Current stage: check_convergence
[2025-07-23 21:54:04,027] INFO: Processing task `misfit_and_gradient`
[2025-07-23 21:54:04,027] INFO:     -> Mapping between inversion and simulation space: 0 ms
[2025-07-23 21:54:04,259] INFO: Some tasks of iteration #0 are still running. Please check again later.

False

Copy

p.viz.nb.iteration(
    inverse_problem_configuration="my_inversion", iteration_id=0
)

If you feel confident and don't want to be bothered with every single task, you can also tell SalvusOpt to run an entire iteration at once. Note the parameter timeout_in_seconds, which will force the cell to return even if the iteration has not been completed yet, and there might still be a few simulations running in the back.

Again, you can execute the cell several times or mix it with calls to the previous one until the iteration is complete.

Copy

p.inversions.iterate(
    inverse_problem_configuration="my_inversion",
    timeout_in_seconds=360,
    ping_interval_in_seconds=10,
)

p.viz.nb.inversion(inverse_problem_configuration="my_inversion")

[2025-07-23 21:54:04,433] INFO: Resuming iteration #0.

[2025-07-23 21:54:04,433] INFO: Processing task `misfit_and_gradient`
[2025-07-23 21:54:04,536] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-07-23 21:54:14,538] INFO: Processing task `misfit_and_gradient`
[2025-07-23 21:54:14,991] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:54:15,036] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:54:15,037] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-07-23 21:54:25,038] INFO: Processing task `misfit_and_gradient`
[2025-07-23 21:54:25,649] INFO: 
Iteration 0: Number of events: 5	 chi = 0.01638561880327696	 ||g|| = 0.020187598866632676
pred = ---	ared = ---	norm_update = ---	tr_radius = ---
[2025-07-23 21:54:25,650] INFO: 1 new tasks have been issued.
[2025-07-23 21:54:25,650] INFO: Processing task `preconditioner`

[2025-07-23 21:54:25,711] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-07-23 21:54:35,721] INFO: Processing task `preconditioner`
[2025-07-23 21:54:35,835] INFO: 1 new tasks have been issued.
[2025-07-23 21:54:35,836] INFO: Processing task `misfit`
[2025-07-23 21:54:36,024] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:54:36,109] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:54:36,110] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-07-23 21:54:46,126] INFO: Processing task `misfit`
[2025-07-23 21:54:46,784] INFO: 
old misfit control group: 0.016385618803276956
new misfit control group: 0.007147293329141072
predicted reduction control group: -0.007847548803511017
actual reduction control group: -0.009238325474135885
5 out of 5 event(s) improved the misfit.
[2025-07-23 21:54:46,785] INFO: 
Model update accepted.
[2025-07-23 21:54:46,785] INFO: 1 new tasks have been issued.
[2025-07-23 21:54:46,785] INFO: Processing task `finalize_iteration`
[2025-07-23 21:54:46,816] INFO: Successfully completed iteration #0.
[2025-07-23 21:54:46,817] INFO: Adding new iteration #1.

There is still quite a strong imprint of the source locations, but we start moving into the right direction. The high velocity inclusion in the top part also shows up in the density update.

The first update gave us confidence in the setup. For the moment, our work here is done. Let's run a few more iterations, lean back and wait for the results.

We just loop over the iterate function and perform two more model updates. Intermediate data can pile up quickly during an inversion. Each iteration computes waveforms and gradients for a list of events and creates additional auxiliary files for preconditioning and approximating the curvature. When running low on disk-space it is thus advisable to remove those files using the argument delete_disposable_files. It will automatically clear data from waveforms, gradients and auxiliary files for completed iterations, which are no longer needed to continue with the inversion.

This operation is safe in the sense that those data can always be recreated with the provenance information about the model and the simulation. Note, however, that recomputing would involve additional simulations.

Copy

for i in range(2):
    p.inversions.iterate(
        inverse_problem_configuration="my_inversion",
        timeout_in_seconds=360,
        ping_interval_in_seconds=10,
        delete_disposable_files="all",
    )

[2025-07-23 21:54:47,606] INFO: Resuming iteration #1.

[2025-07-23 21:54:47,607] INFO: 1 new tasks have been issued.
[2025-07-23 21:54:47,607] INFO: Processing task `gradient`
[2025-07-23 21:54:48,007] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:54:48,052] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:54:48,053] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-07-23 21:54:58,054] INFO: Processing task `gradient`
[2025-07-23 21:54:58,561] INFO: 
Iteration 1: Number of events: 5	 chi = 0.007147293329141072	 ||g|| = 0.009673226858076788
pred = -0.007847548803511017	ared = -0.009238325474135885	norm_update = 0.8696894857186028	tr_radius = 0.8696894179694716
[2025-07-23 21:54:58,570] INFO: 1 new tasks have been issued.
[2025-07-23 21:54:58,571] INFO: Processing task `preconditioner`

[2025-07-23 21:54:58,642] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-07-23 21:55:08,661] INFO: Processing task `preconditioner`
[2025-07-23 21:55:08,779] INFO: 1 new tasks have been issued.
[2025-07-23 21:55:08,780] INFO: Processing task `misfit`
[2025-07-23 21:55:08,819] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:55:08,902] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:55:08,903] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-07-23 21:55:18,929] INFO: Processing task `misfit`
[2025-07-23 21:55:19,585] INFO: 
old misfit control group: 0.007147293329141072
new misfit control group: 0.005363319098489491
predicted reduction control group: -0.0009135108170994499
actual reduction control group: -0.001783974230651581
5 out of 5 event(s) improved the misfit.
[2025-07-23 21:55:19,585] INFO: 
Model update accepted.
[2025-07-23 21:55:19,586] INFO: 1 new tasks have been issued.
[2025-07-23 21:55:19,586] INFO: Processing task `finalize_iteration`
[2025-07-23 21:55:19,634] INFO: ... searching for obsolete files in project/INVERSIONS/my_inversion/00001
[2025-07-23 21:55:19,740] INFO: Freed up 3.7 MB of space.
[2025-07-23 21:55:19,740] INFO: Successfully completed iteration #1.
[2025-07-23 21:55:19,742] INFO: Adding new iteration #2.
[2025-07-23 21:55:19,747] INFO: Resuming iteration #2.

[2025-07-23 21:55:19,747] INFO: 1 new tasks have been issued.
[2025-07-23 21:55:19,747] INFO: Processing task `gradient`
[2025-07-23 21:55:19,924] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:55:19,969] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:55:19,970] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-07-23 21:55:29,972] INFO: Processing task `gradient`
[2025-07-23 21:55:30,646] INFO: 
Iteration 2: Number of events: 5	 chi = 0.005363319098489491	 ||g|| = 0.006710450904451374
pred = -0.0009135108170994499	ared = -0.001783974230651581	norm_update = 0.23593351121777997	tr_radius = 1.7393788359389433
[2025-07-23 21:55:30,661] INFO: 1 new tasks have been issued.
[2025-07-23 21:55:30,661] INFO: Processing task `preconditioner`

[2025-07-23 21:55:30,737] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-07-23 21:55:40,763] INFO: Processing task `preconditioner`
[2025-07-23 21:55:40,897] INFO: 1 new tasks have been issued.
[2025-07-23 21:55:40,898] INFO: Processing task `misfit`
[2025-07-23 21:55:40,935] INFO: Submitting job array with 5 jobs ...

[2025-07-23 21:55:41,018] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-07-23 21:55:41,019] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-07-23 21:55:51,052] INFO: Processing task `misfit`
[2025-07-23 21:55:51,709] INFO: 
old misfit control group: 0.005363319098489491
new misfit control group: 0.003537546107004201
predicted reduction control group: -0.001225382196935243
actual reduction control group: -0.00182577299148529
5 out of 5 event(s) improved the misfit.
[2025-07-23 21:55:51,710] INFO: 
Model update accepted.
[2025-07-23 21:55:51,710] INFO: 1 new tasks have been issued.
[2025-07-23 21:55:51,710] INFO: Processing task `finalize_iteration`
[2025-07-23 21:55:51,773] INFO: ... searching for obsolete files in project/INVERSIONS/my_inversion/00002
[2025-07-23 21:55:51,882] INFO: Freed up 4.3 MB of space.
[2025-07-23 21:55:51,882] INFO: Successfully completed iteration #2.
[2025-07-23 21:55:51,884] INFO: Adding new iteration #3.

Done!

Did we converge towards the ground truth?

Copy

p.viz.nb.inversion(inverse_problem_configuration="my_inversion")

Well, at least we are getting closer and the heterogeneities start to form. You can, of course, continue with more iterations or play around with the settings.

That's the end of this tutorial. We ran a few iterations of full-waveform inversion in a typical aperture for ultrasound screenings. Note that although the problem size is small and we did not apply any sophisticated inversion technique, we were able to create a suitable initial model and to perform a few model update with just a few lines of code. Nothing would change really if we applied this at a larger scale. By changing the site name to a remote site with more compute capabilities, we could easily scale up the problem.

Part 3 - Sensitivities

Part 5 - Extensions

Site Map Contact Us

Impressum

Academic License Agreement

Credits

PAGE CONTENTS