Mondaic DocumentationIt is finally time to start the inversion and compute model updates.
A quick recap of what we have done so far:
- Part 0 defined the preferred frequency band for the inversion.
- Part 1 collected data for several earthquakes and stations all across Europe.
- Part 2 set up the domain and physics we need for creating simulated data.
- Part 3 prepared a dataset for the inversion and configured how to compare simulated seismograms to the observed data
Full-waveform inversion is a non-linear problem that can only be solved iteratively.
This entails several tasks of orchestrating the recurring simulations to create synthetic seismograms, compute misfits and sensitivities with respect to changes of the model, preconditioning and updating the model.
The scientific question is how we can formulate this as an optimization problem such that we converge toward the "true Earth" with as little computational cost as possible.
Let's get started!
import os
import os, pathlib
from frequency_band import FrequencyBand
from salvus import namespace as sn
SIMULATION_TIME_IN_SECONDS = 1200.0
SALVUS_FLOW_SITE_NAME = os.environ.get("SITE_NAME", "local")
RANKS_PER_JOB = 4
PROJECT_DIR = "project_dir_central_europe"
fband_file = pathlib.Path("./frequency_band_70_120.pkl")
fband = FrequencyBand.load(fband_file)
fbandFrequency band: 8.33 mHz - 14.3 mHz, 70.0 s - 120.0 s
FrequencyBand(min_frequency_in_hertz=0.008333333333333333, max_frequency_in_hertz=0.014285714285714285)
To safe some time, we will only iterate on a subset of events.
Make sure to use the same events for which we have selected windows in the
previous notebook.
SELECTED_EVENTS = [
"event_CRETE_GREECE_Mag_6.15_2015-04-16-18-07",
"event_ICELAND_REGION_Mag_5.75_2020-06-20-19-26",
"event_STRAIT_OF_GIBRALTAR_Mag_6.38_2016-01-25-04-22",
]# Load the project with the data from the previous parts.
p = sn.Project(path=PROJECT_DIR)
Inverse problems are "ill-posed", which makes them hard to solve.
Specifically, inverse modelling requires to carefully set up the interplay of several components in order to retrieve a reasonable model as the result of the inversion. This entails several design choices:
Data
- Which events to consider? (
events) - How to compare synthetic to observed data? (
misfit_configuration)
Model
- What is the prior model? (
prior_model) - Which parameters to invert for? (
mapping)
Optimization
- How to find a better model? (
method) - Which additional information can be used to constrain the model? (
preconditioner)
Workflow
- Where to run the jobs?
(
job_submission)
The
InverseProblemConfiguration collects these parameters to setup the inverse problem.
Often enough, it is hard to choose the best setting a-prior, which is why the inverse problem is constructed as a tree that allows to branch off an ongoing inversion with changed parameters, or to run several scenarios concurrently.Let's consider only two events at first. We will demonstrate how to add the third event to an ongoing inversion later on.
p += sn.InverseProblemConfiguration(
name=f"inv_{fband.period_band_name}",
events=SELECTED_EVENTS[:2],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
wavefield_compression=sn.WavefieldCompression(
forward_wavefield_sampling_interval=10
),
prior_model=f"initial_model_{fband.period_band_name}",
mapping=sn.Mapping(
scaling="relative_deviation_from_prior",
inversion_parameters=["VSH", "VSV", "VP"],
map_to_physical_parameters={
"VPV": "VP",
"VPH": "VP",
},
source_cutout_radius_in_meters=300000,
receiver_cutout_radius_in_meters=50000,
),
method=sn.TrustRegion(initial_trust_region_linf=0.02),
preconditioner=sn.ModelDependentSmoothing(
smoothing_lengths_in_wavelengths={
"VSH": [0.2, 0.5, 0.5],
"VSV": [0.2, 0.5, 0.5],
"VP": [0.2, 0.5, 0.5],
},
reference_frequency_in_hertz=fband.max_frequency_in_hertz,
reference_model="prior",
reference_velocities={"VSH": "VSH", "VSV": "VSH", "VP": "VSH"},
),
job_submission={
"forward": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"adjoint": sn.SiteConfig(
site_name=SALVUS_FLOW_SITE_NAME, ranks_per_job=RANKS_PER_JOB
),
"preconditioner": sn.SiteConfig(
site_name="local", ranks_per_job=RANKS_PER_JOB
),
},
)Let's start by adding the first iteration.
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)[2025-10-03 22:42:48,090] INFO: Adding new iteration #0.
True
The progress of the inversion can be visualized in a widget. At the moment, there is not a lot to show, obviously...
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
Many steps within a single iteration involve expensive simulations, e.g., for computing misfits or adjoint simulations to compute gradients. This is why
Salvus operates a task-based workflow.In order to be able to closely monitor the progress,
SalvusOpt steps through an iteration, and automatically dispatches simulations whenever necessary. The function resume will return whenever SalvusOpt is waiting for other tasks to finish first. Calling it several time, will step through the iteration in sequence. The log messages inform about the current status and tasks.p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)[2025-10-03 22:42:48,946] INFO: Resuming iteration #0. Current stage: initialize [2025-10-03 22:42:48,950] INFO: 1 new tasks have been issued. [2025-10-03 22:42:48,950] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:42:48,950] INFO: -> Mapping between inversion and simulation space: 0 ms [2025-10-03 22:42:50,623] INFO: Submitting job array with 2 jobs ... [2025-10-03 22:42:50,681] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2025-10-03 22:42:50,682] INFO: Some tasks of iteration #0 are still running. Please check again later.
True
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)[2025-10-03 22:42:50,689] INFO: Resuming iteration #0. Current stage: check_convergence [2025-10-03 22:42:50,689] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:42:50,690] INFO: -> Mapping between inversion and simulation space: 0 ms [2025-10-03 22:42:50,836] INFO: Some tasks of iteration #0 are still running. Please check again later.
False
While the simulations are running, we can take a closer look at tasks within an iteration, and how they can efficiently be dealt with.
p.inversions.resume(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)[2025-10-03 22:42:50,842] INFO: Resuming iteration #0. Current stage: check_convergence [2025-10-03 22:42:50,843] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:42:50,843] INFO: -> Mapping between inversion and simulation space: 0 ms [2025-10-03 22:42:51,034] INFO: Some tasks of iteration #0 are still running. Please check again later.
False
You can always visualize the current state of an iteration in a widget, where you find a lot of information about the model, misfits and gradients. The contents will be updated as you step through the iteration. The widget will be updated once gradients are available.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
If you feel confident and don't want to be bothered with every single task, you can also tell
Salvus to run an entire iteration at once. Note the parameter timeout_in_seconds, which will force the cell to return even if the iteration has not been completed yet, and there might still be a few simulations running in the back.
Again, you can execute the cell several times or mix it with calls to the previous one until the iteration is complete.p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=1800,
ping_interval_in_seconds=100,
)[2025-10-03 22:42:51,548] INFO: Resuming iteration #0. [2025-10-03 22:42:51,549] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:42:51,733] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-10-03 22:44:31,735] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:44:32,056] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-10-03 22:46:12,059] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:46:12,391] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-10-03 22:47:52,394] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:47:53,097] INFO: Iteration 0: Number of events: 2 chi = 290.87482935994683 ||g|| = 1.1918071926234806e-05 pred = --- ared = --- norm_update = --- tr_radius = --- [2025-10-03 22:47:53,098] INFO: 1 new tasks have been issued. [2025-10-03 22:47:53,099] INFO: Processing task `preconditioner` [2025-10-03 22:47:53,357] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-10-03 22:49:33,405] INFO: Processing task `preconditioner` [2025-10-03 22:49:33,851] INFO: 1 new tasks have been issued. [2025-10-03 22:49:33,852] INFO: Processing task `misfit` [2025-10-03 22:49:34,092] INFO: Submitting job array with 2 jobs ... [2025-10-03 22:49:34,191] INFO: Launched simulations for 2 events. Please check again to see if they are finished. [2025-10-03 22:49:34,192] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-10-03 22:51:14,340] INFO: Processing task `misfit` [2025-10-03 22:51:14,426] INFO: Some tasks of iteration #0 are still running. Please check again later. [2025-10-03 22:52:54,633] INFO: Processing task `misfit` [2025-10-03 22:52:57,213] INFO: old misfit control group: 290.87482935994683 new misfit control group: 212.97157335531546 predicted reduction control group: -39.550830364227295 actual reduction control group: -77.90325600463137 2 out of 2 event(s) improved the misfit. [2025-10-03 22:52:57,213] INFO: Model update accepted. [2025-10-03 22:52:57,214] INFO: 1 new tasks have been issued. [2025-10-03 22:52:57,215] INFO: Processing task `finalize_iteration` [2025-10-03 22:52:57,475] INFO: Successfully completed iteration #0. [2025-10-03 22:52:57,477] INFO: Adding new iteration #1.
Here are some more details on the preconditioner, and how the trust region method determines model updates.
At the end of an iteration, a new iteration with the previous model update is automatically added.
We can now visualize the entire inversion in a widget to get an overview of the evolution of models and misfits during the iterations.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
Before continuing with more iterations, let's have a more detailed look at intermediate data of the iteration.
The widget above provides only a preview of the model, and there are a lot more details to discover when visualizing models and gradients in Paraview. The cell below prints the model directory of the first iteration. Copy the path and use
scp to transfer the data to your local machine.print(
p.inversions.get_iteration_directory(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=0,
).absolute()
)/tmp/tmpi94pp1yi/project_dir_central_europe/INVERSIONS/inv_70.0_120.0_seconds/00000
Before we explore the model in Paraview, let's continue to run the inversion in the background.
Instead of continuing with the previous set of events, we can now add the
third event. We can simply do so by taking all from the previous iteration
and adding the desired events.
Here we first add the event to the inversion at large, and subsequently, we
create a new iteration that explicitly also includes new events. By default,
the created new iteration will inherit the events selected by its parent.
p.inversions.add_events(
f"inv_{fband.period_band_name}",
SELECTED_EVENTS[-1],
)
previous_events = p.inversions.get_iteration(
f"inv_{fband.period_band_name}", 0
).events
new_events = [
SELECTED_EVENTS[-1],
]
p.inversions.add_iteration(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
parent_id=0,
events=previous_events + new_events,
)[2025-10-03 22:52:58,903] INFO: Adding new iteration #2.
True
Let's have a look at the inversion tree. You will notice that we now have two iterations that inherit from the first iteration. While iteration #1 was automatically created previously, we have now added iteration #2 which operates on the same model, but contains the three events. You can hover over the green nodes in the graph to see this info. By default,
p.inversion.iterate will always continue on the last added iteration marked by the green edge in the inversion tree.The misfit for the third event has not yet been computed, but
SalvusOpt will automatically trigger the simulation once we continue with the inversion.p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
)
The first iteration was only one step in model space to reduce the misfit, but the nonlinear problem will take many iterations to converge, of course. We can loop around the
iterate function and create a few more model updates.for i in range(2):
p.inversions.iterate(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
timeout_in_seconds=3600,
ping_interval_in_seconds=120,
)[2025-10-03 22:52:59,864] INFO: Resuming iteration #2. [2025-10-03 22:52:59,868] INFO: 2 new tasks have been issued. [2025-10-03 22:52:59,869] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:53:00,000] INFO: Submitting job ... [2025-10-03 22:53:00,100] INFO: Launched simulations for 1 events. Please check again to see if they are finished. [2025-10-03 22:53:00,101] INFO: Processing task `gradient` [2025-10-03 22:53:00,316] INFO: Submitting job array with 2 jobs ... [2025-10-03 22:53:00,402] INFO: Launched adjoint simulations for 2 events. Please check again to see if they are finished. [2025-10-03 22:53:00,405] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-10-03 22:55:00,411] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:55:01,845] INFO: Submitting job ... [2025-10-03 22:55:01,935] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished. [2025-10-03 22:55:01,936] INFO: Processing task `gradient` [2025-10-03 22:55:02,128] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-10-03 22:57:02,132] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:57:02,418] INFO: Processing task `gradient` [2025-10-03 22:57:02,691] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-10-03 22:59:02,697] INFO: Processing task `misfit_and_gradient` [2025-10-03 22:59:03,054] INFO: Processing task `gradient` [2025-10-03 22:59:03,718] INFO: Iteration 2: Number of events: 3 chi = 244.54261020800652 ||g|| = 1.1151048223257593e-05 pred = -39.550830364227295 ared = -77.90325600463137 norm_update = 9033287.05520997 tr_radius = 9033287.05520997 [2025-10-03 22:59:03,860] INFO: 1 new tasks have been issued. [2025-10-03 22:59:03,860] INFO: Processing task `preconditioner` [2025-10-03 22:59:04,146] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-10-03 23:01:04,239] INFO: Processing task `preconditioner` [2025-10-03 23:01:04,741] INFO: 1 new tasks have been issued. [2025-10-03 23:01:04,745] INFO: Processing task `misfit` [2025-10-03 23:01:05,046] INFO: Submitting job array with 3 jobs ... [2025-10-03 23:01:05,203] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2025-10-03 23:01:05,205] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-10-03 23:03:05,448] INFO: Processing task `misfit` [2025-10-03 23:03:05,562] INFO: Some tasks of iteration #2 are still running. Please check again later. [2025-10-03 23:05:05,875] INFO: Processing task `misfit` [2025-10-03 23:05:09,556] INFO: old misfit control group: 244.54261020800652 new misfit control group: 160.9897234097533 predicted reduction control group: -59.06042766571045 actual reduction control group: -83.55288679825321 3 out of 3 event(s) improved the misfit. [2025-10-03 23:05:09,556] INFO: Model update accepted. [2025-10-03 23:05:09,557] INFO: 1 new tasks have been issued. [2025-10-03 23:05:09,557] INFO: Processing task `finalize_iteration` [2025-10-03 23:05:09,940] INFO: Successfully completed iteration #2. [2025-10-03 23:05:09,944] INFO: Adding new iteration #3. [2025-10-03 23:05:09,964] INFO: Resuming iteration #3. [2025-10-03 23:05:09,969] INFO: 1 new tasks have been issued. [2025-10-03 23:05:09,972] INFO: Processing task `gradient` [2025-10-03 23:05:10,237] INFO: Submitting job array with 3 jobs ... [2025-10-03 23:05:10,311] INFO: Launched adjoint simulations for 3 events. Please check again to see if they are finished. [2025-10-03 23:05:10,311] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:07:10,315] INFO: Processing task `gradient` [2025-10-03 23:07:10,675] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:09:10,685] INFO: Processing task `gradient` [2025-10-03 23:09:11,115] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:11:11,131] INFO: Processing task `gradient` [2025-10-03 23:11:11,433] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:13:11,436] INFO: Processing task `gradient` [2025-10-03 23:13:13,012] INFO: Iteration 3: Number of events: 3 chi = 160.9897234097533 ||g|| = 9.092755084101174e-06 pred = -59.06042766571045 ared = -83.55288679825321 norm_update = 18066574.632775966 tr_radius = 18066574.11041994 [2025-10-03 23:13:13,471] INFO: 1 new tasks have been issued. [2025-10-03 23:13:13,472] INFO: Processing task `preconditioner` [2025-10-03 23:13:14,404] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:15:14,682] INFO: Processing task `preconditioner` [2025-10-03 23:15:15,427] INFO: 1 new tasks have been issued. [2025-10-03 23:15:15,428] INFO: Processing task `misfit` [2025-10-03 23:15:15,812] INFO: Submitting job array with 3 jobs ... [2025-10-03 23:15:15,997] INFO: Launched simulations for 3 events. Please check again to see if they are finished. [2025-10-03 23:15:16,000] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:17:16,655] INFO: Processing task `misfit` [2025-10-03 23:17:16,793] INFO: Some tasks of iteration #3 are still running. Please check again later. [2025-10-03 23:19:17,224] INFO: Processing task `misfit` [2025-10-03 23:19:21,133] INFO: old misfit control group: 160.9897234097533 new misfit control group: 130.66573437716988 predicted reduction control group: -21.03584885597229 actual reduction control group: -30.323989032583427 3 out of 3 event(s) improved the misfit. [2025-10-03 23:19:21,134] INFO: Model update accepted. [2025-10-03 23:19:21,134] INFO: 1 new tasks have been issued. [2025-10-03 23:19:21,135] INFO: Processing task `finalize_iteration` [2025-10-03 23:19:21,684] INFO: Successfully completed iteration #3. [2025-10-03 23:19:21,688] INFO: Adding new iteration #4.
p.viz.nb.inversion(
inverse_problem_configuration=f"inv_{fband.period_band_name}"
)
The trust-region method ensures that the cumulative misfit of all selected events goes down. This is of course only an incomplete picture of what happened. SalvusProject offers a few ways to analyze the data in detail.
One option is to have a look at the misfit reduction from the starting model to the final iteration at each receiver on a map. Keep in mind that the misfit at each receiver is still summed over all components and windows.
p.viz.nb.misfit_comparison(
reference_data=f"initial_model_{fband.period_band_name}",
other_data=[
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
)
],
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
event=SELECTED_EVENTS[-1],
)| ini... (reference) | inv... | Reduction inv... | |
|---|---|---|---|
| II.BORG.00 | 5.4908e+01 | 4.5690e+01 | 9.2182e+00 |
| II.ESK.00 | 6.5725e+01 | 2.7721e+01 | 3.8004e+01 |
| II.LVZ.00 | 3.9906e+01 | 2.2471e+01 | 1.7435e+01 |
| II.OBN.00 | 1.6779e+01 | 7.4664e+00 | 9.3130e+00 |
| IU.GRFO. | 5.9782e+01 | 2.3019e+01 | 3.6763e+01 |
| IU.KEV.00 | 6.4322e+01 | 1.7545e+01 | 4.6777e+01 |
| IU.KONO.00 | 3.8939e+01 | 2.5005e+01 | 1.3934e+01 |
| IU.PAB.00 | 2.4876e+00 | 1.9263e+00 | 5.6127e-01 |
<pandas.io.formats.style.Styler at 0x79329a43b390>
Another option is to look at misfit statistics in the form of histograms.
p.viz.misfit_histogram(
simulation_configuration_a=f"initial_model_{fband.period_band_name}",
simulation_configuration_b=p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
misfit_configuration=f"tf_phase_misfit_{fband.period_band_name}",
events=SELECTED_EVENTS[-1],
merge_all_components=False,
)
Last but not least the most important thing is that the waveform got better so let's now also take a look at the waveforms after the three iterations. The misfit has been reduced for all events.
And although the changes are small, we do observe a better phase match on average and on some stations the difference is significant. Check out the event
STRAIT_OF_GIBRALTAR and station GB.GAL1 for a good example.
Note that only the parts of the waveforms within the shaded windows informed the inverse problem about the model space.
Interestingly, however, as we approach better models some wiggles that were not selected also match the data better now then they did initially; see station RO.BUR31 for an example.
We could make use of this, re-pick windows and then continue with more iterations and a modified MisfitConfiguration.p.viz.nb.waveforms(
[
f"PROCESSED_DATA:{fband.period_band_name}",
f"initial_model_{fband.period_band_name}",
p.inversions.get_simulation_name(
inverse_problem_configuration=f"inv_{fband.period_band_name}",
iteration_id=4,
),
],
receiver_field="displacement",
data_selection_configuration=f"initial_selection_{fband.period_band_name}",
)
That's the end of the 4-part tutorial on continental waveform inversion.
Although we restricted the dataset to only three events, we have seen all the ingredients to automate the entire toolchain for full-waveform inversion from composing a dataset to computing model updates.
In fact, this is not the end, but rather the starting point to dive into more complicated domains and settings.
Keep in mind that there are tons of options to play around with that will help to find the best setup for the inversion.
PAGE CONTENTS