[2025-04-15 19:24:53,496] INFO: Adding new iteration #0.
[2025-04-15 19:24:53,504] INFO: Resuming iteration #0.
[2025-04-15 19:24:53,509] INFO: 1 new tasks have been issued.
[2025-04-15 19:24:53,510] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:24:53,588] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2025-04-15 19:24:53,613] INFO: Submitting job array with 5 jobs ...
[2025-04-15 19:24:53,984] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-04-15 19:24:53,985] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:24:55,986] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:24:56,113] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:24:58,115] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:24:58,212] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:00,214] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:25:01,977] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2025-04-15 19:25:02,036] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-04-15 19:25:02,037] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:04,039] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:25:04,330] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-04-15 19:25:04,416] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-04-15 19:25:04,418] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:06,420] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:25:06,660] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-04-15 19:25:06,725] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-04-15 19:25:06,726] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:08,727] INFO: Processing task `misfit_and_gradient`
[2025-04-15 19:25:09,165] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-04-15 19:25:09,565] INFO:
Iteration 0: Number of events: 5 chi = 0.01769007842184961 ||g|| = 0.022565013801719613
pred = --- ared = --- norm_update = --- tr_radius = ---
[2025-04-15 19:25:09,567] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:09,567] INFO: Processing task `preconditioner`
[2025-04-15 19:25:09,637] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:11,647] INFO: Processing task `preconditioner`
[2025-04-15 19:25:11,771] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:11,772] INFO: Processing task `misfit`
[2025-04-15 19:25:11,824] INFO: Submitting job array with 5 jobs ...
[2025-04-15 19:25:11,940] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-04-15 19:25:11,941] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:13,970] INFO: Processing task `misfit`
[2025-04-15 19:25:14,030] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:16,058] INFO: Processing task `misfit`
[2025-04-15 19:25:16,124] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-04-15 19:25:18,152] INFO: Processing task `misfit`
[2025-04-15 19:25:20,794] INFO:
old misfit control group: 0.01769007842184961
new misfit control group: 0.006215538423932317
predicted reduction control group: -0.009679727771306901
actual reduction control group: -0.011474539997917293
5 out of 5 event(s) improved the misfit.
[2025-04-15 19:25:20,794] INFO:
Model update accepted.
[2025-04-15 19:25:20,795] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:20,796] INFO: Processing task `finalize_iteration`
[2025-04-15 19:25:20,919] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2025-04-15 19:25:20,944] INFO: Freed up 1.2 MB of space.
[2025-04-15 19:25:20,964] INFO: Succesfully completed iteration #0.
[2025-04-15 19:25:20,966] INFO: Adding new iteration #1.
[2025-04-15 19:25:20,981] INFO: Resuming iteration #1.
[2025-04-15 19:25:20,981] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:20,982] INFO: Processing task `gradient`
[2025-04-15 19:25:21,436] INFO: Submitting job array with 5 jobs ...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
Uploading 1 files...
[2025-04-15 19:25:21,715] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-04-15 19:25:21,722] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:23,733] INFO: Processing task `gradient`
[2025-04-15 19:25:24,068] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-04-15 19:25:24,295] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-04-15 19:25:24,297] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:26,310] INFO: Processing task `gradient`
[2025-04-15 19:25:26,647] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-04-15 19:25:26,793] INFO: Some simulations are still running. Please check again to see if they are finished.
[2025-04-15 19:25:26,803] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:28,809] INFO: Processing task `gradient`
[2025-04-15 19:25:29,458] INFO: 5 events have already been submitted. They will not be submitted again.
[2025-04-15 19:25:30,186] INFO:
Iteration 1: Number of events: 5 chi = 0.006215538423932317 ||g|| = 0.009661533661238208
pred = -0.009679727771306901 ared = -0.011474539997917293 norm_update = 0.9534161427441019 tr_radius = 0.9534163618503646
[2025-04-15 19:25:30,203] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:30,204] INFO: Processing task `preconditioner`
[2025-04-15 19:25:30,385] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:32,477] INFO: Processing task `preconditioner`
[2025-04-15 19:25:32,623] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:32,624] INFO: Processing task `misfit`
[2025-04-15 19:25:32,674] INFO: Submitting job array with 5 jobs ...
[2025-04-15 19:25:32,777] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-04-15 19:25:32,778] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:34,816] INFO: Processing task `misfit`
[2025-04-15 19:25:34,870] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:36,911] INFO: Processing task `misfit`
[2025-04-15 19:25:36,968] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-04-15 19:25:39,006] INFO: Processing task `misfit`
[2025-04-15 19:25:41,253] INFO:
old misfit control group: 0.006215538423932317
new misfit control group: 0.004515146195660988
predicted reduction control group: -0.0008915795911773164
actual reduction control group: -0.0017003922282713291
5 out of 5 event(s) improved the misfit.
[2025-04-15 19:25:41,254] INFO:
Model update accepted.
[2025-04-15 19:25:41,254] INFO: 1 new tasks have been issued.
[2025-04-15 19:25:41,255] INFO: Processing task `finalize_iteration`
[2025-04-15 19:25:41,368] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2025-04-15 19:25:41,567] INFO: Freed up 4.8 MB of space.
[2025-04-15 19:25:41,568] INFO: Succesfully completed iteration #1.
[2025-04-15 19:25:41,570] INFO: Adding new iteration #2.