[2025-05-21 02:07:58,185] INFO: Adding new iteration #0.
[2025-05-21 02:07:58,202] INFO: Resuming iteration #0.
[2025-05-21 02:07:58,204] INFO: 1 new tasks have been issued.
[2025-05-21 02:07:58,209] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:07:58,412] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event_0', 'event_1', 'event_2', 'event_3', 'event_4']
[2025-05-21 02:07:58,536] INFO: Submitting job array with 5 jobs ...
[2025-05-21 02:07:58,931] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-05-21 02:07:58,935] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:00,939] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:01,131] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:03,137] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:03,453] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:05,457] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:06,347] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:08,350] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:13,215] INFO: Submitting job array with 5 jobs ...
[2025-05-21 02:08:13,433] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-05-21 02:08:13,441] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:15,450] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:16,602] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:18,607] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:19,303] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:21,306] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:22,218] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:24,221] INFO: Processing task `misfit_and_gradient`
[2025-05-21 02:08:25,473] INFO:
Iteration 0: Number of events: 5 chi = 0.01768645831434902 ||g|| = 0.022567382929465314
pred = --- ared = --- norm_update = --- tr_radius = ---
[2025-05-21 02:08:25,480] INFO: 1 new tasks have been issued.
[2025-05-21 02:08:25,480] INFO: Processing task `preconditioner`
[2025-05-21 02:08:25,668] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:27,700] INFO: Processing task `preconditioner`
[2025-05-21 02:08:27,860] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:29,899] INFO: Processing task `preconditioner`
[2025-05-21 02:08:30,177] INFO: 1 new tasks have been issued.
[2025-05-21 02:08:30,178] INFO: Processing task `misfit`
[2025-05-21 02:08:30,300] INFO: Submitting job array with 5 jobs ...
[2025-05-21 02:08:30,630] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-05-21 02:08:30,633] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:32,684] INFO: Processing task `misfit`
[2025-05-21 02:08:32,866] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:34,908] INFO: Processing task `misfit`
[2025-05-21 02:08:35,542] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:37,574] INFO: Processing task `misfit`
[2025-05-21 02:08:37,668] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:39,702] INFO: Processing task `misfit`
[2025-05-21 02:08:39,854] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 02:08:41,900] INFO: Processing task `misfit`
[2025-05-21 02:08:42,975] INFO:
old misfit control group: 0.01768645831434902
new misfit control group: 0.006209860003200825
predicted reduction control group: -0.009680381232101354
actual reduction control group: -0.011476598311148195
5 out of 5 event(s) improved the misfit.
[2025-05-21 02:08:42,976] INFO:
Model update accepted.
[2025-05-21 02:08:42,977] INFO: 1 new tasks have been issued.
[2025-05-21 02:08:42,977] INFO: Processing task `finalize_iteration`
[2025-05-21 02:08:43,064] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00000
[2025-05-21 02:08:43,081] INFO: Freed up 613.0 KB of space.
[2025-05-21 02:08:43,083] INFO: Successfully completed iteration #0.
[2025-05-21 02:08:43,087] INFO: Adding new iteration #1.
[2025-05-21 02:08:43,095] INFO: Resuming iteration #1.
[2025-05-21 02:08:43,098] INFO: 1 new tasks have been issued.
[2025-05-21 02:08:43,100] INFO: Processing task `gradient`
[2025-05-21 02:08:43,511] INFO: Submitting job array with 5 jobs ...
[2025-05-21 02:08:43,661] INFO: Launched adjoint simulations for 5 events. Please check again to see if they are finished.
[2025-05-21 02:08:43,664] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:08:45,667] INFO: Processing task `gradient`
[2025-05-21 02:08:46,046] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:08:48,049] INFO: Processing task `gradient`
[2025-05-21 02:08:48,366] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:08:50,368] INFO: Processing task `gradient`
[2025-05-21 02:08:50,755] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:08:52,757] INFO: Processing task `gradient`
[2025-05-21 02:08:53,052] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:08:55,057] INFO: Processing task `gradient`
[2025-05-21 02:08:55,459] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:08:57,466] INFO: Processing task `gradient`
[2025-05-21 02:08:58,705] INFO:
Iteration 1: Number of events: 5 chi = 0.006209860003200826 ||g|| = 0.009661370770506869
pred = -0.009680381232101354 ared = -0.011476598311148195 norm_update = 0.9533702356708309 tr_radius = 0.9533703948402213
[2025-05-21 02:08:58,728] INFO: 1 new tasks have been issued.
[2025-05-21 02:08:58,732] INFO: Processing task `preconditioner`
[2025-05-21 02:08:58,910] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:00,937] INFO: Processing task `preconditioner`
[2025-05-21 02:09:01,014] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:03,053] INFO: Processing task `preconditioner`
[2025-05-21 02:09:03,284] INFO: 1 new tasks have been issued.
[2025-05-21 02:09:03,285] INFO: Processing task `misfit`
[2025-05-21 02:09:03,388] INFO: Submitting job array with 5 jobs ...
[2025-05-21 02:09:03,535] INFO: Launched simulations for 5 events. Please check again to see if they are finished.
[2025-05-21 02:09:03,538] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:05,581] INFO: Processing task `misfit`
[2025-05-21 02:09:05,672] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:07,713] INFO: Processing task `misfit`
[2025-05-21 02:09:07,820] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:09,875] INFO: Processing task `misfit`
[2025-05-21 02:09:10,028] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:12,076] INFO: Processing task `misfit`
[2025-05-21 02:09:12,223] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 02:09:14,293] INFO: Processing task `misfit`
[2025-05-21 02:09:15,349] INFO:
old misfit control group: 0.006209860003200825
new misfit control group: 0.004509133855903811
predicted reduction control group: -0.0008916694835079397
actual reduction control group: -0.0017007261472970145
5 out of 5 event(s) improved the misfit.
[2025-05-21 02:09:15,352] INFO:
Model update accepted.
[2025-05-21 02:09:15,353] INFO: 1 new tasks have been issued.
[2025-05-21 02:09:15,354] INFO: Processing task `finalize_iteration`
[2025-05-21 02:09:15,428] INFO: ... searching for obsolete files in project_full_waveform_inversion/INVERSIONS/my_inversion/00001
[2025-05-21 02:09:15,631] INFO: Freed up 3.6 MB of space.
[2025-05-21 02:09:15,633] INFO: Successfully completed iteration #1.
[2025-05-21 02:09:15,637] INFO: Adding new iteration #2.