[2025-05-21 03:37:21,270] INFO: Adding new iteration #0.
[2025-05-21 03:37:21,275] INFO: Resuming iteration #0.
[2025-05-21 03:37:21,276] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:21,276] INFO: Processing task `misfit_and_gradient`
[2025-05-21 03:37:21,291] INFO: The following events have been simulated before, but checkpoints are not available for this combination of `site_name` and `ranks_per_job` and wavefield compression settings. They will be run again: ['event']
[2025-05-21 03:37:21,302] INFO: Submitting job ...
[2025-05-21 03:37:21,345] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:21,346] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 03:37:22,347] INFO: Processing task `misfit_and_gradient`
[2025-05-21 03:37:22,552] INFO: Submitting job ...
[2025-05-21 03:37:22,591] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:22,591] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 03:37:23,592] INFO: Processing task `misfit_and_gradient`
[2025-05-21 03:37:23,672] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 03:37:24,674] INFO: Processing task `misfit_and_gradient`
[2025-05-21 03:37:24,857] INFO:
Iteration 0: Number of events: 1 chi = 3.720595434250637e-10 ||g|| = 3.3917093610134566e-12
pred = --- ared = --- norm_update = --- tr_radius = ---
[2025-05-21 03:37:24,861] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:24,861] INFO: Processing task `misfit`
[2025-05-21 03:37:24,884] INFO: Submitting job ...
[2025-05-21 03:37:24,927] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:24,927] INFO: Some tasks of iteration #0 are still running. Please check again later.
[2025-05-21 03:37:25,936] INFO: Processing task `misfit`
[2025-05-21 03:37:26,044] INFO:
old misfit control group: 3.720595434250637e-10
new misfit control group: 1.0760448857825922e-10
predicted reduction control group: -2.1660988963872335e-10
actual reduction control group: -2.6445505484680453e-10
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:26,044] INFO:
Model update accepted.
[2025-05-21 03:37:26,045] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:26,045] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:26,061] INFO: Successfully completed iteration #0.
[2025-05-21 03:37:26,064] INFO: Adding new iteration #1.
[2025-05-21 03:37:26,068] INFO: Resuming iteration #1.
[2025-05-21 03:37:26,069] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:26,069] INFO: Processing task `gradient`
[2025-05-21 03:37:26,116] INFO: Submitting job ...
[2025-05-21 03:37:26,156] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:26,156] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 03:37:27,157] INFO: Processing task `gradient`
[2025-05-21 03:37:27,297] INFO:
Iteration 1: Number of events: 1 chi = 1.0760448857825922e-10 ||g|| = 1.0378109419557951e-12
pred = -2.1660988963872335e-10 ared = -2.6445505484680453e-10 norm_update = 127.72906648856653 tr_radius = 127.72904464939013
[2025-05-21 03:37:27,305] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:27,306] INFO: Processing task `misfit`
[2025-05-21 03:37:27,331] INFO: Submitting job ...
[2025-05-21 03:37:27,375] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:27,375] INFO: Some tasks of iteration #1 are still running. Please check again later.
[2025-05-21 03:37:28,383] INFO: Processing task `misfit`
[2025-05-21 03:37:28,511] INFO:
old misfit control group: 1.0760448857825922e-10
new misfit control group: 7.302845722434396e-11
predicted reduction control group: -1.8959662541394393e-11
actual reduction control group: -3.457603135391526e-11
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:28,512] INFO:
Model update accepted.
[2025-05-21 03:37:28,512] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:28,512] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:28,533] INFO: Successfully completed iteration #1.
[2025-05-21 03:37:28,535] INFO: Adding new iteration #2.
[2025-05-21 03:37:28,541] INFO: Resuming iteration #2.
[2025-05-21 03:37:28,542] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:28,543] INFO: Processing task `gradient`
[2025-05-21 03:37:28,591] INFO: Submitting job ...
[2025-05-21 03:37:28,632] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:28,632] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-05-21 03:37:29,633] INFO: Processing task `gradient`
[2025-05-21 03:37:29,794] INFO:
Iteration 2: Number of events: 1 chi = 7.302845722434396e-11 ||g|| = 8.865559234339849e-13
pred = -1.8959662541394393e-11 ared = -3.457603135391526e-11 norm_update = 38.230054884872935 tr_radius = 255.45808929878027
[2025-05-21 03:37:29,806] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:29,806] INFO: Processing task `misfit`
[2025-05-21 03:37:29,827] INFO: Submitting job ...
[2025-05-21 03:37:29,868] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:29,868] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-05-21 03:37:30,877] INFO: Processing task `misfit`
[2025-05-21 03:37:30,913] INFO: Some tasks of iteration #2 are still running. Please check again later.
[2025-05-21 03:37:31,923] INFO: Processing task `misfit`
[2025-05-21 03:37:32,047] INFO:
old misfit control group: 7.302845722434396e-11
new misfit control group: 7.241697804208357e-12
predicted reduction control group: -5.408397892676855e-11
actual reduction control group: -6.57867594201356e-11
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:32,047] INFO:
Model update accepted.
[2025-05-21 03:37:32,047] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:32,048] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:32,065] INFO: Successfully completed iteration #2.
[2025-05-21 03:37:32,067] INFO: Adding new iteration #3.
[2025-05-21 03:37:32,073] INFO: Resuming iteration #3.
[2025-05-21 03:37:32,074] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:32,074] INFO: Processing task `gradient`
[2025-05-21 03:37:32,121] INFO: Submitting job ...
[2025-05-21 03:37:32,162] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:32,163] INFO: Some tasks of iteration #3 are still running. Please check again later.
[2025-05-21 03:37:33,165] INFO: Processing task `gradient`
[2025-05-21 03:37:33,320] INFO:
Iteration 3: Number of events: 1 chi = 7.241697804208357e-12 ||g|| = 2.1714864363034736e-13
pred = -5.408397892676855e-11 ared = -6.57867594201356e-11 norm_update = 127.72910906934997 tr_radius = 127.72904464939013
[2025-05-21 03:37:33,337] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:33,337] INFO: Processing task `misfit`
[2025-05-21 03:37:33,359] INFO: Submitting job ...
[2025-05-21 03:37:33,404] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:33,404] INFO: Some tasks of iteration #3 are still running. Please check again later.
[2025-05-21 03:37:34,415] INFO: Processing task `misfit`
[2025-05-21 03:37:34,545] INFO:
old misfit control group: 7.241697804208357e-12
new misfit control group: 3.7356273654447375e-12
predicted reduction control group: -3.703992977133544e-12
actual reduction control group: -3.50607043876362e-12
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:34,545] INFO:
Model update accepted.
[2025-05-21 03:37:34,546] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:34,546] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:34,562] INFO: Successfully completed iteration #3.
[2025-05-21 03:37:34,565] INFO: Adding new iteration #4.
[2025-05-21 03:37:34,574] INFO: Resuming iteration #4.
[2025-05-21 03:37:34,575] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:34,575] INFO: Processing task `gradient`
[2025-05-21 03:37:34,624] INFO: Submitting job ...
[2025-05-21 03:37:34,662] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:34,663] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-05-21 03:37:35,665] INFO: Processing task `gradient`
[2025-05-21 03:37:35,812] INFO:
Iteration 4: Number of events: 1 chi = 3.7356273654447375e-12 ||g|| = 5.64655327736181e-14
pred = -3.703992977133544e-12 ared = -3.50607043876362e-12 norm_update = 37.89899789698217 tr_radius = 255.45808929878027
[2025-05-21 03:37:35,835] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:35,836] INFO: Processing task `misfit`
[2025-05-21 03:37:35,859] INFO: Submitting job ...
[2025-05-21 03:37:35,899] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:35,900] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-05-21 03:37:36,909] INFO: Processing task `misfit`
[2025-05-21 03:37:37,084] INFO: Some tasks of iteration #4 are still running. Please check again later.
[2025-05-21 03:37:38,093] INFO: Processing task `misfit`
[2025-05-21 03:37:38,217] INFO:
old misfit control group: 3.7356273654447375e-12
new misfit control group: 3.5501211688930347e-12
predicted reduction control group: -1.1112906620095673e-13
actual reduction control group: -1.8550619655170285e-13
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:38,217] INFO:
Model update accepted.
[2025-05-21 03:37:38,218] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:38,218] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:38,237] INFO: Successfully completed iteration #4.
[2025-05-21 03:37:38,240] INFO: Adding new iteration #5.
[2025-05-21 03:37:38,246] INFO: Resuming iteration #5.
[2025-05-21 03:37:38,247] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:38,247] INFO: Processing task `gradient`
[2025-05-21 03:37:38,296] INFO: Submitting job ...
[2025-05-21 03:37:38,335] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:38,335] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2025-05-21 03:37:39,337] INFO: Processing task `gradient`
[2025-05-21 03:37:39,494] INFO:
Iteration 5: Number of events: 1 chi = 3.5501211688930347e-12 ||g|| = 3.6539128197210645e-14
pred = -1.1112906620095673e-13 ared = -1.8550619655170285e-13 norm_update = 4.751473853028537 tr_radius = 127.72904464939013
[2025-05-21 03:37:39,523] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:39,523] INFO: Processing task `misfit`
[2025-05-21 03:37:39,545] INFO: Submitting job ...
[2025-05-21 03:37:39,584] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:39,584] INFO: Some tasks of iteration #5 are still running. Please check again later.
[2025-05-21 03:37:40,594] INFO: Processing task `misfit`
[2025-05-21 03:37:40,698] INFO:
old misfit control group: 3.5501211688930347e-12
new misfit control group: 2.974717472944843e-12
predicted reduction control group: -3.728195281722693e-13
actual reduction control group: -5.754036959481917e-13
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:40,699] INFO:
Model update accepted.
[2025-05-21 03:37:40,699] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:40,699] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:40,716] INFO: Successfully completed iteration #5.
[2025-05-21 03:37:40,719] INFO: Adding new iteration #6.
[2025-05-21 03:37:40,729] INFO: Resuming iteration #6.
[2025-05-21 03:37:40,730] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:40,730] INFO: Processing task `gradient`
[2025-05-21 03:37:40,777] INFO: Submitting job ...
[2025-05-21 03:37:40,816] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:40,816] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-05-21 03:37:41,819] INFO: Processing task `gradient`
[2025-05-21 03:37:41,974] INFO:
Iteration 6: Number of events: 1 chi = 2.974717472944843e-12 ||g|| = 1.0651379759477398e-13
pred = -3.728195281722693e-13 ared = -5.754036959481917e-13 norm_update = 21.937566197395977 tr_radius = 63.864522324695066
[2025-05-21 03:37:42,009] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:42,009] INFO: Processing task `misfit`
[2025-05-21 03:37:42,028] INFO: Submitting job ...
[2025-05-21 03:37:42,071] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:42,071] INFO: Some tasks of iteration #6 are still running. Please check again later.
[2025-05-21 03:37:43,081] INFO: Processing task `misfit`
[2025-05-21 03:37:43,218] INFO:
old misfit control group: 2.974717472944843e-12
new misfit control group: 2.0812511787034594e-12
predicted reduction control group: -5.685072367956258e-13
actual reduction control group: -8.934662942413836e-13
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:43,219] INFO:
Model update accepted.
[2025-05-21 03:37:43,219] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:43,219] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:43,236] INFO: Successfully completed iteration #6.
[2025-05-21 03:37:43,238] INFO: Adding new iteration #7.
[2025-05-21 03:37:43,248] INFO: Resuming iteration #7.
[2025-05-21 03:37:43,249] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:43,249] INFO: Processing task `gradient`
[2025-05-21 03:37:43,296] INFO: Submitting job ...
[2025-05-21 03:37:43,334] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:43,335] INFO: Some tasks of iteration #7 are still running. Please check again later.
[2025-05-21 03:37:44,337] INFO: Processing task `gradient`
[2025-05-21 03:37:44,490] INFO:
Iteration 7: Number of events: 1 chi = 2.0812511787034594e-12 ||g|| = 1.7389392145768846e-13
pred = -5.685072367956258e-13 ared = -8.934662942413836e-13 norm_update = 43.87518718611967 tr_radius = 43.875132394791954
[2025-05-21 03:37:44,529] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:44,529] INFO: Processing task `misfit`
[2025-05-21 03:37:44,551] INFO: Submitting job ...
[2025-05-21 03:37:44,592] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:44,592] INFO: Some tasks of iteration #7 are still running. Please check again later.
[2025-05-21 03:37:45,604] INFO: Processing task `misfit`
[2025-05-21 03:37:45,726] INFO:
old misfit control group: 2.0812511787034594e-12
new misfit control group: 8.064149649116824e-13
predicted reduction control group: -9.09634563006146e-13
actual reduction control group: -1.2748362137917771e-12
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:45,726] INFO:
Model update accepted.
[2025-05-21 03:37:45,727] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:45,727] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:45,749] INFO: Successfully completed iteration #7.
[2025-05-21 03:37:45,752] INFO: Adding new iteration #8.
[2025-05-21 03:37:45,764] INFO: Resuming iteration #8.
[2025-05-21 03:37:45,765] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:45,765] INFO: Processing task `gradient`
[2025-05-21 03:37:45,812] INFO: Submitting job ...
[2025-05-21 03:37:45,851] INFO: Launched adjoint simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:45,852] INFO: Some tasks of iteration #8 are still running. Please check again later.
[2025-05-21 03:37:46,854] INFO: Processing task `gradient`
[2025-05-21 03:37:47,000] INFO:
Iteration 8: Number of events: 1 chi = 8.064149649116824e-13 ||g|| = 1.7955265349563855e-13
pred = -9.09634563006146e-13 ared = -1.2748362137917771e-12 norm_update = 82.38718000440856 tr_radius = 87.75026478958391
[2025-05-21 03:37:47,042] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:47,042] INFO: Processing task `misfit`
[2025-05-21 03:37:47,064] INFO: Submitting job ...
[2025-05-21 03:37:47,105] INFO: Launched simulations for 1 events. Please check again to see if they are finished.
[2025-05-21 03:37:47,106] INFO: Some tasks of iteration #8 are still running. Please check again later.
[2025-05-21 03:37:48,116] INFO: Processing task `misfit`
[2025-05-21 03:37:48,235] INFO:
old misfit control group: 8.064149649116824e-13
new misfit control group: 1.646649328934484e-14
predicted reduction control group: -6.997768758497305e-13
actual reduction control group: -7.899484716223375e-13
1 out of 1 event(s) improved the misfit.
[2025-05-21 03:37:48,236] INFO:
Model update accepted.
[2025-05-21 03:37:48,236] INFO: 1 new tasks have been issued.
[2025-05-21 03:37:48,236] INFO: Processing task `finalize_iteration`
[2025-05-21 03:37:48,254] INFO: Successfully completed iteration #8.
[2025-05-21 03:37:48,260] INFO: Adding new iteration #9.
[2025-05-21 03:37:48,273] INFO: Resuming iteration #9.
[2025-05-21 03:37:48,274] INFO: The total number of iterations in this inversion tree has been reached and no new iteration will be added. If you want to continue with this inversion, you need to adjust `max_iterations_global` using `set_stopping_criteria`.