隨手記錄log, 512×512 與 384×384 相比, 訓練時間拉長約 2.28倍, 首輪 log 數據比較有感, 差異比較大.
figure:
log:
Epoch: [ 0], Batch: [ 0/ 831] | Total Time: 3s
d_loss: 1.8555, g_loss: 48.5514, const_loss: 0.0003, l1_loss: 35.8118, fm_loss: 0.0314, perc_loss: 11.8152, edge: 0.1999
Checkpoint step 100 reached, but saving starts after step 200.
Epoch: [ 0], Batch: [ 100/ 831] | Total Time: 3m 42s
d_loss: 1.3870, g_loss: 41.4099, const_loss: 0.0001, l1_loss: 30.0919, fm_loss: 0.0299, perc_loss: 10.3960, edge: 0.1991
Epoch: [ 0], Batch: [ 200/ 831] | Total Time: 7m 29s
d_loss: 1.3868, g_loss: 36.1580, const_loss: 0.0001, l1_loss: 25.9894, fm_loss: 0.0238, perc_loss: 9.2647, edge: 0.1872
Epoch: [ 0], Batch: [ 300/ 831] | Total Time: 11m 14s
d_loss: 1.3878, g_loss: 39.6941, const_loss: 0.0001, l1_loss: 28.1615, fm_loss: 0.0257, perc_loss: 10.6019, edge: 0.2120
Epoch: [ 0], Batch: [ 400/ 831] | Total Time: 14m 55s
d_loss: 1.3868, g_loss: 38.5238, const_loss: 0.0001, l1_loss: 27.4452, fm_loss: 0.0246, perc_loss: 10.1615, edge: 0.1991
Epoch: [ 0], Batch: [ 500/ 831] | Total Time: 18m 36s
d_loss: 1.3868, g_loss: 38.4518, const_loss: 0.0002, l1_loss: 27.4393, fm_loss: 0.0242, perc_loss: 10.1010, edge: 0.1942
Epoch: [ 0], Batch: [ 600/ 831] | Total Time: 22m 17s
d_loss: 1.3872, g_loss: 37.0487, const_loss: 0.0001, l1_loss: 26.8958, fm_loss: 0.0240, perc_loss: 9.2513, edge: 0.1846
Epoch: [ 0], Batch: [ 700/ 831] | Total Time: 25m 58s
d_loss: 1.3867, g_loss: 38.7305, const_loss: 0.0001, l1_loss: 27.9832, fm_loss: 0.0224, perc_loss: 9.8449, edge: 0.1871
Epoch: [ 0], Batch: [ 800/ 831] | Total Time: 29m 39s
d_loss: 1.3868, g_loss: 37.9838, const_loss: 0.0004, l1_loss: 28.0558, fm_loss: 0.0239, perc_loss: 9.0325, edge: 0.1784
--- End of Epoch 0 --- Time: 1843.7s ---
LR Scheduler stepped. Current LR G: 0.000200, LR D: 0.000200
Epoch: [ 1], Batch: [ 0/ 831] | Total Time: 30m 45s
d_loss: 1.3882, g_loss: 36.0948, const_loss: 0.0002, l1_loss: 25.8642, fm_loss: 0.0187, perc_loss: 9.3354, edge: 0.1831
Epoch: [ 1], Batch: [ 100/ 831] | Total Time: 34m 30s
d_loss: 1.3868, g_loss: 36.8987, const_loss: 0.0002, l1_loss: 26.4981, fm_loss: 0.0205, perc_loss: 9.4850, edge: 0.2014
Epoch: [ 1], Batch: [ 200/ 831] | Total Time: 38m 10s
d_loss: 1.3867, g_loss: 36.2823, const_loss: 0.0001, l1_loss: 26.2852, fm_loss: 0.0202, perc_loss: 9.0958, edge: 0.1881
Epoch: [ 1], Batch: [ 300/ 831] | Total Time: 41m 52s
d_loss: 1.3868, g_loss: 39.1493, const_loss: 0.0001, l1_loss: 27.9802, fm_loss: 0.0206, perc_loss: 10.2437, edge: 0.2113
Epoch: [ 1], Batch: [ 400/ 831] | Total Time: 45m 33s
d_loss: 1.3869, g_loss: 35.6709, const_loss: 0.0001, l1_loss: 25.6619, fm_loss: 0.0182, perc_loss: 9.1091, edge: 0.1886
Epoch: [ 1], Batch: [ 500/ 831] | Total Time: 49m 19s
d_loss: 1.3872, g_loss: 33.1971, const_loss: 0.0002, l1_loss: 24.2608, fm_loss: 0.0166, perc_loss: 8.0567, edge: 0.1694
Epoch: [ 1], Batch: [ 600/ 831] | Total Time: 52m 59s
d_loss: 1.3867, g_loss: 37.4253, const_loss: 0.0002, l1_loss: 26.8524, fm_loss: 0.0191, perc_loss: 9.6647, edge: 0.1960
Epoch: [ 1], Batch: [ 700/ 831] | Total Time: 56m 40s
d_loss: 1.3870, g_loss: 37.0443, const_loss: 0.0001, l1_loss: 26.4690, fm_loss: 0.0181, perc_loss: 9.6677, edge: 0.1959
Epoch: [ 1], Batch: [ 800/ 831] | Total Time: 1h 24s
d_loss: 1.3868, g_loss: 36.6171, const_loss: 0.0001, l1_loss: 26.7109, fm_loss: 0.0185, perc_loss: 9.0182, edge: 0.1766
--- End of Epoch 1 --- Time: 1844.9s ---
LR Scheduler stepped. Current LR G: 0.000199, LR D: 0.000199
Epoch: [ 2], Batch: [ 0/ 831] | Total Time: 1h 1m 30s
d_loss: 1.3869, g_loss: 33.3594, const_loss: 0.0001, l1_loss: 23.8799, fm_loss: 0.0157, perc_loss: 8.5950, edge: 0.1758
Epoch: [ 2], Batch: [ 100/ 831] | Total Time: 1h 5m 15s
d_loss: 1.3867, g_loss: 32.8736, const_loss: 0.0001, l1_loss: 23.7862, fm_loss: 0.0139, perc_loss: 8.2090, edge: 0.1712
Epoch: [ 2], Batch: [ 200/ 831] | Total Time: 1h 8m 57s
d_loss: 1.3869, g_loss: 35.8764, const_loss: 0.0001, l1_loss: 25.9877, fm_loss: 0.0159, perc_loss: 8.9948, edge: 0.1847
Epoch: [ 2], Batch: [ 300/ 831] | Total Time: 1h 12m 37s
d_loss: 1.3873, g_loss: 27.9786, const_loss: 0.0003, l1_loss: 20.1063, fm_loss: 0.0107, perc_loss: 7.0191, edge: 0.1487
Epoch: [ 2], Batch: [ 400/ 831] | Total Time: 1h 16m 18s
d_loss: 1.3868, g_loss: 32.5896, const_loss: 0.0002, l1_loss: 22.9733, fm_loss: 0.0131, perc_loss: 8.7263, edge: 0.1835
Epoch: [ 2], Batch: [ 500/ 831] | Total Time: 1h 19m 59s
d_loss: 1.3876, g_loss: 31.8721, const_loss: 0.0001, l1_loss: 22.6266, fm_loss: 0.0123, perc_loss: 8.3598, edge: 0.1800
Epoch: [ 2], Batch: [ 600/ 831] | Total Time: 1h 23m 39s
d_loss: 1.3867, g_loss: 32.4699, const_loss: 0.0001, l1_loss: 23.2523, fm_loss: 0.0126, perc_loss: 8.3410, edge: 0.1706
Epoch: [ 2], Batch: [ 700/ 831] | Total Time: 1h 27m 25s
d_loss: 1.3867, g_loss: 31.5842, const_loss: 0.0001, l1_loss: 22.5872, fm_loss: 0.0118, perc_loss: 8.1213, edge: 0.1706
Epoch: [ 2], Batch: [ 800/ 831] | Total Time: 1h 31m 7s
d_loss: 1.3872, g_loss: 36.3781, const_loss: 0.0001, l1_loss: 25.8096, fm_loss: 0.0141, perc_loss: 9.6648, edge: 0.1961
--- End of Epoch 2 --- Time: 1843.3s ---
LR Scheduler stepped. Current LR G: 0.000197, LR D: 0.000197
Epoch: [ 3], Batch: [ 0/ 831] | Total Time: 1h 32m 14s
d_loss: 1.3867, g_loss: 32.6280, const_loss: 0.0001, l1_loss: 23.3588, fm_loss: 0.0116, perc_loss: 8.3858, edge: 0.1783
Epoch: [ 3], Batch: [ 100/ 831] | Total Time: 1h 35m 54s
d_loss: 1.3867, g_loss: 36.0779, const_loss: 0.0001, l1_loss: 26.0461, fm_loss: 0.0137, perc_loss: 9.1326, edge: 0.1921
Epoch: [ 3], Batch: [ 200/ 831] | Total Time: 1h 39m 34s
d_loss: 1.3869, g_loss: 32.3989, const_loss: 0.0002, l1_loss: 23.0922, fm_loss: 0.0122, perc_loss: 8.4207, edge: 0.1802
Epoch: [ 3], Batch: [ 300/ 831] | Total Time: 1h 43m 19s
d_loss: 1.3870, g_loss: 32.2563, const_loss: 0.0001, l1_loss: 23.1509, fm_loss: 0.0112, perc_loss: 8.2281, edge: 0.1727
Epoch: [ 3], Batch: [ 400/ 831] | Total Time: 1h 46m 59s
d_loss: 1.3875, g_loss: 32.0288, const_loss: 0.0001, l1_loss: 22.8382, fm_loss: 0.0105, perc_loss: 8.3199, edge: 0.1668
Epoch: [ 3], Batch: [ 500/ 831] | Total Time: 1h 50m 42s
d_loss: 1.3869, g_loss: 33.7018, const_loss: 0.0001, l1_loss: 24.1624, fm_loss: 0.0124, perc_loss: 8.6473, edge: 0.1863
Epoch: [ 3], Batch: [ 600/ 831] | Total Time: 1h 54m 23s
d_loss: 1.3869, g_loss: 35.0490, const_loss: 0.0002, l1_loss: 25.0036, fm_loss: 0.0114, perc_loss: 9.1460, edge: 0.1944
Epoch: [ 3], Batch: [ 700/ 831] | Total Time: 1h 58m 4s
d_loss: 1.3867, g_loss: 30.9307, const_loss: 0.0001, l1_loss: 22.2519, fm_loss: 0.0096, perc_loss: 7.8038, edge: 0.1720
Epoch: [ 3], Batch: [ 800/ 831] | Total Time: 2h 1m 45s
d_loss: 1.3874, g_loss: 31.3321, const_loss: 0.0001, l1_loss: 22.3879, fm_loss: 0.0106, perc_loss: 8.0673, edge: 0.1729
--- End of Epoch 3 --- Time: 1840.6s ---
LR Scheduler stepped. Current LR G: 0.000195, LR D: 0.000195
Epoch: [ 4], Batch: [ 0/ 831] | Total Time: 2h 2m 54s
d_loss: 1.3867, g_loss: 31.1261, const_loss: 0.0001, l1_loss: 22.3814, fm_loss: 0.0110, perc_loss: 7.8735, edge: 0.1668
Epoch: [ 4], Batch: [ 100/ 831] | Total Time: 2h 6m 35s
d_loss: 1.3867, g_loss: 34.3805, const_loss: 0.0001, l1_loss: 24.8031, fm_loss: 0.0119, perc_loss: 8.6855, edge: 0.1865
Epoch: [ 4], Batch: [ 200/ 831] | Total Time: 2h 10m 15s
d_loss: 1.3868, g_loss: 32.0789, const_loss: 0.0001, l1_loss: 22.9830, fm_loss: 0.0107, perc_loss: 8.2176, edge: 0.1741
Epoch: [ 4], Batch: [ 300/ 831] | Total Time: 2h 13m 56s
d_loss: 1.3873, g_loss: 34.2499, const_loss: 0.0001, l1_loss: 24.5459, fm_loss: 0.0112, perc_loss: 8.8121, edge: 0.1872
Epoch: [ 4], Batch: [ 400/ 831] | Total Time: 2h 17m 43s
d_loss: 1.3887, g_loss: 34.3537, const_loss: 0.0001, l1_loss: 24.6973, fm_loss: 0.0113, perc_loss: 8.7695, edge: 0.1828
Epoch: [ 4], Batch: [ 500/ 831] | Total Time: 2h 21m 28s
d_loss: 1.3867, g_loss: 30.5413, const_loss: 0.0001, l1_loss: 22.1016, fm_loss: 0.0096, perc_loss: 7.5840, edge: 0.1527
Epoch: [ 4], Batch: [ 600/ 831] | Total Time: 2h 25m 8s
d_loss: 1.3868, g_loss: 34.3100, const_loss: 0.0001, l1_loss: 24.8765, fm_loss: 0.0110, perc_loss: 8.5498, edge: 0.1793
Epoch: [ 4], Batch: [ 700/ 831] | Total Time: 2h 28m 49s
d_loss: 1.3870, g_loss: 37.7089, const_loss: 0.0000, l1_loss: 26.7227, fm_loss: 0.0137, perc_loss: 10.0693, edge: 0.2098
Epoch: [ 4], Batch: [ 800/ 831] | Total Time: 2h 32m 29s
d_loss: 1.3869, g_loss: 33.5865, const_loss: 0.0001, l1_loss: 24.3283, fm_loss: 0.0126, perc_loss: 8.3819, edge: 0.1707
--- End of Epoch 4 --- Time: 1841.9s ---
LR Scheduler stepped. Current LR G: 0.000192, LR D: 0.000192
Epoch: [ 5], Batch: [ 0/ 831] | Total Time: 2h 33m 36s
d_loss: 1.3873, g_loss: 31.7718, const_loss: 0.0001, l1_loss: 22.7510, fm_loss: 0.0100, perc_loss: 8.1445, edge: 0.1729
Epoch: [ 5], Batch: [ 100/ 831] | Total Time: 2h 37m 17s
d_loss: 1.3868, g_loss: 29.6126, const_loss: 0.0001, l1_loss: 21.1932, fm_loss: 0.0093, perc_loss: 7.5534, edge: 0.1632
Epoch: [ 5], Batch: [ 200/ 831] | Total Time: 2h 40m 58s
d_loss: 1.3867, g_loss: 28.3905, const_loss: 0.0000, l1_loss: 20.3083, fm_loss: 0.0085, perc_loss: 7.2206, edge: 0.1598
Epoch: [ 5], Batch: [ 300/ 831] | Total Time: 2h 44m 40s
d_loss: 1.3870, g_loss: 28.6722, const_loss: 0.0001, l1_loss: 20.5758, fm_loss: 0.0095, perc_loss: 7.2429, edge: 0.1506
Epoch: [ 5], Batch: [ 400/ 831] | Total Time: 2h 48m 21s
d_loss: 1.3872, g_loss: 30.3480, const_loss: 0.0001, l1_loss: 21.7726, fm_loss: 0.0092, perc_loss: 7.6991, edge: 0.1737
Epoch: [ 5], Batch: [ 500/ 831] | Total Time: 2h 52m 2s
d_loss: 1.3869, g_loss: 28.6966, const_loss: 0.0001, l1_loss: 20.8293, fm_loss: 0.0092, perc_loss: 7.0095, edge: 0.1552
Epoch: [ 5], Batch: [ 600/ 831] | Total Time: 2h 55m 49s
d_loss: 1.3867, g_loss: 30.3430, const_loss: 0.0001, l1_loss: 21.7186, fm_loss: 0.0090, perc_loss: 7.7619, edge: 0.1602
Epoch: [ 5], Batch: [ 700/ 831] | Total Time: 2h 59m 31s
d_loss: 1.3867, g_loss: 26.9955, const_loss: 0.0001, l1_loss: 19.2643, fm_loss: 0.0076, perc_loss: 6.8735, edge: 0.1568
Epoch: [ 5], Batch: [ 800/ 831] | Total Time: 3h 3m 12s
d_loss: 1.3870, g_loss: 30.2710, const_loss: 0.0001, l1_loss: 21.4690, fm_loss: 0.0085, perc_loss: 7.9313, edge: 0.1687
--- End of Epoch 5 --- Time: 1842.3s ---
LR Scheduler stepped. Current LR G: 0.000189, LR D: 0.000189
Epoch: [ 6], Batch: [ 0/ 831] | Total Time: 3h 4m 18s
d_loss: 1.3867, g_loss: 29.1717, const_loss: 0.0000, l1_loss: 20.8315, fm_loss: 0.0087, perc_loss: 7.4665, edge: 0.1716
Epoch: [ 6], Batch: [ 100/ 831] | Total Time: 3h 8m 0s
d_loss: 1.3868, g_loss: 28.7507, const_loss: 0.0000, l1_loss: 20.4751, fm_loss: 0.0085, perc_loss: 7.4092, edge: 0.1645
Epoch: [ 6], Batch: [ 200/ 831] | Total Time: 3h 11m 41s
d_loss: 1.3868, g_loss: 28.8108, const_loss: 0.0000, l1_loss: 20.3923, fm_loss: 0.0081, perc_loss: 7.5529, edge: 0.1643
Epoch: [ 6], Batch: [ 300/ 831] | Total Time: 3h 15m 22s
d_loss: 1.3868, g_loss: 26.9529, const_loss: 0.0000, l1_loss: 19.3393, fm_loss: 0.0073, perc_loss: 6.7591, edge: 0.1538
Epoch: [ 6], Batch: [ 400/ 831] | Total Time: 3h 19m 3s
d_loss: 1.3867, g_loss: 30.6014, const_loss: 0.0001, l1_loss: 21.8041, fm_loss: 0.0088, perc_loss: 7.9229, edge: 0.1723
Epoch: [ 6], Batch: [ 500/ 831] | Total Time: 3h 22m 43s
d_loss: 1.3868, g_loss: 30.8951, const_loss: 0.0001, l1_loss: 22.4039, fm_loss: 0.0092, perc_loss: 7.6097, edge: 0.1789
Epoch: [ 6], Batch: [ 600/ 831] | Total Time: 3h 26m 24s
d_loss: 1.3867, g_loss: 32.4369, const_loss: 0.0000, l1_loss: 23.1193, fm_loss: 0.0103, perc_loss: 8.4328, edge: 0.1816
Epoch: [ 6], Batch: [ 700/ 831] | Total Time: 3h 30m 5s
d_loss: 1.3867, g_loss: 29.6670, const_loss: 0.0001, l1_loss: 21.0724, fm_loss: 0.0085, perc_loss: 7.7221, edge: 0.1707
Epoch: [ 6], Batch: [ 800/ 831] | Total Time: 3h 33m 46s
d_loss: 1.3867, g_loss: 28.0461, const_loss: 0.0001, l1_loss: 20.1075, fm_loss: 0.0074, perc_loss: 7.0802, edge: 0.1576
--- End of Epoch 6 --- Time: 1836.4s ---
LR Scheduler stepped. Current LR G: 0.000185, LR D: 0.000185
Epoch: [ 7], Batch: [ 0/ 831] | Total Time: 3h 34m 55s
d_loss: 1.3867, g_loss: 27.3546, const_loss: 0.0001, l1_loss: 19.8167, fm_loss: 0.0079, perc_loss: 6.6833, edge: 0.1533
Epoch: [ 7], Batch: [ 100/ 831] | Total Time: 3h 38m 35s
d_loss: 1.3868, g_loss: 28.8592, const_loss: 0.0000, l1_loss: 20.6384, fm_loss: 0.0080, perc_loss: 7.3468, edge: 0.1726
Epoch: [ 7], Batch: [ 200/ 831] | Total Time: 3h 42m 21s
d_loss: 1.3868, g_loss: 28.2519, const_loss: 0.0000, l1_loss: 20.1436, fm_loss: 0.0087, perc_loss: 7.2524, edge: 0.1539
Epoch: [ 7], Batch: [ 300/ 831] | Total Time: 3h 46m 2s
d_loss: 1.3867, g_loss: 26.5044, const_loss: 0.0002, l1_loss: 19.0200, fm_loss: 0.0071, perc_loss: 6.6267, edge: 0.1570
Epoch: [ 7], Batch: [ 400/ 831] | Total Time: 3h 49m 48s
d_loss: 1.3868, g_loss: 30.1898, const_loss: 0.0001, l1_loss: 21.3035, fm_loss: 0.0088, perc_loss: 8.0104, edge: 0.1736
Epoch: [ 7], Batch: [ 500/ 831] | Total Time: 3h 53m 36s
d_loss: 1.3868, g_loss: 32.9112, const_loss: 0.0000, l1_loss: 23.4795, fm_loss: 0.0102, perc_loss: 8.5379, edge: 0.1902
Epoch: [ 7], Batch: [ 600/ 831] | Total Time: 3h 57m 22s
d_loss: 1.3868, g_loss: 32.9340, const_loss: 0.0001, l1_loss: 23.5750, fm_loss: 0.0115, perc_loss: 8.4768, edge: 0.1773
from 1154 to 1156, 這次有增加一些訓練資料, 所以每一個 epoch 花費的時間有拉長, figure:
log:
Epoch: [ 0], Batch: [ 0/1009] | Total Time: 2s
d_loss: 1.3868, g_loss: 30.8982, const_loss: 0.0000, l1_loss: 21.9767, fm_loss: 0.0096, perc_loss: 8.0531, edge: 0.1655
Checkpoint step 100 reached, but saving starts after step 200.
Epoch: [ 0], Batch: [ 100/1009] | Total Time: 3m 37s
d_loss: 1.3869, g_loss: 28.9803, const_loss: 0.0000, l1_loss: 20.4443, fm_loss: 0.0078, perc_loss: 7.6666, edge: 0.1682
Epoch: [ 0], Batch: [ 200/1009] | Total Time: 7m 21s
d_loss: 1.3867, g_loss: 30.1839, const_loss: 0.0000, l1_loss: 21.2023, fm_loss: 0.0082, perc_loss: 8.0982, edge: 0.1818
Epoch: [ 0], Batch: [ 300/1009] | Total Time: 11m 2s
d_loss: 1.3867, g_loss: 32.5938, const_loss: 0.0001, l1_loss: 23.2653, fm_loss: 0.0093, perc_loss: 8.4389, edge: 0.1869
Epoch: [ 0], Batch: [ 400/1009] | Total Time: 14m 49s
d_loss: 1.3868, g_loss: 29.0254, const_loss: 0.0000, l1_loss: 20.6883, fm_loss: 0.0080, perc_loss: 7.4740, edge: 0.1617
Epoch: [ 0], Batch: [ 500/1009] | Total Time: 18m 31s
d_loss: 1.3869, g_loss: 32.2797, const_loss: 0.0000, l1_loss: 23.0599, fm_loss: 0.0090, perc_loss: 8.3407, edge: 0.1768
Epoch: [ 0], Batch: [ 600/1009] | Total Time: 22m 12s
d_loss: 1.3867, g_loss: 28.9536, const_loss: 0.0001, l1_loss: 20.5581, fm_loss: 0.0077, perc_loss: 7.5244, edge: 0.1698
Epoch: [ 0], Batch: [ 700/1009] | Total Time: 25m 53s
d_loss: 1.3867, g_loss: 30.3983, const_loss: 0.0001, l1_loss: 21.6764, fm_loss: 0.0082, perc_loss: 7.8442, edge: 0.1761
Epoch: [ 0], Batch: [ 800/1009] | Total Time: 29m 34s
d_loss: 1.3869, g_loss: 30.7399, const_loss: 0.0000, l1_loss: 22.0017, fm_loss: 0.0097, perc_loss: 7.8638, edge: 0.1718
Epoch: [ 0], Batch: [ 900/1009] | Total Time: 33m 15s
d_loss: 1.3868, g_loss: 32.3250, const_loss: 0.0000, l1_loss: 22.9278, fm_loss: 0.0094, perc_loss: 8.5072, edge: 0.1871
Epoch: [ 0], Batch: [1000/1009] | Total Time: 36m 57s
d_loss: 1.3869, g_loss: 26.8201, const_loss: 0.0000, l1_loss: 19.0496, fm_loss: 0.0072, perc_loss: 6.9205, edge: 0.1495
--- End of Epoch 0 --- Time: 2234.1s ---
LR Scheduler stepped. Current LR G: 0.000190, LR D: 0.000190
Epoch: [ 1], Batch: [ 0/1009] | Total Time: 37m 16s
d_loss: 1.3868, g_loss: 28.3229, const_loss: 0.0000, l1_loss: 20.0727, fm_loss: 0.0067, perc_loss: 7.3894, edge: 0.1608
Epoch: [ 1], Batch: [ 100/1009] | Total Time: 41m 2s
d_loss: 1.3869, g_loss: 31.8044, const_loss: 0.0000, l1_loss: 22.6914, fm_loss: 0.0095, perc_loss: 8.2369, edge: 0.1732
Epoch: [ 1], Batch: [ 200/1009] | Total Time: 44m 44s
d_loss: 1.3867, g_loss: 29.9654, const_loss: 0.0000, l1_loss: 21.2983, fm_loss: 0.0078, perc_loss: 7.7923, edge: 0.1735
Epoch: [ 1], Batch: [ 300/1009] | Total Time: 48m 25s
d_loss: 1.3867, g_loss: 29.1098, const_loss: 0.0000, l1_loss: 20.8463, fm_loss: 0.0084, perc_loss: 7.3878, edge: 0.1740
Epoch: [ 1], Batch: [ 400/1009] | Total Time: 52m 8s
d_loss: 1.3882, g_loss: 26.3511, const_loss: 0.0000, l1_loss: 18.8556, fm_loss: 0.0071, perc_loss: 6.6387, edge: 0.1562
Epoch: [ 1], Batch: [ 500/1009] | Total Time: 55m 48s
d_loss: 1.3867, g_loss: 33.1218, const_loss: 0.0000, l1_loss: 23.4463, fm_loss: 0.0096, perc_loss: 8.7765, edge: 0.1965
Epoch: [ 1], Batch: [ 600/1009] | Total Time: 59m 29s
d_loss: 1.3869, g_loss: 27.1650, const_loss: 0.0000, l1_loss: 19.6032, fm_loss: 0.0068, perc_loss: 6.7061, edge: 0.1555
Epoch: [ 1], Batch: [ 700/1009] | Total Time: 1h 3m 16s
d_loss: 1.3867, g_loss: 34.4517, const_loss: 0.0000, l1_loss: 24.6461, fm_loss: 0.0096, perc_loss: 8.9072, edge: 0.1955
Epoch: [ 1], Batch: [ 800/1009] | Total Time: 1h 6m 57s
d_loss: 1.3867, g_loss: 25.9927, const_loss: 0.0000, l1_loss: 18.8615, fm_loss: 0.0065, perc_loss: 6.2929, edge: 0.1384
Epoch: [ 1], Batch: [ 900/1009] | Total Time: 1h 10m 38s
d_loss: 1.3867, g_loss: 30.7794, const_loss: 0.0000, l1_loss: 21.7066, fm_loss: 0.0082, perc_loss: 8.1862, edge: 0.1850
Epoch: [ 1], Batch: [1000/1009] | Total Time: 1h 14m 20s
d_loss: 1.3868, g_loss: 27.2384, const_loss: 0.0000, l1_loss: 19.2381, fm_loss: 0.0076, perc_loss: 7.1477, edge: 0.1516
--- End of Epoch 1 --- Time: 2243.0s ---
LR Scheduler stepped. Current LR G: 0.000189, LR D: 0.000189
Epoch: [ 2], Batch: [ 0/1009] | Total Time: 1h 14m 39s
d_loss: 1.3867, g_loss: 28.9539, const_loss: 0.0000, l1_loss: 20.7931, fm_loss: 0.0083, perc_loss: 7.2900, edge: 0.1690
Epoch: [ 2], Batch: [ 100/1009] | Total Time: 1h 18m 22s
d_loss: 1.3869, g_loss: 28.5101, const_loss: 0.0000, l1_loss: 20.3565, fm_loss: 0.0073, perc_loss: 7.2837, edge: 0.1692
Epoch: [ 2], Batch: [ 200/1009] | Total Time: 1h 22m 3s
d_loss: 1.3867, g_loss: 30.9734, const_loss: 0.0000, l1_loss: 22.0455, fm_loss: 0.0080, perc_loss: 8.0474, edge: 0.1792
Epoch: [ 2], Batch: [ 300/1009] | Total Time: 1h 25m 45s
d_loss: 1.3867, g_loss: 30.1032, const_loss: 0.0000, l1_loss: 21.5729, fm_loss: 0.0076, perc_loss: 7.6616, edge: 0.1676
Epoch: [ 2], Batch: [ 400/1009] | Total Time: 1h 29m 26s
d_loss: 1.3868, g_loss: 31.2766, const_loss: 0.0000, l1_loss: 22.1735, fm_loss: 0.0079, perc_loss: 8.2260, edge: 0.1759
Epoch: [ 2], Batch: [ 500/1009] | Total Time: 1h 33m 11s
d_loss: 1.3867, g_loss: 27.0743, const_loss: 0.0000, l1_loss: 19.2202, fm_loss: 0.0066, perc_loss: 6.9907, edge: 0.1634
Epoch: [ 2], Batch: [ 600/1009] | Total Time: 1h 36m 52s
d_loss: 1.3868, g_loss: 29.5774, const_loss: 0.0000, l1_loss: 21.1771, fm_loss: 0.0076, perc_loss: 7.5297, edge: 0.1697
Epoch: [ 2], Batch: [ 700/1009] | Total Time: 1h 40m 36s
d_loss: 1.3871, g_loss: 30.8395, const_loss: 0.0000, l1_loss: 22.0560, fm_loss: 0.0077, perc_loss: 7.9053, edge: 0.1776
Epoch: [ 2], Batch: [ 800/1009] | Total Time: 1h 44m 17s
d_loss: 1.3867, g_loss: 27.0825, const_loss: 0.0000, l1_loss: 19.2367, fm_loss: 0.0063, perc_loss: 6.9839, edge: 0.1623
Epoch: [ 2], Batch: [ 900/1009] | Total Time: 1h 47m 59s
d_loss: 1.3869, g_loss: 26.7615, const_loss: 0.0000, l1_loss: 19.2430, fm_loss: 0.0074, perc_loss: 6.6605, edge: 0.1572
Epoch: [ 2], Batch: [1000/1009] | Total Time: 1h 51m 40s
d_loss: 1.3867, g_loss: 29.6087, const_loss: 0.0000, l1_loss: 20.9553, fm_loss: 0.0074, perc_loss: 7.7765, edge: 0.1761
--- End of Epoch 2 --- Time: 2240.5s ---
LR Scheduler stepped. Current LR G: 0.000187, LR D: 0.000187
Epoch: [ 3], Batch: [ 0/1009] | Total Time: 1h 51m 59s
d_loss: 1.3867, g_loss: 29.7576, const_loss: 0.0000, l1_loss: 20.9788, fm_loss: 0.0074, perc_loss: 7.9003, edge: 0.1777
Epoch: [ 3], Batch: [ 100/1009] | Total Time: 1h 55m 40s
d_loss: 1.3868, g_loss: 27.0645, const_loss: 0.0000, l1_loss: 19.1410, fm_loss: 0.0077, perc_loss: 7.0605, edge: 0.1619
Epoch: [ 3], Batch: [ 200/1009] | Total Time: 1h 59m 25s
d_loss: 1.3868, g_loss: 27.4857, const_loss: 0.0000, l1_loss: 19.4952, fm_loss: 0.0071, perc_loss: 7.1304, edge: 0.1596
Epoch: [ 3], Batch: [ 300/1009] | Total Time: 2h 3m 6s
d_loss: 1.3869, g_loss: 30.4294, const_loss: 0.0000, l1_loss: 21.7812, fm_loss: 0.0087, perc_loss: 7.7735, edge: 0.1726
Epoch: [ 3], Batch: [ 400/1009] | Total Time: 2h 6m 47s
d_loss: 1.3867, g_loss: 30.4152, const_loss: 0.0000, l1_loss: 21.6118, fm_loss: 0.0085, perc_loss: 7.9246, edge: 0.1769
Epoch: [ 3], Batch: [ 500/1009] | Total Time: 2h 10m 28s
d_loss: 1.3868, g_loss: 24.7118, const_loss: 0.0000, l1_loss: 17.6776, fm_loss: 0.0064, perc_loss: 6.1923, edge: 0.1421
Epoch: [ 3], Batch: [ 600/1009] | Total Time: 2h 14m 10s
d_loss: 1.3870, g_loss: 27.6480, const_loss: 0.0000, l1_loss: 19.7870, fm_loss: 0.0069, perc_loss: 7.0050, edge: 0.1557
Epoch: [ 3], Batch: [ 700/1009] | Total Time: 2h 17m 50s
d_loss: 1.3872, g_loss: 29.1706, const_loss: 0.0000, l1_loss: 20.8029, fm_loss: 0.0063, perc_loss: 7.4896, edge: 0.1784
Epoch: [ 3], Batch: [ 800/1009] | Total Time: 2h 21m 36s
d_loss: 1.3867, g_loss: 27.0851, const_loss: 0.0000, l1_loss: 19.3615, fm_loss: 0.0058, perc_loss: 6.8649, edge: 0.1594
Epoch: [ 3], Batch: [ 900/1009] | Total Time: 2h 25m 17s
d_loss: 1.3867, g_loss: 29.4218, const_loss: 0.0000, l1_loss: 21.0527, fm_loss: 0.0071, perc_loss: 7.5092, edge: 0.1595
Epoch: [ 3], Batch: [1000/1009] | Total Time: 2h 29m 4s
d_loss: 1.3868, g_loss: 27.5423, const_loss: 0.0000, l1_loss: 19.8119, fm_loss: 0.0063, perc_loss: 6.8660, edge: 0.1648
--- End of Epoch 3 --- Time: 2243.7s ---
LR Scheduler stepped. Current LR G: 0.000185, LR D: 0.000185
Epoch: [ 4], Batch: [ 0/1009] | Total Time: 2h 29m 23s
d_loss: 1.3867, g_loss: 26.1920, const_loss: 0.0000, l1_loss: 18.7728, fm_loss: 0.0068, perc_loss: 6.5619, edge: 0.1571
Epoch: [ 4], Batch: [ 100/1009] | Total Time: 2h 33m 5s
d_loss: 1.3874, g_loss: 27.6825, const_loss: 0.0000, l1_loss: 19.5811, fm_loss: 0.0075, perc_loss: 7.2365, edge: 0.1641
Epoch: [ 4], Batch: [ 200/1009] | Total Time: 2h 36m 46s
d_loss: 1.3868, g_loss: 24.6662, const_loss: 0.0000, l1_loss: 17.5898, fm_loss: 0.0059, perc_loss: 6.2324, edge: 0.1447
Epoch: [ 4], Batch: [ 300/1009] | Total Time: 2h 40m 27s
d_loss: 1.3873, g_loss: 29.2333, const_loss: 0.0000, l1_loss: 20.7596, fm_loss: 0.0071, perc_loss: 7.5919, edge: 0.1814
Epoch: [ 4], Batch: [ 400/1009] | Total Time: 2h 44m 8s
d_loss: 1.3868, g_loss: 28.3472, const_loss: 0.0000, l1_loss: 20.1356, fm_loss: 0.0072, perc_loss: 7.3441, edge: 0.1669
Epoch: [ 4], Batch: [ 500/1009] | Total Time: 2h 47m 50s
d_loss: 1.3867, g_loss: 29.1811, const_loss: 0.0000, l1_loss: 20.5613, fm_loss: 0.0072, perc_loss: 7.7458, edge: 0.1735
Epoch: [ 4], Batch: [ 600/1009] | Total Time: 2h 51m 31s
d_loss: 1.3870, g_loss: 27.2661, const_loss: 0.0000, l1_loss: 19.4570, fm_loss: 0.0063, perc_loss: 6.9491, edge: 0.1603
Epoch: [ 4], Batch: [ 700/1009] | Total Time: 2h 55m 12s
d_loss: 1.3867, g_loss: 30.0584, const_loss: 0.0000, l1_loss: 21.2344, fm_loss: 0.0071, perc_loss: 7.9527, edge: 0.1709
Epoch: [ 4], Batch: [ 800/1009] | Total Time: 2h 58m 53s
d_loss: 1.3867, g_loss: 28.7596, const_loss: 0.0000, l1_loss: 20.2697, fm_loss: 0.0072, perc_loss: 7.6180, edge: 0.1718
Epoch: [ 4], Batch: [ 900/1009] | Total Time: 3h 2m 35s
d_loss: 1.3868, g_loss: 30.3219, const_loss: 0.0000, l1_loss: 21.4957, fm_loss: 0.0071, perc_loss: 7.9548, edge: 0.1709
Epoch: [ 4], Batch: [1000/1009] | Total Time: 3h 6m 16s
d_loss: 1.3867, g_loss: 26.5545, const_loss: 0.0000, l1_loss: 19.0008, fm_loss: 0.0061, perc_loss: 6.7011, edge: 0.1531
--- End of Epoch 4 --- Time: 2232.2s ---
LR Scheduler stepped. Current LR G: 0.000183, LR D: 0.000183
Epoch: [ 5], Batch: [ 0/1009] | Total Time: 3h 6m 35s
d_loss: 1.3869, g_loss: 25.9558, const_loss: 0.0000, l1_loss: 18.2952, fm_loss: 0.0059, perc_loss: 6.8037, edge: 0.1577
Epoch: [ 5], Batch: [ 100/1009] | Total Time: 3h 10m 18s
d_loss: 1.3869, g_loss: 27.9378, const_loss: 0.0000, l1_loss: 20.0886, fm_loss: 0.0065, perc_loss: 6.9790, edge: 0.1702
Epoch: [ 5], Batch: [ 200/1009] | Total Time: 3h 13m 59s
d_loss: 1.3867, g_loss: 25.6503, const_loss: 0.0000, l1_loss: 18.2971, fm_loss: 0.0062, perc_loss: 6.5019, edge: 0.1518
Epoch: [ 5], Batch: [ 300/1009] | Total Time: 3h 17m 40s
d_loss: 1.3868, g_loss: 25.8447, const_loss: 0.0000, l1_loss: 18.3820, fm_loss: 0.0058, perc_loss: 6.6138, edge: 0.1499
Epoch: [ 5], Batch: [ 400/1009] | Total Time: 3h 21m 21s
d_loss: 1.3869, g_loss: 25.8463, const_loss: 0.0000, l1_loss: 18.3924, fm_loss: 0.0065, perc_loss: 6.6053, edge: 0.1488
Epoch: [ 5], Batch: [ 500/1009] | Total Time: 3h 25m 3s
d_loss: 1.3869, g_loss: 27.0186, const_loss: 0.0000, l1_loss: 19.3287, fm_loss: 0.0066, perc_loss: 6.8275, edge: 0.1625
Epoch: [ 5], Batch: [ 600/1009] | Total Time: 3h 28m 44s
d_loss: 1.3868, g_loss: 24.8367, const_loss: 0.0000, l1_loss: 17.7645, fm_loss: 0.0050, perc_loss: 6.2275, edge: 0.1463
Epoch: [ 5], Batch: [ 700/1009] | Total Time: 3h 32m 25s
d_loss: 1.3869, g_loss: 26.6676, const_loss: 0.0000, l1_loss: 19.0766, fm_loss: 0.0059, perc_loss: 6.7345, edge: 0.1572
Epoch: [ 5], Batch: [ 800/1009] | Total Time: 3h 36m 6s
d_loss: 1.3867, g_loss: 25.8846, const_loss: 0.0000, l1_loss: 18.3836, fm_loss: 0.0063, perc_loss: 6.6498, edge: 0.1515
Epoch: [ 5], Batch: [ 900/1009] | Total Time: 3h 39m 49s
d_loss: 1.3867, g_loss: 28.1501, const_loss: 0.0000, l1_loss: 20.1098, fm_loss: 0.0070, perc_loss: 7.1771, edge: 0.1629
Epoch: [ 5], Batch: [1000/1009] | Total Time: 3h 43m 30s
d_loss: 1.3868, g_loss: 29.9697, const_loss: 0.0000, l1_loss: 21.3228, fm_loss: 0.0070, perc_loss: 7.7649, edge: 0.1817
--- End of Epoch 5 --- Time: 2234.3s ---
LR Scheduler stepped. Current LR G: 0.000180, LR D: 0.000180
Epoch: [ 6], Batch: [ 0/1009] | Total Time: 3h 43m 49s
d_loss: 1.3868, g_loss: 27.0016, const_loss: 0.0000, l1_loss: 19.3607, fm_loss: 0.0062, perc_loss: 6.7821, edge: 0.1593
Epoch: [ 6], Batch: [ 100/1009] | Total Time: 3h 47m 31s
d_loss: 1.3868, g_loss: 27.0326, const_loss: 0.0000, l1_loss: 19.3591, fm_loss: 0.0058, perc_loss: 6.8141, edge: 0.1603
Epoch: [ 6], Batch: [ 200/1009] | Total Time: 3h 51m 12s
d_loss: 1.3868, g_loss: 26.7903, const_loss: 0.0000, l1_loss: 19.2709, fm_loss: 0.0059, perc_loss: 6.6624, edge: 0.1576
Epoch: [ 6], Batch: [ 300/1009] | Total Time: 3h 54m 53s
d_loss: 1.3867, g_loss: 28.7907, const_loss: 0.0000, l1_loss: 20.7099, fm_loss: 0.0072, perc_loss: 7.2099, edge: 0.1703
Epoch: [ 6], Batch: [ 400/1009] | Total Time: 3h 58m 40s
d_loss: 1.3868, g_loss: 27.4810, const_loss: 0.0000, l1_loss: 19.6863, fm_loss: 0.0069, perc_loss: 6.9346, edge: 0.1598
Epoch: [ 6], Batch: [ 500/1009] | Total Time: 4h 2m 22s
d_loss: 1.3867, g_loss: 26.0709, const_loss: 0.0000, l1_loss: 18.5806, fm_loss: 0.0067, perc_loss: 6.6352, edge: 0.1551
Epoch: [ 6], Batch: [ 600/1009] | Total Time: 4h 6m 3s
d_loss: 1.3867, g_loss: 27.7621, const_loss: 0.0000, l1_loss: 19.8631, fm_loss: 0.0063, perc_loss: 7.0288, edge: 0.1706
Epoch: [ 6], Batch: [ 700/1009] | Total Time: 4h 9m 45s
d_loss: 1.3868, g_loss: 26.2892, const_loss: 0.0000, l1_loss: 18.5212, fm_loss: 0.0060, perc_loss: 6.9108, edge: 0.1578
Epoch: [ 6], Batch: [ 800/1009] | Total Time: 4h 13m 26s
d_loss: 1.3872, g_loss: 26.2446, const_loss: 0.0000, l1_loss: 18.6828, fm_loss: 0.0067, perc_loss: 6.7054, edge: 0.1563
Epoch: [ 6], Batch: [ 900/1009] | Total Time: 4h 17m 48s
d_loss: 1.3867, g_loss: 26.5457, const_loss: 0.0000, l1_loss: 18.9076, fm_loss: 0.0059, perc_loss: 6.7872, edge: 0.1517
Epoch: [ 6], Batch: [1000/1009] | Total Time: 4h 21m 29s
d_loss: 1.3868, g_loss: 26.4308, const_loss: 0.0000, l1_loss: 18.6193, fm_loss: 0.0056, perc_loss: 6.9545, edge: 0.1581
--- End of Epoch 6 --- Time: 2279.0s ---
LR Scheduler stepped. Current LR G: 0.000176, LR D: 0.000176
Epoch: [ 7], Batch: [ 0/1009] | Total Time: 4h 21m 49s
d_loss: 1.3867, g_loss: 26.4335, const_loss: 0.0000, l1_loss: 18.9572, fm_loss: 0.0055, perc_loss: 6.6257, edge: 0.1516
Epoch: [ 7], Batch: [ 100/1009] | Total Time: 4h 25m 33s
d_loss: 1.3867, g_loss: 26.3623, const_loss: 0.0000, l1_loss: 18.8974, fm_loss: 0.0054, perc_loss: 6.6119, edge: 0.1543
Epoch: [ 7], Batch: [ 200/1009] | Total Time: 4h 29m 15s
d_loss: 1.3868, g_loss: 25.3038, const_loss: 0.0000, l1_loss: 18.0112, fm_loss: 0.0052, perc_loss: 6.4438, edge: 0.1501
from 1156 to 1158, 數據還下降了, 感動.
log:
Epoch: [ 0], Batch: [ 0/1009] | Total Time: 4s
d_loss: 1.3867, g_loss: 27.4468, const_loss: 0.0000, l1_loss: 19.4603, fm_loss: 0.0061, perc_loss: 7.1275, edge: 0.1595
Checkpoint step 100 reached, but saving starts after step 200.
Epoch: [ 0], Batch: [ 100/1009] | Total Time: 3m 32s
d_loss: 1.3869, g_loss: 26.0118, const_loss: 0.0000, l1_loss: 18.4836, fm_loss: 0.0055, perc_loss: 6.6740, edge: 0.1554
Checkpoint saved at step 200
Epoch: [ 0], Batch: [ 200/1009] | Total Time: 7m 14s
d_loss: 1.3867, g_loss: 27.6323, const_loss: 0.0000, l1_loss: 19.5828, fm_loss: 0.0059, perc_loss: 7.1838, edge: 0.1664
Checkpoint saved at step 300
Epoch: [ 0], Batch: [ 300/1009] | Total Time: 10m 52s
d_loss: 1.3868, g_loss: 28.6344, const_loss: 0.0000, l1_loss: 20.5764, fm_loss: 0.0070, perc_loss: 7.1857, edge: 0.1721
Checkpoint saved at step 400
Epoch: [ 0], Batch: [ 400/1009] | Total Time: 14m 30s
d_loss: 1.3868, g_loss: 25.8924, const_loss: 0.0000, l1_loss: 18.4857, fm_loss: 0.0059, perc_loss: 6.5566, edge: 0.1510
Checkpoint saved at step 500
Epoch: [ 0], Batch: [ 500/1009] | Total Time: 18m 9s
d_loss: 1.3870, g_loss: 27.8126, const_loss: 0.0000, l1_loss: 19.8412, fm_loss: 0.0065, perc_loss: 7.1083, edge: 0.1633
Checkpoint saved at step 600
Epoch: [ 0], Batch: [ 600/1009] | Total Time: 21m 49s
d_loss: 1.3867, g_loss: 25.7959, const_loss: 0.0000, l1_loss: 18.4473, fm_loss: 0.0060, perc_loss: 6.4922, edge: 0.1569
Checkpoint saved at step 700
Epoch: [ 0], Batch: [ 700/1009] | Total Time: 25m 27s
d_loss: 1.3867, g_loss: 27.2869, const_loss: 0.0000, l1_loss: 19.5104, fm_loss: 0.0063, perc_loss: 6.9137, edge: 0.1631
Checkpoint saved at step 800
Epoch: [ 0], Batch: [ 800/1009] | Total Time: 29m 5s
d_loss: 1.3872, g_loss: 27.2351, const_loss: 0.0000, l1_loss: 19.5685, fm_loss: 0.0068, perc_loss: 6.8088, edge: 0.1577
Checkpoint saved at step 900
Epoch: [ 0], Batch: [ 900/1009] | Total Time: 32m 47s
d_loss: 1.3868, g_loss: 29.0892, const_loss: 0.0000, l1_loss: 20.7493, fm_loss: 0.0072, perc_loss: 7.4635, edge: 0.1759
Checkpoint saved at step 1000
Epoch: [ 0], Batch: [1000/1009] | Total Time: 36m 25s
d_loss: 1.3869, g_loss: 24.4382, const_loss: 0.0000, l1_loss: 17.4027, fm_loss: 0.0048, perc_loss: 6.1999, edge: 0.1375
--- End of Epoch 0 --- Time: 2202.0s ---
LR Scheduler stepped. Current LR G: 0.000190, LR D: 0.000190
Epoch: [ 1], Batch: [ 0/1009] | Total Time: 36m 44s
d_loss: 1.3867, g_loss: 25.4377, const_loss: 0.0000, l1_loss: 18.1201, fm_loss: 0.0052, perc_loss: 6.4641, edge: 0.1549
Checkpoint saved at step 1100
Epoch: [ 1], Batch: [ 100/1009] | Total Time: 40m 22s
d_loss: 1.3870, g_loss: 27.8938, const_loss: 0.0000, l1_loss: 19.7808, fm_loss: 0.0065, perc_loss: 7.2477, edge: 0.1655
Checkpoint saved at step 1200
Epoch: [ 1], Batch: [ 200/1009] | Total Time: 44m 0s
d_loss: 1.3867, g_loss: 28.4796, const_loss: 0.0000, l1_loss: 20.2706, fm_loss: 0.0059, perc_loss: 7.3381, edge: 0.1715
Checkpoint saved at step 1300
Epoch: [ 1], Batch: [ 300/1009] | Total Time: 47m 44s
d_loss: 1.3871, g_loss: 26.9728, const_loss: 0.0000, l1_loss: 19.4934, fm_loss: 0.0067, perc_loss: 6.6164, edge: 0.1629
Checkpoint saved at step 1400
Epoch: [ 1], Batch: [ 400/1009] | Total Time: 51m 23s
d_loss: 1.3879, g_loss: 25.1016, const_loss: 0.0000, l1_loss: 18.0415, fm_loss: 0.0055, perc_loss: 6.2116, edge: 0.1497
Checkpoint saved at step 1500
Epoch: [ 1], Batch: [ 500/1009] | Total Time: 55m 1s
d_loss: 1.3867, g_loss: 29.2177, const_loss: 0.0000, l1_loss: 20.8823, fm_loss: 0.0065, perc_loss: 7.4543, edge: 0.1812
Checkpoint saved at step 1600
Epoch: [ 1], Batch: [ 600/1009] | Total Time: 58m 39s
d_loss: 1.3868, g_loss: 26.1449, const_loss: 0.0000, l1_loss: 18.8758, fm_loss: 0.0058, perc_loss: 6.4136, edge: 0.1563
Checkpoint saved at step 1700
Epoch: [ 1], Batch: [ 700/1009] | Total Time: 1h 2m 19s
d_loss: 1.3867, g_loss: 29.8010, const_loss: 0.0000, l1_loss: 21.1752, fm_loss: 0.0069, perc_loss: 7.7447, edge: 0.1808
Checkpoint saved at step 1800
Epoch: [ 1], Batch: [ 800/1009] | Total Time: 1h 5m 57s
d_loss: 1.3867, g_loss: 23.6851, const_loss: 0.0000, l1_loss: 17.1872, fm_loss: 0.0052, perc_loss: 5.6700, edge: 0.1294
Checkpoint saved at step 1900
Epoch: [ 1], Batch: [ 900/1009] | Total Time: 1h 9m 35s
d_loss: 1.3867, g_loss: 28.4856, const_loss: 0.0000, l1_loss: 20.2635, fm_loss: 0.0065, perc_loss: 7.3490, edge: 0.1733
Checkpoint saved at step 2000
Epoch: [ 1], Batch: [1000/1009] | Total Time: 1h 13m 14s
d_loss: 1.3867, g_loss: 24.5244, const_loss: 0.0000, l1_loss: 17.3751, fm_loss: 0.0056, perc_loss: 6.3054, edge: 0.1449
--- End of Epoch 1 --- Time: 2208.9s ---
LR Scheduler stepped. Current LR G: 0.000189, LR D: 0.000189
Epoch: [ 2], Batch: [ 0/1009] | Total Time: 1h 13m 33s
d_loss: 1.3867, g_loss: 26.6472, const_loss: 0.0000, l1_loss: 19.2029, fm_loss: 0.0062, perc_loss: 6.5874, edge: 0.1573
Checkpoint saved at step 2100
Epoch: [ 2], Batch: [ 100/1009] | Total Time: 1h 17m 11s
d_loss: 1.3868, g_loss: 26.7355, const_loss: 0.0000, l1_loss: 19.1291, fm_loss: 0.0063, perc_loss: 6.7501, edge: 0.1566
Checkpoint saved at step 2200
Epoch: [ 2], Batch: [ 200/1009] | Total Time: 1h 20m 51s
d_loss: 1.3867, g_loss: 28.8581, const_loss: 0.0000, l1_loss: 20.6037, fm_loss: 0.0065, perc_loss: 7.3799, edge: 0.1747
Checkpoint saved at step 2300
Epoch: [ 2], Batch: [ 300/1009] | Total Time: 1h 24m 29s
d_loss: 1.3868, g_loss: 27.5763, const_loss: 0.0000, l1_loss: 19.8683, fm_loss: 0.0059, perc_loss: 6.8524, edge: 0.1563
Checkpoint saved at step 2400
Epoch: [ 2], Batch: [ 400/1009] | Total Time: 1h 28m 8s
d_loss: 1.3868, g_loss: 28.7304, const_loss: 0.0000, l1_loss: 20.3308, fm_loss: 0.0063, perc_loss: 7.5292, edge: 0.1707
Checkpoint saved at step 2500
Epoch: [ 2], Batch: [ 500/1009] | Total Time: 1h 31m 52s
d_loss: 1.3867, g_loss: 25.5138, const_loss: 0.0000, l1_loss: 18.2073, fm_loss: 0.0053, perc_loss: 6.4543, edge: 0.1535
Checkpoint saved at step 2600
Epoch: [ 2], Batch: [ 600/1009] | Total Time: 1h 35m 31s
d_loss: 1.3867, g_loss: 26.8008, const_loss: 0.0000, l1_loss: 19.4062, fm_loss: 0.0063, perc_loss: 6.5386, edge: 0.1563
Checkpoint saved at step 2700
Epoch: [ 2], Batch: [ 700/1009] | Total Time: 1h 39m 14s
d_loss: 1.3868, g_loss: 28.1078, const_loss: 0.0000, l1_loss: 20.1147, fm_loss: 0.0057, perc_loss: 7.1275, edge: 0.1666
Checkpoint saved at step 2800
Epoch: [ 2], Batch: [ 800/1009] | Total Time: 1h 42m 52s
d_loss: 1.3868, g_loss: 25.1592, const_loss: 0.0000, l1_loss: 17.9936, fm_loss: 0.0049, perc_loss: 6.3095, edge: 0.1579
Checkpoint saved at step 2900
Epoch: [ 2], Batch: [ 900/1009] | Total Time: 1h 46m 31s
d_loss: 1.3868, g_loss: 25.1547, const_loss: 0.0000, l1_loss: 18.0206, fm_loss: 0.0057, perc_loss: 6.2805, edge: 0.1551
Checkpoint saved at step 3000
Epoch: [ 2], Batch: [1000/1009] | Total Time: 1h 50m 9s
d_loss: 1.3867, g_loss: 27.5919, const_loss: 0.0000, l1_loss: 19.6670, fm_loss: 0.0058, perc_loss: 7.0602, edge: 0.1655
--- End of Epoch 2 --- Time: 2214.9s ---
LR Scheduler stepped. Current LR G: 0.000187, LR D: 0.000187
Epoch: [ 3], Batch: [ 0/1009] | Total Time: 1h 50m 27s
d_loss: 1.3868, g_loss: 27.2271, const_loss: 0.0000, l1_loss: 19.3584, fm_loss: 0.0056, perc_loss: 7.0062, edge: 0.1635
Checkpoint saved at step 3100
Epoch: [ 3], Batch: [ 100/1009] | Total Time: 1h 54m 10s
d_loss: 1.3868, g_loss: 25.5382, const_loss: 0.0000, l1_loss: 18.1504, fm_loss: 0.0058, perc_loss: 6.5283, edge: 0.1603
Checkpoint saved at step 3200
Epoch: [ 3], Batch: [ 200/1009] | Total Time: 1h 57m 49s
d_loss: 1.3868, g_loss: 24.9940, const_loss: 0.0000, l1_loss: 17.8663, fm_loss: 0.0049, perc_loss: 6.2795, edge: 0.1500
Checkpoint saved at step 3300
Epoch: [ 3], Batch: [ 300/1009] | Total Time: 2h 1m 27s
d_loss: 1.3868, g_loss: 27.6818, const_loss: 0.0000, l1_loss: 19.8678, fm_loss: 0.0061, perc_loss: 6.9518, edge: 0.1627
Checkpoint saved at step 3400
Epoch: [ 3], Batch: [ 400/1009] | Total Time: 2h 5m 5s
d_loss: 1.3868, g_loss: 27.5276, const_loss: 0.0000, l1_loss: 19.6712, fm_loss: 0.0057, perc_loss: 6.9935, edge: 0.1639
Checkpoint saved at step 3500
Epoch: [ 3], Batch: [ 500/1009] | Total Time: 2h 8m 43s
d_loss: 1.3867, g_loss: 23.2102, const_loss: 0.0000, l1_loss: 16.6920, fm_loss: 0.0049, perc_loss: 5.6766, edge: 0.1433
Checkpoint saved at step 3600
Epoch: [ 3], Batch: [ 600/1009] | Total Time: 2h 12m 21s
d_loss: 1.3869, g_loss: 25.9864, const_loss: 0.0000, l1_loss: 18.6269, fm_loss: 0.0051, perc_loss: 6.5096, edge: 0.1515
Checkpoint saved at step 3700
Epoch: [ 3], Batch: [ 700/1009] | Total Time: 2h 16m 1s
d_loss: 1.3873, g_loss: 27.4017, const_loss: 0.0000, l1_loss: 19.6077, fm_loss: 0.0055, perc_loss: 6.9278, edge: 0.1673
Checkpoint saved at step 3800
Epoch: [ 3], Batch: [ 800/1009] | Total Time: 2h 19m 39s
d_loss: 1.3867, g_loss: 25.0102, const_loss: 0.0000, l1_loss: 17.9769, fm_loss: 0.0048, perc_loss: 6.1759, edge: 0.1592
Checkpoint saved at step 3900
Epoch: [ 3], Batch: [ 900/1009] | Total Time: 2h 23m 21s
d_loss: 1.3868, g_loss: 26.4315, const_loss: 0.0000, l1_loss: 19.0407, fm_loss: 0.0057, perc_loss: 6.5392, edge: 0.1526
Checkpoint saved at step 4000
Epoch: [ 3], Batch: [1000/1009] | Total Time: 2h 27m 0s
d_loss: 1.3867, g_loss: 26.2646, const_loss: 0.0000, l1_loss: 18.9668, fm_loss: 0.0052, perc_loss: 6.4426, edge: 0.1566
--- End of Epoch 3 --- Time: 2211.2s ---
LR Scheduler stepped. Current LR G: 0.000185, LR D: 0.000185
Epoch: [ 4], Batch: [ 0/1009] | Total Time: 2h 27m 19s
d_loss: 1.3867, g_loss: 24.8697, const_loss: 0.0000, l1_loss: 17.9426, fm_loss: 0.0056, perc_loss: 6.0806, edge: 0.1475
Checkpoint saved at step 4100
Epoch: [ 4], Batch: [ 100/1009] | Total Time: 2h 31m 0s
d_loss: 1.3873, g_loss: 25.0928, const_loss: 0.0000, l1_loss: 17.9985, fm_loss: 0.0052, perc_loss: 6.2492, edge: 0.1465
Checkpoint saved at step 4200
Epoch: [ 4], Batch: [ 200/1009] | Total Time: 2h 34m 38s
d_loss: 1.3867, g_loss: 23.3064, const_loss: 0.0000, l1_loss: 16.7801, fm_loss: 0.0050, perc_loss: 5.6932, edge: 0.1347
Checkpoint saved at step 4300
Epoch: [ 4], Batch: [ 300/1009] | Total Time: 2h 38m 17s
d_loss: 1.3871, g_loss: 26.8814, const_loss: 0.0000, l1_loss: 19.2733, fm_loss: 0.0057, perc_loss: 6.7456, edge: 0.1634
Checkpoint saved at step 4400
Epoch: [ 4], Batch: [ 400/1009] | Total Time: 2h 42m 0s
d_loss: 1.3870, g_loss: 27.5083, const_loss: 0.0000, l1_loss: 19.6014, fm_loss: 0.0061, perc_loss: 7.0413, edge: 0.1662
Checkpoint saved at step 4500
Epoch: [ 4], Batch: [ 500/1009] | Total Time: 2h 45m 47s
d_loss: 1.3868, g_loss: 26.1615, const_loss: 0.0000, l1_loss: 18.6360, fm_loss: 0.0056, perc_loss: 6.6804, edge: 0.1462
Checkpoint saved at step 4600
Epoch: [ 4], Batch: [ 600/1009] | Total Time: 2h 49m 33s
d_loss: 1.3869, g_loss: 24.6412, const_loss: 0.0000, l1_loss: 17.7397, fm_loss: 0.0053, perc_loss: 6.0594, edge: 0.1435
Checkpoint saved at step 4700
Epoch: [ 4], Batch: [ 700/1009] | Total Time: 2h 53m 12s
d_loss: 1.3867, g_loss: 27.1844, const_loss: 0.0000, l1_loss: 19.2991, fm_loss: 0.0055, perc_loss: 7.0269, edge: 0.1595
Checkpoint saved at step 4800
Epoch: [ 4], Batch: [ 800/1009] | Total Time: 2h 56m 51s
d_loss: 1.3867, g_loss: 27.0156, const_loss: 0.0000, l1_loss: 19.1828, fm_loss: 0.0060, perc_loss: 6.9711, edge: 0.1628
Checkpoint saved at step 4900
Epoch: [ 4], Batch: [ 900/1009] | Total Time: 3h 30s
d_loss: 1.3868, g_loss: 27.8600, const_loss: 0.0000, l1_loss: 19.7286, fm_loss: 0.0054, perc_loss: 7.2718, edge: 0.1607
Checkpoint saved at step 5000
Epoch: [ 4], Batch: [1000/1009] | Total Time: 3h 4m 8s
d_loss: 1.3868, g_loss: 24.1605, const_loss: 0.0000, l1_loss: 17.3423, fm_loss: 0.0049, perc_loss: 5.9795, edge: 0.1404
--- End of Epoch 4 --- Time: 2228.7s ---
LR Scheduler stepped. Current LR G: 0.000183, LR D: 0.000183
Epoch: [ 5], Batch: [ 0/1009] | Total Time: 3h 4m 27s
d_loss: 1.3868, g_loss: 24.6256, const_loss: 0.0000, l1_loss: 17.4912, fm_loss: 0.0052, perc_loss: 6.2934, edge: 0.1424
Checkpoint saved at step 5100
Epoch: [ 5], Batch: [ 100/1009] | Total Time: 3h 8m 5s
d_loss: 1.3868, g_loss: 27.2796, const_loss: 0.0000, l1_loss: 19.7554, fm_loss: 0.0062, perc_loss: 6.6581, edge: 0.1666
Checkpoint saved at step 5200
Epoch: [ 5], Batch: [ 200/1009] | Total Time: 3h 11m 46s
d_loss: 1.3867, g_loss: 25.2086, const_loss: 0.0000, l1_loss: 18.0151, fm_loss: 0.0056, perc_loss: 6.3390, edge: 0.1555
Checkpoint saved at step 5300
Epoch: [ 5], Batch: [ 300/1009] | Total Time: 3h 15m 24s
d_loss: 1.3868, g_loss: 23.7966, const_loss: 0.0000, l1_loss: 17.0142, fm_loss: 0.0049, perc_loss: 5.9445, edge: 0.1397
Checkpoint saved at step 5400
Epoch: [ 5], Batch: [ 400/1009] | Total Time: 3h 19m 2s
d_loss: 1.3868, g_loss: 23.9481, const_loss: 0.0000, l1_loss: 17.1553, fm_loss: 0.0055, perc_loss: 5.9527, edge: 0.1412
Checkpoint saved at step 5500
Epoch: [ 5], Batch: [ 500/1009] | Total Time: 3h 22m 44s
d_loss: 1.3868, g_loss: 24.6401, const_loss: 0.0000, l1_loss: 17.8327, fm_loss: 0.0049, perc_loss: 5.9649, edge: 0.1442
Checkpoint saved at step 5600
Epoch: [ 5], Batch: [ 600/1009] | Total Time: 3h 26m 27s
d_loss: 1.3868, g_loss: 24.0142, const_loss: 0.0000, l1_loss: 17.2575, fm_loss: 0.0045, perc_loss: 5.9171, edge: 0.1418
Checkpoint saved at step 5700
Epoch: [ 5], Batch: [ 700/1009] | Total Time: 3h 30m 5s
d_loss: 1.3870, g_loss: 24.8988, const_loss: 0.0000, l1_loss: 17.9521, fm_loss: 0.0053, perc_loss: 6.1006, edge: 0.1475
Checkpoint saved at step 5800
Epoch: [ 5], Batch: [ 800/1009] | Total Time: 3h 33m 43s
d_loss: 1.3867, g_loss: 24.6327, const_loss: 0.0000, l1_loss: 17.5678, fm_loss: 0.0054, perc_loss: 6.2193, edge: 0.1468
Checkpoint saved at step 5900
Epoch: [ 5], Batch: [ 900/1009] | Total Time: 3h 37m 24s
d_loss: 1.3867, g_loss: 26.1928, const_loss: 0.0000, l1_loss: 18.7654, fm_loss: 0.0057, perc_loss: 6.5712, edge: 0.1572
Checkpoint saved at step 6000
Epoch: [ 5], Batch: [1000/1009] | Total Time: 3h 41m 7s
d_loss: 1.3868, g_loss: 26.9755, const_loss: 0.0000, l1_loss: 19.4256, fm_loss: 0.0053, perc_loss: 6.6837, edge: 0.1674
--- End of Epoch 5 --- Time: 2218.9s ---
LR Scheduler stepped. Current LR G: 0.000180, LR D: 0.000180
Epoch: [ 6], Batch: [ 0/1009] | Total Time: 3h 41m 26s
d_loss: 1.3867, g_loss: 25.8302, const_loss: 0.0000, l1_loss: 18.5985, fm_loss: 0.0049, perc_loss: 6.3772, edge: 0.1562
Checkpoint saved at step 6100
Epoch: [ 6], Batch: [ 100/1009] | Total Time: 3h 45m 4s
d_loss: 1.3867, g_loss: 24.5542, const_loss: 0.0000, l1_loss: 17.7908, fm_loss: 0.0048, perc_loss: 5.9185, edge: 0.1467
Checkpoint saved at step 6200
Epoch: [ 6], Batch: [ 200/1009] | Total Time: 3h 48m 42s
d_loss: 1.3868, g_loss: 25.1046, const_loss: 0.0000, l1_loss: 18.2424, fm_loss: 0.0050, perc_loss: 6.0237, edge: 0.1402
Checkpoint saved at step 6300
Epoch: [ 6], Batch: [ 300/1009] | Total Time: 3h 52m 25s
d_loss: 1.3867, g_loss: 26.1998, const_loss: 0.0000, l1_loss: 18.9886, fm_loss: 0.0054, perc_loss: 6.3564, edge: 0.1560
Checkpoint saved at step 6400
Epoch: [ 6], Batch: [ 400/1009] | Total Time: 3h 56m 3s
d_loss: 1.3867, g_loss: 25.4709, const_loss: 0.0000, l1_loss: 18.4774, fm_loss: 0.0052, perc_loss: 6.1472, edge: 0.1477
Checkpoint saved at step 6500
Epoch: [ 6], Batch: [ 500/1009] | Total Time: 3h 59m 45s
d_loss: 1.3867, g_loss: 24.1072, const_loss: 0.0000, l1_loss: 17.3882, fm_loss: 0.0048, perc_loss: 5.8741, edge: 0.1467
Checkpoint saved at step 6600
Epoch: [ 6], Batch: [ 600/1009] | Total Time: 4h 3m 26s
d_loss: 1.3867, g_loss: 25.9461, const_loss: 0.0000, l1_loss: 18.7469, fm_loss: 0.0050, perc_loss: 6.3451, edge: 0.1557
Checkpoint saved at step 6700
Epoch: [ 6], Batch: [ 700/1009] | Total Time: 4h 7m 9s
d_loss: 1.3868, g_loss: 25.5958, const_loss: 0.0000, l1_loss: 18.1268, fm_loss: 0.0052, perc_loss: 6.6175, edge: 0.1529
Checkpoint saved at step 6800
Epoch: [ 6], Batch: [ 800/1009] | Total Time: 4h 10m 52s
d_loss: 1.3870, g_loss: 24.7392, const_loss: 0.0000, l1_loss: 17.7183, fm_loss: 0.0053, perc_loss: 6.1796, edge: 0.1427
Checkpoint saved at step 6900
Epoch: [ 6], Batch: [ 900/1009] | Total Time: 4h 14m 36s
d_loss: 1.3868, g_loss: 24.2026, const_loss: 0.0000, l1_loss: 17.3675, fm_loss: 0.0046, perc_loss: 6.0009, edge: 0.1363
Checkpoint saved at step 7000
Epoch: [ 6], Batch: [1000/1009] | Total Time: 4h 18m 14s
d_loss: 1.3868, g_loss: 24.8848, const_loss: 0.0000, l1_loss: 17.6767, fm_loss: 0.0045, perc_loss: 6.3608, edge: 0.1494
--- End of Epoch 6 --- Time: 2226.2s ---
LR Scheduler stepped. Current LR G: 0.000176, LR D: 0.000176
Epoch: [ 7], Batch: [ 0/1009] | Total Time: 4h 18m 32s
d_loss: 1.3868, g_loss: 25.3357, const_loss: 0.0000, l1_loss: 18.2289, fm_loss: 0.0046, perc_loss: 6.2517, edge: 0.1571
Checkpoint saved at step 7100
Epoch: [ 7], Batch: [ 100/1009] | Total Time: 4h 22m 16s
d_loss: 1.3867, g_loss: 25.1645, const_loss: 0.0000, l1_loss: 18.1652, fm_loss: 0.0045, perc_loss: 6.1519, edge: 0.1496
Checkpoint saved at step 7200
Epoch: [ 7], Batch: [ 200/1009] | Total Time: 4h 25m 54s
d_loss: 1.3868, g_loss: 24.6297, const_loss: 0.0000, l1_loss: 17.5838, fm_loss: 0.0045, perc_loss: 6.2022, edge: 0.1458
Checkpoint saved at step 7300
Epoch: [ 7], Batch: [ 300/1009] | Total Time: 4h 29m 35s
d_loss: 1.3870, g_loss: 24.2667, const_loss: 0.0000, l1_loss: 17.3304, fm_loss: 0.0053, perc_loss: 6.0914, edge: 0.1462
Checkpoint saved at step 7400
Epoch: [ 7], Batch: [ 400/1009] | Total Time: 4h 33m 13s
d_loss: 1.3871, g_loss: 26.0116, const_loss: 0.0000, l1_loss: 18.6272, fm_loss: 0.0054, perc_loss: 6.5268, edge: 0.1588
Checkpoint saved at step 7500
Epoch: [ 7], Batch: [ 500/1009] | Total Time: 4h 36m 51s
d_loss: 1.3867, g_loss: 24.8220, const_loss: 0.0000, l1_loss: 17.9253, fm_loss: 0.0051, perc_loss: 6.0462, edge: 0.1521
Checkpoint saved at step 7600
Epoch: [ 7], Batch: [ 600/1009] | Total Time: 4h 40m 31s
d_loss: 1.3868, g_loss: 24.7958, const_loss: 0.0000, l1_loss: 17.7864, fm_loss: 0.0047, perc_loss: 6.1582, edge: 0.1537
Checkpoint saved at step 7700
Epoch: [ 7], Batch: [ 700/1009] | Total Time: 4h 44m 9s
d_loss: 1.3868, g_loss: 26.4558, const_loss: 0.0000, l1_loss: 18.9894, fm_loss: 0.0057, perc_loss: 6.6074, edge: 0.1600
Checkpoint saved at step 7800
Epoch: [ 7], Batch: [ 800/1009] | Total Time: 4h 47m 53s
d_loss: 1.3867, g_loss: 25.9501, const_loss: 0.0000, l1_loss: 18.4271, fm_loss: 0.0049, perc_loss: 6.6655, edge: 0.1592