隨手記錄log, 512×512 與 384×384 相比, 訓練時間拉長約 2.28倍, 首輪 log 數據比較有感, 差異比較大.
figure:
log:
Epoch: [ 0], Batch: [ 0/ 831] | Total Time: 3s
d_loss: 1.8555, g_loss: 48.5514, const_loss: 0.0003, l1_loss: 35.8118, fm_loss: 0.0314, perc_loss: 11.8152, edge: 0.1999
Checkpoint step 100 reached, but saving starts after step 200.
Epoch: [ 0], Batch: [ 100/ 831] | Total Time: 3m 42s
d_loss: 1.3870, g_loss: 41.4099, const_loss: 0.0001, l1_loss: 30.0919, fm_loss: 0.0299, perc_loss: 10.3960, edge: 0.1991
Epoch: [ 0], Batch: [ 200/ 831] | Total Time: 7m 29s
d_loss: 1.3868, g_loss: 36.1580, const_loss: 0.0001, l1_loss: 25.9894, fm_loss: 0.0238, perc_loss: 9.2647, edge: 0.1872
Epoch: [ 0], Batch: [ 300/ 831] | Total Time: 11m 14s
d_loss: 1.3878, g_loss: 39.6941, const_loss: 0.0001, l1_loss: 28.1615, fm_loss: 0.0257, perc_loss: 10.6019, edge: 0.2120
Epoch: [ 0], Batch: [ 400/ 831] | Total Time: 14m 55s
d_loss: 1.3868, g_loss: 38.5238, const_loss: 0.0001, l1_loss: 27.4452, fm_loss: 0.0246, perc_loss: 10.1615, edge: 0.1991
Epoch: [ 0], Batch: [ 500/ 831] | Total Time: 18m 36s
d_loss: 1.3868, g_loss: 38.4518, const_loss: 0.0002, l1_loss: 27.4393, fm_loss: 0.0242, perc_loss: 10.1010, edge: 0.1942
Epoch: [ 0], Batch: [ 600/ 831] | Total Time: 22m 17s
d_loss: 1.3872, g_loss: 37.0487, const_loss: 0.0001, l1_loss: 26.8958, fm_loss: 0.0240, perc_loss: 9.2513, edge: 0.1846
Epoch: [ 0], Batch: [ 700/ 831] | Total Time: 25m 58s
d_loss: 1.3867, g_loss: 38.7305, const_loss: 0.0001, l1_loss: 27.9832, fm_loss: 0.0224, perc_loss: 9.8449, edge: 0.1871
Epoch: [ 0], Batch: [ 800/ 831] | Total Time: 29m 39s
d_loss: 1.3868, g_loss: 37.9838, const_loss: 0.0004, l1_loss: 28.0558, fm_loss: 0.0239, perc_loss: 9.0325, edge: 0.1784
--- End of Epoch 0 --- Time: 1843.7s ---
LR Scheduler stepped. Current LR G: 0.000200, LR D: 0.000200
Epoch: [ 1], Batch: [ 0/ 831] | Total Time: 30m 45s
d_loss: 1.3882, g_loss: 36.0948, const_loss: 0.0002, l1_loss: 25.8642, fm_loss: 0.0187, perc_loss: 9.3354, edge: 0.1831
Epoch: [ 1], Batch: [ 100/ 831] | Total Time: 34m 30s
d_loss: 1.3868, g_loss: 36.8987, const_loss: 0.0002, l1_loss: 26.4981, fm_loss: 0.0205, perc_loss: 9.4850, edge: 0.2014
Epoch: [ 1], Batch: [ 200/ 831] | Total Time: 38m 10s
d_loss: 1.3867, g_loss: 36.2823, const_loss: 0.0001, l1_loss: 26.2852, fm_loss: 0.0202, perc_loss: 9.0958, edge: 0.1881
Epoch: [ 1], Batch: [ 300/ 831] | Total Time: 41m 52s
d_loss: 1.3868, g_loss: 39.1493, const_loss: 0.0001, l1_loss: 27.9802, fm_loss: 0.0206, perc_loss: 10.2437, edge: 0.2113
Epoch: [ 1], Batch: [ 400/ 831] | Total Time: 45m 33s
d_loss: 1.3869, g_loss: 35.6709, const_loss: 0.0001, l1_loss: 25.6619, fm_loss: 0.0182, perc_loss: 9.1091, edge: 0.1886
Epoch: [ 1], Batch: [ 500/ 831] | Total Time: 49m 19s
d_loss: 1.3872, g_loss: 33.1971, const_loss: 0.0002, l1_loss: 24.2608, fm_loss: 0.0166, perc_loss: 8.0567, edge: 0.1694
Epoch: [ 1], Batch: [ 600/ 831] | Total Time: 52m 59s
d_loss: 1.3867, g_loss: 37.4253, const_loss: 0.0002, l1_loss: 26.8524, fm_loss: 0.0191, perc_loss: 9.6647, edge: 0.1960
Epoch: [ 1], Batch: [ 700/ 831] | Total Time: 56m 40s
d_loss: 1.3870, g_loss: 37.0443, const_loss: 0.0001, l1_loss: 26.4690, fm_loss: 0.0181, perc_loss: 9.6677, edge: 0.1959
Epoch: [ 1], Batch: [ 800/ 831] | Total Time: 1h 24s
d_loss: 1.3868, g_loss: 36.6171, const_loss: 0.0001, l1_loss: 26.7109, fm_loss: 0.0185, perc_loss: 9.0182, edge: 0.1766
--- End of Epoch 1 --- Time: 1844.9s ---
LR Scheduler stepped. Current LR G: 0.000199, LR D: 0.000199
Epoch: [ 2], Batch: [ 0/ 831] | Total Time: 1h 1m 30s
d_loss: 1.3869, g_loss: 33.3594, const_loss: 0.0001, l1_loss: 23.8799, fm_loss: 0.0157, perc_loss: 8.5950, edge: 0.1758
Epoch: [ 2], Batch: [ 100/ 831] | Total Time: 1h 5m 15s
d_loss: 1.3867, g_loss: 32.8736, const_loss: 0.0001, l1_loss: 23.7862, fm_loss: 0.0139, perc_loss: 8.2090, edge: 0.1712
Epoch: [ 2], Batch: [ 200/ 831] | Total Time: 1h 8m 57s
d_loss: 1.3869, g_loss: 35.8764, const_loss: 0.0001, l1_loss: 25.9877, fm_loss: 0.0159, perc_loss: 8.9948, edge: 0.1847
Epoch: [ 2], Batch: [ 300/ 831] | Total Time: 1h 12m 37s
d_loss: 1.3873, g_loss: 27.9786, const_loss: 0.0003, l1_loss: 20.1063, fm_loss: 0.0107, perc_loss: 7.0191, edge: 0.1487
Epoch: [ 2], Batch: [ 400/ 831] | Total Time: 1h 16m 18s
d_loss: 1.3868, g_loss: 32.5896, const_loss: 0.0002, l1_loss: 22.9733, fm_loss: 0.0131, perc_loss: 8.7263, edge: 0.1835
Epoch: [ 2], Batch: [ 500/ 831] | Total Time: 1h 19m 59s
d_loss: 1.3876, g_loss: 31.8721, const_loss: 0.0001, l1_loss: 22.6266, fm_loss: 0.0123, perc_loss: 8.3598, edge: 0.1800
Epoch: [ 2], Batch: [ 600/ 831] | Total Time: 1h 23m 39s
d_loss: 1.3867, g_loss: 32.4699, const_loss: 0.0001, l1_loss: 23.2523, fm_loss: 0.0126, perc_loss: 8.3410, edge: 0.1706
Epoch: [ 2], Batch: [ 700/ 831] | Total Time: 1h 27m 25s
d_loss: 1.3867, g_loss: 31.5842, const_loss: 0.0001, l1_loss: 22.5872, fm_loss: 0.0118, perc_loss: 8.1213, edge: 0.1706
Epoch: [ 2], Batch: [ 800/ 831] | Total Time: 1h 31m 7s
d_loss: 1.3872, g_loss: 36.3781, const_loss: 0.0001, l1_loss: 25.8096, fm_loss: 0.0141, perc_loss: 9.6648, edge: 0.1961
--- End of Epoch 2 --- Time: 1843.3s ---
LR Scheduler stepped. Current LR G: 0.000197, LR D: 0.000197
Epoch: [ 3], Batch: [ 0/ 831] | Total Time: 1h 32m 14s
d_loss: 1.3867, g_loss: 32.6280, const_loss: 0.0001, l1_loss: 23.3588, fm_loss: 0.0116, perc_loss: 8.3858, edge: 0.1783
Epoch: [ 3], Batch: [ 100/ 831] | Total Time: 1h 35m 54s
d_loss: 1.3867, g_loss: 36.0779, const_loss: 0.0001, l1_loss: 26.0461, fm_loss: 0.0137, perc_loss: 9.1326, edge: 0.1921
Epoch: [ 3], Batch: [ 200/ 831] | Total Time: 1h 39m 34s
d_loss: 1.3869, g_loss: 32.3989, const_loss: 0.0002, l1_loss: 23.0922, fm_loss: 0.0122, perc_loss: 8.4207, edge: 0.1802
Epoch: [ 3], Batch: [ 300/ 831] | Total Time: 1h 43m 19s
d_loss: 1.3870, g_loss: 32.2563, const_loss: 0.0001, l1_loss: 23.1509, fm_loss: 0.0112, perc_loss: 8.2281, edge: 0.1727
Epoch: [ 3], Batch: [ 400/ 831] | Total Time: 1h 46m 59s
d_loss: 1.3875, g_loss: 32.0288, const_loss: 0.0001, l1_loss: 22.8382, fm_loss: 0.0105, perc_loss: 8.3199, edge: 0.1668
Epoch: [ 3], Batch: [ 500/ 831] | Total Time: 1h 50m 42s
d_loss: 1.3869, g_loss: 33.7018, const_loss: 0.0001, l1_loss: 24.1624, fm_loss: 0.0124, perc_loss: 8.6473, edge: 0.1863
Epoch: [ 3], Batch: [ 600/ 831] | Total Time: 1h 54m 23s
d_loss: 1.3869, g_loss: 35.0490, const_loss: 0.0002, l1_loss: 25.0036, fm_loss: 0.0114, perc_loss: 9.1460, edge: 0.1944
Epoch: [ 3], Batch: [ 700/ 831] | Total Time: 1h 58m 4s
d_loss: 1.3867, g_loss: 30.9307, const_loss: 0.0001, l1_loss: 22.2519, fm_loss: 0.0096, perc_loss: 7.8038, edge: 0.1720
Epoch: [ 3], Batch: [ 800/ 831] | Total Time: 2h 1m 45s
d_loss: 1.3874, g_loss: 31.3321, const_loss: 0.0001, l1_loss: 22.3879, fm_loss: 0.0106, perc_loss: 8.0673, edge: 0.1729
--- End of Epoch 3 --- Time: 1840.6s ---
LR Scheduler stepped. Current LR G: 0.000195, LR D: 0.000195
Epoch: [ 4], Batch: [ 0/ 831] | Total Time: 2h 2m 54s
d_loss: 1.3867, g_loss: 31.1261, const_loss: 0.0001, l1_loss: 22.3814, fm_loss: 0.0110, perc_loss: 7.8735, edge: 0.1668
Epoch: [ 4], Batch: [ 100/ 831] | Total Time: 2h 6m 35s
d_loss: 1.3867, g_loss: 34.3805, const_loss: 0.0001, l1_loss: 24.8031, fm_loss: 0.0119, perc_loss: 8.6855, edge: 0.1865
Epoch: [ 4], Batch: [ 200/ 831] | Total Time: 2h 10m 15s
d_loss: 1.3868, g_loss: 32.0789, const_loss: 0.0001, l1_loss: 22.9830, fm_loss: 0.0107, perc_loss: 8.2176, edge: 0.1741
Epoch: [ 4], Batch: [ 300/ 831] | Total Time: 2h 13m 56s
d_loss: 1.3873, g_loss: 34.2499, const_loss: 0.0001, l1_loss: 24.5459, fm_loss: 0.0112, perc_loss: 8.8121, edge: 0.1872
Epoch: [ 4], Batch: [ 400/ 831] | Total Time: 2h 17m 43s
d_loss: 1.3887, g_loss: 34.3537, const_loss: 0.0001, l1_loss: 24.6973, fm_loss: 0.0113, perc_loss: 8.7695, edge: 0.1828
Epoch: [ 4], Batch: [ 500/ 831] | Total Time: 2h 21m 28s
d_loss: 1.3867, g_loss: 30.5413, const_loss: 0.0001, l1_loss: 22.1016, fm_loss: 0.0096, perc_loss: 7.5840, edge: 0.1527
Epoch: [ 4], Batch: [ 600/ 831] | Total Time: 2h 25m 8s
d_loss: 1.3868, g_loss: 34.3100, const_loss: 0.0001, l1_loss: 24.8765, fm_loss: 0.0110, perc_loss: 8.5498, edge: 0.1793
Epoch: [ 4], Batch: [ 700/ 831] | Total Time: 2h 28m 49s
d_loss: 1.3870, g_loss: 37.7089, const_loss: 0.0000, l1_loss: 26.7227, fm_loss: 0.0137, perc_loss: 10.0693, edge: 0.2098
Epoch: [ 4], Batch: [ 800/ 831] | Total Time: 2h 32m 29s
d_loss: 1.3869, g_loss: 33.5865, const_loss: 0.0001, l1_loss: 24.3283, fm_loss: 0.0126, perc_loss: 8.3819, edge: 0.1707
--- End of Epoch 4 --- Time: 1841.9s ---
LR Scheduler stepped. Current LR G: 0.000192, LR D: 0.000192
Epoch: [ 5], Batch: [ 0/ 831] | Total Time: 2h 33m 36s
d_loss: 1.3873, g_loss: 31.7718, const_loss: 0.0001, l1_loss: 22.7510, fm_loss: 0.0100, perc_loss: 8.1445, edge: 0.1729
Epoch: [ 5], Batch: [ 100/ 831] | Total Time: 2h 37m 17s
d_loss: 1.3868, g_loss: 29.6126, const_loss: 0.0001, l1_loss: 21.1932, fm_loss: 0.0093, perc_loss: 7.5534, edge: 0.1632
Epoch: [ 5], Batch: [ 200/ 831] | Total Time: 2h 40m 58s
d_loss: 1.3867, g_loss: 28.3905, const_loss: 0.0000, l1_loss: 20.3083, fm_loss: 0.0085, perc_loss: 7.2206, edge: 0.1598
Epoch: [ 5], Batch: [ 300/ 831] | Total Time: 2h 44m 40s
d_loss: 1.3870, g_loss: 28.6722, const_loss: 0.0001, l1_loss: 20.5758, fm_loss: 0.0095, perc_loss: 7.2429, edge: 0.1506
Epoch: [ 5], Batch: [ 400/ 831] | Total Time: 2h 48m 21s
d_loss: 1.3872, g_loss: 30.3480, const_loss: 0.0001, l1_loss: 21.7726, fm_loss: 0.0092, perc_loss: 7.6991, edge: 0.1737
Epoch: [ 5], Batch: [ 500/ 831] | Total Time: 2h 52m 2s
d_loss: 1.3869, g_loss: 28.6966, const_loss: 0.0001, l1_loss: 20.8293, fm_loss: 0.0092, perc_loss: 7.0095, edge: 0.1552
Epoch: [ 5], Batch: [ 600/ 831] | Total Time: 2h 55m 49s
d_loss: 1.3867, g_loss: 30.3430, const_loss: 0.0001, l1_loss: 21.7186, fm_loss: 0.0090, perc_loss: 7.7619, edge: 0.1602
Epoch: [ 5], Batch: [ 700/ 831] | Total Time: 2h 59m 31s
d_loss: 1.3867, g_loss: 26.9955, const_loss: 0.0001, l1_loss: 19.2643, fm_loss: 0.0076, perc_loss: 6.8735, edge: 0.1568
Epoch: [ 5], Batch: [ 800/ 831] | Total Time: 3h 3m 12s
d_loss: 1.3870, g_loss: 30.2710, const_loss: 0.0001, l1_loss: 21.4690, fm_loss: 0.0085, perc_loss: 7.9313, edge: 0.1687
--- End of Epoch 5 --- Time: 1842.3s ---
LR Scheduler stepped. Current LR G: 0.000189, LR D: 0.000189
Epoch: [ 6], Batch: [ 0/ 831] | Total Time: 3h 4m 18s
d_loss: 1.3867, g_loss: 29.1717, const_loss: 0.0000, l1_loss: 20.8315, fm_loss: 0.0087, perc_loss: 7.4665, edge: 0.1716
Epoch: [ 6], Batch: [ 100/ 831] | Total Time: 3h 8m 0s
d_loss: 1.3868, g_loss: 28.7507, const_loss: 0.0000, l1_loss: 20.4751, fm_loss: 0.0085, perc_loss: 7.4092, edge: 0.1645
Epoch: [ 6], Batch: [ 200/ 831] | Total Time: 3h 11m 41s
d_loss: 1.3868, g_loss: 28.8108, const_loss: 0.0000, l1_loss: 20.3923, fm_loss: 0.0081, perc_loss: 7.5529, edge: 0.1643
Epoch: [ 6], Batch: [ 300/ 831] | Total Time: 3h 15m 22s
d_loss: 1.3868, g_loss: 26.9529, const_loss: 0.0000, l1_loss: 19.3393, fm_loss: 0.0073, perc_loss: 6.7591, edge: 0.1538
Epoch: [ 6], Batch: [ 400/ 831] | Total Time: 3h 19m 3s
d_loss: 1.3867, g_loss: 30.6014, const_loss: 0.0001, l1_loss: 21.8041, fm_loss: 0.0088, perc_loss: 7.9229, edge: 0.1723
Epoch: [ 6], Batch: [ 500/ 831] | Total Time: 3h 22m 43s
d_loss: 1.3868, g_loss: 30.8951, const_loss: 0.0001, l1_loss: 22.4039, fm_loss: 0.0092, perc_loss: 7.6097, edge: 0.1789
Epoch: [ 6], Batch: [ 600/ 831] | Total Time: 3h 26m 24s
d_loss: 1.3867, g_loss: 32.4369, const_loss: 0.0000, l1_loss: 23.1193, fm_loss: 0.0103, perc_loss: 8.4328, edge: 0.1816
Epoch: [ 6], Batch: [ 700/ 831] | Total Time: 3h 30m 5s
d_loss: 1.3867, g_loss: 29.6670, const_loss: 0.0001, l1_loss: 21.0724, fm_loss: 0.0085, perc_loss: 7.7221, edge: 0.1707
Epoch: [ 6], Batch: [ 800/ 831] | Total Time: 3h 33m 46s
d_loss: 1.3867, g_loss: 28.0461, const_loss: 0.0001, l1_loss: 20.1075, fm_loss: 0.0074, perc_loss: 7.0802, edge: 0.1576
--- End of Epoch 6 --- Time: 1836.4s ---
LR Scheduler stepped. Current LR G: 0.000185, LR D: 0.000185
Epoch: [ 7], Batch: [ 0/ 831] | Total Time: 3h 34m 55s
d_loss: 1.3867, g_loss: 27.3546, const_loss: 0.0001, l1_loss: 19.8167, fm_loss: 0.0079, perc_loss: 6.6833, edge: 0.1533
Epoch: [ 7], Batch: [ 100/ 831] | Total Time: 3h 38m 35s
d_loss: 1.3868, g_loss: 28.8592, const_loss: 0.0000, l1_loss: 20.6384, fm_loss: 0.0080, perc_loss: 7.3468, edge: 0.1726
Epoch: [ 7], Batch: [ 200/ 831] | Total Time: 3h 42m 21s
d_loss: 1.3868, g_loss: 28.2519, const_loss: 0.0000, l1_loss: 20.1436, fm_loss: 0.0087, perc_loss: 7.2524, edge: 0.1539
Epoch: [ 7], Batch: [ 300/ 831] | Total Time: 3h 46m 2s
d_loss: 1.3867, g_loss: 26.5044, const_loss: 0.0002, l1_loss: 19.0200, fm_loss: 0.0071, perc_loss: 6.6267, edge: 0.1570
Epoch: [ 7], Batch: [ 400/ 831] | Total Time: 3h 49m 48s
d_loss: 1.3868, g_loss: 30.1898, const_loss: 0.0001, l1_loss: 21.3035, fm_loss: 0.0088, perc_loss: 8.0104, edge: 0.1736
Epoch: [ 7], Batch: [ 500/ 831] | Total Time: 3h 53m 36s
d_loss: 1.3868, g_loss: 32.9112, const_loss: 0.0000, l1_loss: 23.4795, fm_loss: 0.0102, perc_loss: 8.5379, edge: 0.1902
Epoch: [ 7], Batch: [ 600/ 831] | Total Time: 3h 57m 22s
d_loss: 1.3868, g_loss: 32.9340, const_loss: 0.0001, l1_loss: 23.5750, fm_loss: 0.0115, perc_loss: 8.4768, edge: 0.1773
from 1154 to 1156, 這次有增加一些訓練資料, 所以每一個 epoch 花費的時間有拉長, figure:
log:
Epoch: [ 0], Batch: [ 0/1009] | Total Time: 2s
d_loss: 1.3868, g_loss: 30.8982, const_loss: 0.0000, l1_loss: 21.9767, fm_loss: 0.0096, perc_loss: 8.0531, edge: 0.1655
Checkpoint step 100 reached, but saving starts after step 200.
Epoch: [ 0], Batch: [ 100/1009] | Total Time: 3m 37s
d_loss: 1.3869, g_loss: 28.9803, const_loss: 0.0000, l1_loss: 20.4443, fm_loss: 0.0078, perc_loss: 7.6666, edge: 0.1682
Epoch: [ 0], Batch: [ 200/1009] | Total Time: 7m 21s
d_loss: 1.3867, g_loss: 30.1839, const_loss: 0.0000, l1_loss: 21.2023, fm_loss: 0.0082, perc_loss: 8.0982, edge: 0.1818
Epoch: [ 0], Batch: [ 300/1009] | Total Time: 11m 2s
d_loss: 1.3867, g_loss: 32.5938, const_loss: 0.0001, l1_loss: 23.2653, fm_loss: 0.0093, perc_loss: 8.4389, edge: 0.1869
Epoch: [ 0], Batch: [ 400/1009] | Total Time: 14m 49s
d_loss: 1.3868, g_loss: 29.0254, const_loss: 0.0000, l1_loss: 20.6883, fm_loss: 0.0080, perc_loss: 7.4740, edge: 0.1617
Epoch: [ 0], Batch: [ 500/1009] | Total Time: 18m 31s
d_loss: 1.3869, g_loss: 32.2797, const_loss: 0.0000, l1_loss: 23.0599, fm_loss: 0.0090, perc_loss: 8.3407, edge: 0.1768
Epoch: [ 0], Batch: [ 600/1009] | Total Time: 22m 12s
d_loss: 1.3867, g_loss: 28.9536, const_loss: 0.0001, l1_loss: 20.5581, fm_loss: 0.0077, perc_loss: 7.5244, edge: 0.1698
Epoch: [ 0], Batch: [ 700/1009] | Total Time: 25m 53s
d_loss: 1.3867, g_loss: 30.3983, const_loss: 0.0001, l1_loss: 21.6764, fm_loss: 0.0082, perc_loss: 7.8442, edge: 0.1761
Epoch: [ 0], Batch: [ 800/1009] | Total Time: 29m 34s
d_loss: 1.3869, g_loss: 30.7399, const_loss: 0.0000, l1_loss: 22.0017, fm_loss: 0.0097, perc_loss: 7.8638, edge: 0.1718
Epoch: [ 0], Batch: [ 900/1009] | Total Time: 33m 15s
d_loss: 1.3868, g_loss: 32.3250, const_loss: 0.0000, l1_loss: 22.9278, fm_loss: 0.0094, perc_loss: 8.5072, edge: 0.1871
Epoch: [ 0], Batch: [1000/1009] | Total Time: 36m 57s
d_loss: 1.3869, g_loss: 26.8201, const_loss: 0.0000, l1_loss: 19.0496, fm_loss: 0.0072, perc_loss: 6.9205, edge: 0.1495
--- End of Epoch 0 --- Time: 2234.1s ---
LR Scheduler stepped. Current LR G: 0.000190, LR D: 0.000190
Epoch: [ 1], Batch: [ 0/1009] | Total Time: 37m 16s
d_loss: 1.3868, g_loss: 28.3229, const_loss: 0.0000, l1_loss: 20.0727, fm_loss: 0.0067, perc_loss: 7.3894, edge: 0.1608
Epoch: [ 1], Batch: [ 100/1009] | Total Time: 41m 2s
d_loss: 1.3869, g_loss: 31.8044, const_loss: 0.0000, l1_loss: 22.6914, fm_loss: 0.0095, perc_loss: 8.2369, edge: 0.1732
Epoch: [ 1], Batch: [ 200/1009] | Total Time: 44m 44s
d_loss: 1.3867, g_loss: 29.9654, const_loss: 0.0000, l1_loss: 21.2983, fm_loss: 0.0078, perc_loss: 7.7923, edge: 0.1735
Epoch: [ 1], Batch: [ 300/1009] | Total Time: 48m 25s
d_loss: 1.3867, g_loss: 29.1098, const_loss: 0.0000, l1_loss: 20.8463, fm_loss: 0.0084, perc_loss: 7.3878, edge: 0.1740
Epoch: [ 1], Batch: [ 400/1009] | Total Time: 52m 8s
d_loss: 1.3882, g_loss: 26.3511, const_loss: 0.0000, l1_loss: 18.8556, fm_loss: 0.0071, perc_loss: 6.6387, edge: 0.1562
Epoch: [ 1], Batch: [ 500/1009] | Total Time: 55m 48s
d_loss: 1.3867, g_loss: 33.1218, const_loss: 0.0000, l1_loss: 23.4463, fm_loss: 0.0096, perc_loss: 8.7765, edge: 0.1965
Epoch: [ 1], Batch: [ 600/1009] | Total Time: 59m 29s
d_loss: 1.3869, g_loss: 27.1650, const_loss: 0.0000, l1_loss: 19.6032, fm_loss: 0.0068, perc_loss: 6.7061, edge: 0.1555
Epoch: [ 1], Batch: [ 700/1009] | Total Time: 1h 3m 16s
d_loss: 1.3867, g_loss: 34.4517, const_loss: 0.0000, l1_loss: 24.6461, fm_loss: 0.0096, perc_loss: 8.9072, edge: 0.1955
Epoch: [ 1], Batch: [ 800/1009] | Total Time: 1h 6m 57s
d_loss: 1.3867, g_loss: 25.9927, const_loss: 0.0000, l1_loss: 18.8615, fm_loss: 0.0065, perc_loss: 6.2929, edge: 0.1384
Epoch: [ 1], Batch: [ 900/1009] | Total Time: 1h 10m 38s
d_loss: 1.3867, g_loss: 30.7794, const_loss: 0.0000, l1_loss: 21.7066, fm_loss: 0.0082, perc_loss: 8.1862, edge: 0.1850
Epoch: [ 1], Batch: [1000/1009] | Total Time: 1h 14m 20s
d_loss: 1.3868, g_loss: 27.2384, const_loss: 0.0000, l1_loss: 19.2381, fm_loss: 0.0076, perc_loss: 7.1477, edge: 0.1516
--- End of Epoch 1 --- Time: 2243.0s ---
LR Scheduler stepped. Current LR G: 0.000189, LR D: 0.000189
Epoch: [ 2], Batch: [ 0/1009] | Total Time: 1h 14m 39s
d_loss: 1.3867, g_loss: 28.9539, const_loss: 0.0000, l1_loss: 20.7931, fm_loss: 0.0083, perc_loss: 7.2900, edge: 0.1690
Epoch: [ 2], Batch: [ 100/1009] | Total Time: 1h 18m 22s
d_loss: 1.3869, g_loss: 28.5101, const_loss: 0.0000, l1_loss: 20.3565, fm_loss: 0.0073, perc_loss: 7.2837, edge: 0.1692
Epoch: [ 2], Batch: [ 200/1009] | Total Time: 1h 22m 3s
d_loss: 1.3867, g_loss: 30.9734, const_loss: 0.0000, l1_loss: 22.0455, fm_loss: 0.0080, perc_loss: 8.0474, edge: 0.1792
Epoch: [ 2], Batch: [ 300/1009] | Total Time: 1h 25m 45s
d_loss: 1.3867, g_loss: 30.1032, const_loss: 0.0000, l1_loss: 21.5729, fm_loss: 0.0076, perc_loss: 7.6616, edge: 0.1676
Epoch: [ 2], Batch: [ 400/1009] | Total Time: 1h 29m 26s
d_loss: 1.3868, g_loss: 31.2766, const_loss: 0.0000, l1_loss: 22.1735, fm_loss: 0.0079, perc_loss: 8.2260, edge: 0.1759
Epoch: [ 2], Batch: [ 500/1009] | Total Time: 1h 33m 11s
d_loss: 1.3867, g_loss: 27.0743, const_loss: 0.0000, l1_loss: 19.2202, fm_loss: 0.0066, perc_loss: 6.9907, edge: 0.1634
Epoch: [ 2], Batch: [ 600/1009] | Total Time: 1h 36m 52s
d_loss: 1.3868, g_loss: 29.5774, const_loss: 0.0000, l1_loss: 21.1771, fm_loss: 0.0076, perc_loss: 7.5297, edge: 0.1697
Epoch: [ 2], Batch: [ 700/1009] | Total Time: 1h 40m 36s
d_loss: 1.3871, g_loss: 30.8395, const_loss: 0.0000, l1_loss: 22.0560, fm_loss: 0.0077, perc_loss: 7.9053, edge: 0.1776
Epoch: [ 2], Batch: [ 800/1009] | Total Time: 1h 44m 17s
d_loss: 1.3867, g_loss: 27.0825, const_loss: 0.0000, l1_loss: 19.2367, fm_loss: 0.0063, perc_loss: 6.9839, edge: 0.1623
Epoch: [ 2], Batch: [ 900/1009] | Total Time: 1h 47m 59s
d_loss: 1.3869, g_loss: 26.7615, const_loss: 0.0000, l1_loss: 19.2430, fm_loss: 0.0074, perc_loss: 6.6605, edge: 0.1572
Epoch: [ 2], Batch: [1000/1009] | Total Time: 1h 51m 40s
d_loss: 1.3867, g_loss: 29.6087, const_loss: 0.0000, l1_loss: 20.9553, fm_loss: 0.0074, perc_loss: 7.7765, edge: 0.1761
--- End of Epoch 2 --- Time: 2240.5s ---
LR Scheduler stepped. Current LR G: 0.000187, LR D: 0.000187
Epoch: [ 3], Batch: [ 0/1009] | Total Time: 1h 51m 59s
d_loss: 1.3867, g_loss: 29.7576, const_loss: 0.0000, l1_loss: 20.9788, fm_loss: 0.0074, perc_loss: 7.9003, edge: 0.1777
Epoch: [ 3], Batch: [ 100/1009] | Total Time: 1h 55m 40s
d_loss: 1.3868, g_loss: 27.0645, const_loss: 0.0000, l1_loss: 19.1410, fm_loss: 0.0077, perc_loss: 7.0605, edge: 0.1619
Epoch: [ 3], Batch: [ 200/1009] | Total Time: 1h 59m 25s
d_loss: 1.3868, g_loss: 27.4857, const_loss: 0.0000, l1_loss: 19.4952, fm_loss: 0.0071, perc_loss: 7.1304, edge: 0.1596
Epoch: [ 3], Batch: [ 300/1009] | Total Time: 2h 3m 6s
d_loss: 1.3869, g_loss: 30.4294, const_loss: 0.0000, l1_loss: 21.7812, fm_loss: 0.0087, perc_loss: 7.7735, edge: 0.1726
Epoch: [ 3], Batch: [ 400/1009] | Total Time: 2h 6m 47s
d_loss: 1.3867, g_loss: 30.4152, const_loss: 0.0000, l1_loss: 21.6118, fm_loss: 0.0085, perc_loss: 7.9246, edge: 0.1769
Epoch: [ 3], Batch: [ 500/1009] | Total Time: 2h 10m 28s
d_loss: 1.3868, g_loss: 24.7118, const_loss: 0.0000, l1_loss: 17.6776, fm_loss: 0.0064, perc_loss: 6.1923, edge: 0.1421
Epoch: [ 3], Batch: [ 600/1009] | Total Time: 2h 14m 10s
d_loss: 1.3870, g_loss: 27.6480, const_loss: 0.0000, l1_loss: 19.7870, fm_loss: 0.0069, perc_loss: 7.0050, edge: 0.1557
Epoch: [ 3], Batch: [ 700/1009] | Total Time: 2h 17m 50s
d_loss: 1.3872, g_loss: 29.1706, const_loss: 0.0000, l1_loss: 20.8029, fm_loss: 0.0063, perc_loss: 7.4896, edge: 0.1784
Epoch: [ 3], Batch: [ 800/1009] | Total Time: 2h 21m 36s
d_loss: 1.3867, g_loss: 27.0851, const_loss: 0.0000, l1_loss: 19.3615, fm_loss: 0.0058, perc_loss: 6.8649, edge: 0.1594
Epoch: [ 3], Batch: [ 900/1009] | Total Time: 2h 25m 17s
d_loss: 1.3867, g_loss: 29.4218, const_loss: 0.0000, l1_loss: 21.0527, fm_loss: 0.0071, perc_loss: 7.5092, edge: 0.1595
Epoch: [ 3], Batch: [1000/1009] | Total Time: 2h 29m 4s
d_loss: 1.3868, g_loss: 27.5423, const_loss: 0.0000, l1_loss: 19.8119, fm_loss: 0.0063, perc_loss: 6.8660, edge: 0.1648
--- End of Epoch 3 --- Time: 2243.7s ---
LR Scheduler stepped. Current LR G: 0.000185, LR D: 0.000185
Epoch: [ 4], Batch: [ 0/1009] | Total Time: 2h 29m 23s
d_loss: 1.3867, g_loss: 26.1920, const_loss: 0.0000, l1_loss: 18.7728, fm_loss: 0.0068, perc_loss: 6.5619, edge: 0.1571
Epoch: [ 4], Batch: [ 100/1009] | Total Time: 2h 33m 5s
d_loss: 1.3874, g_loss: 27.6825, const_loss: 0.0000, l1_loss: 19.5811, fm_loss: 0.0075, perc_loss: 7.2365, edge: 0.1641
Epoch: [ 4], Batch: [ 200/1009] | Total Time: 2h 36m 46s
d_loss: 1.3868, g_loss: 24.6662, const_loss: 0.0000, l1_loss: 17.5898, fm_loss: 0.0059, perc_loss: 6.2324, edge: 0.1447
Epoch: [ 4], Batch: [ 300/1009] | Total Time: 2h 40m 27s
d_loss: 1.3873, g_loss: 29.2333, const_loss: 0.0000, l1_loss: 20.7596, fm_loss: 0.0071, perc_loss: 7.5919, edge: 0.1814
Epoch: [ 4], Batch: [ 400/1009] | Total Time: 2h 44m 8s
d_loss: 1.3868, g_loss: 28.3472, const_loss: 0.0000, l1_loss: 20.1356, fm_loss: 0.0072, perc_loss: 7.3441, edge: 0.1669
Epoch: [ 4], Batch: [ 500/1009] | Total Time: 2h 47m 50s
d_loss: 1.3867, g_loss: 29.1811, const_loss: 0.0000, l1_loss: 20.5613, fm_loss: 0.0072, perc_loss: 7.7458, edge: 0.1735
Epoch: [ 4], Batch: [ 600/1009] | Total Time: 2h 51m 31s
d_loss: 1.3870, g_loss: 27.2661, const_loss: 0.0000, l1_loss: 19.4570, fm_loss: 0.0063, perc_loss: 6.9491, edge: 0.1603
Epoch: [ 4], Batch: [ 700/1009] | Total Time: 2h 55m 12s
d_loss: 1.3867, g_loss: 30.0584, const_loss: 0.0000, l1_loss: 21.2344, fm_loss: 0.0071, perc_loss: 7.9527, edge: 0.1709
Epoch: [ 4], Batch: [ 800/1009] | Total Time: 2h 58m 53s
d_loss: 1.3867, g_loss: 28.7596, const_loss: 0.0000, l1_loss: 20.2697, fm_loss: 0.0072, perc_loss: 7.6180, edge: 0.1718
Epoch: [ 4], Batch: [ 900/1009] | Total Time: 3h 2m 35s
d_loss: 1.3868, g_loss: 30.3219, const_loss: 0.0000, l1_loss: 21.4957, fm_loss: 0.0071, perc_loss: 7.9548, edge: 0.1709
Epoch: [ 4], Batch: [1000/1009] | Total Time: 3h 6m 16s
d_loss: 1.3867, g_loss: 26.5545, const_loss: 0.0000, l1_loss: 19.0008, fm_loss: 0.0061, perc_loss: 6.7011, edge: 0.1531
--- End of Epoch 4 --- Time: 2232.2s ---
LR Scheduler stepped. Current LR G: 0.000183, LR D: 0.000183
Epoch: [ 5], Batch: [ 0/1009] | Total Time: 3h 6m 35s
d_loss: 1.3869, g_loss: 25.9558, const_loss: 0.0000, l1_loss: 18.2952, fm_loss: 0.0059, perc_loss: 6.8037, edge: 0.1577
Epoch: [ 5], Batch: [ 100/1009] | Total Time: 3h 10m 18s
d_loss: 1.3869, g_loss: 27.9378, const_loss: 0.0000, l1_loss: 20.0886, fm_loss: 0.0065, perc_loss: 6.9790, edge: 0.1702
Epoch: [ 5], Batch: [ 200/1009] | Total Time: 3h 13m 59s
d_loss: 1.3867, g_loss: 25.6503, const_loss: 0.0000, l1_loss: 18.2971, fm_loss: 0.0062, perc_loss: 6.5019, edge: 0.1518
Epoch: [ 5], Batch: [ 300/1009] | Total Time: 3h 17m 40s
d_loss: 1.3868, g_loss: 25.8447, const_loss: 0.0000, l1_loss: 18.3820, fm_loss: 0.0058, perc_loss: 6.6138, edge: 0.1499
Epoch: [ 5], Batch: [ 400/1009] | Total Time: 3h 21m 21s
d_loss: 1.3869, g_loss: 25.8463, const_loss: 0.0000, l1_loss: 18.3924, fm_loss: 0.0065, perc_loss: 6.6053, edge: 0.1488
Epoch: [ 5], Batch: [ 500/1009] | Total Time: 3h 25m 3s
d_loss: 1.3869, g_loss: 27.0186, const_loss: 0.0000, l1_loss: 19.3287, fm_loss: 0.0066, perc_loss: 6.8275, edge: 0.1625
Epoch: [ 5], Batch: [ 600/1009] | Total Time: 3h 28m 44s
d_loss: 1.3868, g_loss: 24.8367, const_loss: 0.0000, l1_loss: 17.7645, fm_loss: 0.0050, perc_loss: 6.2275, edge: 0.1463
Epoch: [ 5], Batch: [ 700/1009] | Total Time: 3h 32m 25s
d_loss: 1.3869, g_loss: 26.6676, const_loss: 0.0000, l1_loss: 19.0766, fm_loss: 0.0059, perc_loss: 6.7345, edge: 0.1572
Epoch: [ 5], Batch: [ 800/1009] | Total Time: 3h 36m 6s
d_loss: 1.3867, g_loss: 25.8846, const_loss: 0.0000, l1_loss: 18.3836, fm_loss: 0.0063, perc_loss: 6.6498, edge: 0.1515
Epoch: [ 5], Batch: [ 900/1009] | Total Time: 3h 39m 49s
d_loss: 1.3867, g_loss: 28.1501, const_loss: 0.0000, l1_loss: 20.1098, fm_loss: 0.0070, perc_loss: 7.1771, edge: 0.1629
Epoch: [ 5], Batch: [1000/1009] | Total Time: 3h 43m 30s
d_loss: 1.3868, g_loss: 29.9697, const_loss: 0.0000, l1_loss: 21.3228, fm_loss: 0.0070, perc_loss: 7.7649, edge: 0.1817
--- End of Epoch 5 --- Time: 2234.3s ---
LR Scheduler stepped. Current LR G: 0.000180, LR D: 0.000180
Epoch: [ 6], Batch: [ 0/1009] | Total Time: 3h 43m 49s
d_loss: 1.3868, g_loss: 27.0016, const_loss: 0.0000, l1_loss: 19.3607, fm_loss: 0.0062, perc_loss: 6.7821, edge: 0.1593
Epoch: [ 6], Batch: [ 100/1009] | Total Time: 3h 47m 31s
d_loss: 1.3868, g_loss: 27.0326, const_loss: 0.0000, l1_loss: 19.3591, fm_loss: 0.0058, perc_loss: 6.8141, edge: 0.1603
Epoch: [ 6], Batch: [ 200/1009] | Total Time: 3h 51m 12s
d_loss: 1.3868, g_loss: 26.7903, const_loss: 0.0000, l1_loss: 19.2709, fm_loss: 0.0059, perc_loss: 6.6624, edge: 0.1576
Epoch: [ 6], Batch: [ 300/1009] | Total Time: 3h 54m 53s
d_loss: 1.3867, g_loss: 28.7907, const_loss: 0.0000, l1_loss: 20.7099, fm_loss: 0.0072, perc_loss: 7.2099, edge: 0.1703
Epoch: [ 6], Batch: [ 400/1009] | Total Time: 3h 58m 40s
d_loss: 1.3868, g_loss: 27.4810, const_loss: 0.0000, l1_loss: 19.6863, fm_loss: 0.0069, perc_loss: 6.9346, edge: 0.1598
Epoch: [ 6], Batch: [ 500/1009] | Total Time: 4h 2m 22s
d_loss: 1.3867, g_loss: 26.0709, const_loss: 0.0000, l1_loss: 18.5806, fm_loss: 0.0067, perc_loss: 6.6352, edge: 0.1551
Epoch: [ 6], Batch: [ 600/1009] | Total Time: 4h 6m 3s
d_loss: 1.3867, g_loss: 27.7621, const_loss: 0.0000, l1_loss: 19.8631, fm_loss: 0.0063, perc_loss: 7.0288, edge: 0.1706
Epoch: [ 6], Batch: [ 700/1009] | Total Time: 4h 9m 45s
d_loss: 1.3868, g_loss: 26.2892, const_loss: 0.0000, l1_loss: 18.5212, fm_loss: 0.0060, perc_loss: 6.9108, edge: 0.1578
Epoch: [ 6], Batch: [ 800/1009] | Total Time: 4h 13m 26s
d_loss: 1.3872, g_loss: 26.2446, const_loss: 0.0000, l1_loss: 18.6828, fm_loss: 0.0067, perc_loss: 6.7054, edge: 0.1563
Epoch: [ 6], Batch: [ 900/1009] | Total Time: 4h 17m 48s
d_loss: 1.3867, g_loss: 26.5457, const_loss: 0.0000, l1_loss: 18.9076, fm_loss: 0.0059, perc_loss: 6.7872, edge: 0.1517
Epoch: [ 6], Batch: [1000/1009] | Total Time: 4h 21m 29s
d_loss: 1.3868, g_loss: 26.4308, const_loss: 0.0000, l1_loss: 18.6193, fm_loss: 0.0056, perc_loss: 6.9545, edge: 0.1581
--- End of Epoch 6 --- Time: 2279.0s ---
LR Scheduler stepped. Current LR G: 0.000176, LR D: 0.000176
Epoch: [ 7], Batch: [ 0/1009] | Total Time: 4h 21m 49s
d_loss: 1.3867, g_loss: 26.4335, const_loss: 0.0000, l1_loss: 18.9572, fm_loss: 0.0055, perc_loss: 6.6257, edge: 0.1516
Epoch: [ 7], Batch: [ 100/1009] | Total Time: 4h 25m 33s
d_loss: 1.3867, g_loss: 26.3623, const_loss: 0.0000, l1_loss: 18.8974, fm_loss: 0.0054, perc_loss: 6.6119, edge: 0.1543
Epoch: [ 7], Batch: [ 200/1009] | Total Time: 4h 29m 15s
d_loss: 1.3868, g_loss: 25.3038, const_loss: 0.0000, l1_loss: 18.0112, fm_loss: 0.0052, perc_loss: 6.4438, edge: 0.1501