TY - GEN
T1 - Rebalancing gradient to improve self-supervised co-training of depth, odometry and optical flow predictions
AU - Hariat, Marwane
AU - Manzanera, Antoine
AU - Filliat, David
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023/1/1
Y1 - 2023/1/1
N2 - We present CoopNet, an approach that improves the co-operation of co-trained networks by dynamically adapting the apportionment of gradient, to ensure equitable learning progress. It is applied to motion-aware self-supervised prediction of depth maps, by introducing a new hybrid loss, based on a distribution model of photo-metric reconstruction errors made by, on the one hand the depth + odometry paired networks, and on the other hand the optical flow network. This model essentially assumes that the pixels from moving objects (that must be discarded for training depth and odometry), correspond to those where the two reconstructions strongly disagree. We justify this model by theoretical considerations and experimental evidences. A comparative evaluation on KITTI and CityScapes datasets shows that CoopNet improves or is comparable to the state-of-the-art in depth, odometry and optical flow predictions. Our code is available here: https://github.com/mhariat/CoopNet.
AB - We present CoopNet, an approach that improves the co-operation of co-trained networks by dynamically adapting the apportionment of gradient, to ensure equitable learning progress. It is applied to motion-aware self-supervised prediction of depth maps, by introducing a new hybrid loss, based on a distribution model of photo-metric reconstruction errors made by, on the one hand the depth + odometry paired networks, and on the other hand the optical flow network. This model essentially assumes that the pixels from moving objects (that must be discarded for training depth and odometry), correspond to those where the two reconstructions strongly disagree. We justify this model by theoretical considerations and experimental evidences. A comparative evaluation on KITTI and CityScapes datasets shows that CoopNet improves or is comparable to the state-of-the-art in depth, odometry and optical flow predictions. Our code is available here: https://github.com/mhariat/CoopNet.
KW - 3D computer vision
KW - Algorithms: Image recognition and understanding (object detection, categorization, segmentation)
KW - Machine learning architectures
KW - and algorithms (including transfer, low-shot, semi-, self-, and un-supervised learning)
KW - formulations
U2 - 10.1109/WACV56688.2023.00132
DO - 10.1109/WACV56688.2023.00132
M3 - Conference contribution
AN - SCOPUS:85149002357
T3 - Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023
SP - 1267
EP - 1276
BT - Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023
Y2 - 3 January 2023 through 7 January 2023
ER -