Multi-resolution convolutional neural networks for inverse problems.

Publisher: Nature Publishing Group Country of Publication: England NLM ID: 101563288 Publication Model: Electronic Cited Medium: Internet ISSN: 2045-2322 (Electronic) Linking ISSN: 20452322 NLM ISO Abbreviation: Sci Rep Subsets: PubMed not MEDLINE; MEDLINE

Original Publication: London : Nature Publishing Group, copyright 2011-

Inverse problems in image processing, phase imaging, and computer vision often share the same structure of mapping input image(s) to output image(s) but are usually solved by different application-specific algorithms. Deep convolutional neural networks have shown great potential for highly variable tasks across many image-based domains, but sometimes can be challenging to train due to their internal non-linearity. We propose a novel, fast-converging neural network architecture capable of solving generic image(s)-to-image(s) inverse problems relevant to a diverse set of domains. We show this approach is useful in recovering wavefronts from direct intensity measurements, imaging objects from diffusely reflected images, and denoising scanning transmission electron microscopy images, just by using different training datasets. These successful applications demonstrate the proposed network to be an ideal candidate solving general inverse problems falling into the category of image(s)-to-image(s) translation.

Russakovsky, O. et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 211–252 (2015). (PMID: 10.1007/s11263-015-0816-y)
Krasin, I.et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. 2, 7 (2016).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015). (PMID: 10.1038/nature14539)
Dong, C., Loy, C. C., He, K. & Tang, X. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 295–307 (2016). (PMID: 10.1109/TPAMI.2015.2439281)
Zhang, K., Zuo, W., Chen, Y., Meng, D. & Zhang, L. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Transactions on Image Processing 26, 3142–3155 (2017). (PMID: 10.1109/TIP.2017.2662206)
Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. CVPR (2017).
Hezaveh, Y. D., Levasseur, L. P. & Marshall, P. J. Fast automated analysis of strong gravitational lenses with convolutional neural networks. Nature 548, 555–557 (2017). (PMID: 10.1038/nature23463)
Sinha, A., Lee, J., Li, S. & Barbastathis, G. Lensless computational imaging through deep learning. Optica 4, 1117–1125 (2017). (PMID: 10.1364/OPTICA.4.001117)
Hornik, K., Stinchcombe, M. & White, H. Multilayer feedforward networks are universal approximators. Neural Networks 2, 359–366 (1989). (PMID: 10.1016/0893-6080(89)90020-8)
Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems 2, 303–314 (1989). (PMID: 10.1007/BF02551274)
Lu, Z., Pu, H., Wang, F., Hu, Z. & Wang, L. The Expressive Power of Neural Networks: A View from the Width. In Guyon, I. et al. (eds.) Advances in Neural Information Processing Systems 30, 6231–6239 (Curran Associates, Inc., 2017).
Yang, Z. et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv:1906.08237 (2019).
Brock, A., Donahue, J. & Simonyan, K. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv:1809.11096 (2018).
Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging  6, 921–943, https://www.osapublishing.org/optica/abstract.cfm?uri=optica-6-8-921 .
Jin, K. H., McCann, M. T., Froustey, E. & Unser, M. Deep Convolutional Neural Network for Inverse Problems in Imaging. In IEEE Transactions on Image Processing, 26, 4509–4522, https://doi.org/10.1109/TIP.2017.2713099 (2017).
Luo, R., Tian, F., Qin, T., Chen, E. & Liu, T.-Y. Neural architecture optimization. In Bengio, S. et al. (eds.) Advances in Neural Information Processing Systems 31, 7816–7827 (Curran Associates, Inc., 2018).
Jin, H., Song, Q. & Hu, X. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1946–1956 (ACM, 2019).
Smith, L. N. A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. arXiv:1803.09820 (2018).
Xu, Z.-Q. J., Zhang, Y. & Xiao, Y. Training behavior of deep neural network in frequency domain. arXiv preprint arXiv:1807.01251 (2018).
Rahaman, N. et al. On the Spectral Bias of Deep Neural Networks. arXiv preprint arXiv:1806.08734 (2018).
Stüben, K. & Trottenberg, U. Multigrid methods: Fundamental algorithms, model problem analysis and applications. In Multigrid Methods, 1–176 (Springer, 1982).
Hinton, G. E. & Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. science 313, 504–507 (2006). (PMID: 10.1126/science.1127647)
Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 234–241 (Springer International Publishing, 2015).
Karras, T., Aila, T., Laine, S. & Lehtinen, J. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv:1710.10196 (2017).
Nah, S., Kim, T. H. & Lee, K. M. Deep multi-scale convolutional neural network for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017).
Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, 448–456 (JMLR.org, 2015).
Wu, Y. & He, K. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), 3–19 (2018).
Gabor, D. A New Microscopic Principle. Nature 161, 777–778 (1948). (PMID: 10.1038/161777a0)
Teague, M. R. Deterministic phase retrieval: A Green’s function solution. JOSA 73, 1434–1441 (1983). (PMID: 10.1364/JOSA.73.001434)
Van Dyck, D., Op de Beeck, M. & Coene, W. Object wavefunction reconstruction in high resolution electron microscopy. In Proceedings of 1st International Conference on Image Processing, vol. 3, 295–298 vol.3 (1994).
Jingshan, Z., Claus, R. A., Dauwels, J., Tian, L. & Waller, L. Transport of Intensity phase imaging by intensity spectrum fitting of exponentially spaced defocus planes. Optics Express 22, 10661 (2014). (PMID: 10.1364/OE.22.010661)
Eljarrat, A., Müller, J., Huang, M. R. S. & Koch, C. T. Multi-focus TIE algorithm including partial spatial coherence and overlapping filters. Optics Express 26, 11819 (2018). (PMID: 10.1364/OE.26.011819)
Nugent, K. A., Gureyev, T. E., Cookson, D., Paganin, D. & Barnea, Z. Quantitative phase imaging using hard x-rays. Phys. Rev. Lett. 77, 2961–2964 (1996). (PMID: 10.1103/PhysRevLett.77.2961)
Parvizi, A., Van den Broek, W. & Koch, C. T. Recovering low spatial frequencies in wavefront sensing based on intensity measurements. Advanced Structural and Chemical Imaging 2, 1 (2017).
Rivenson, Y., Zhang, Y., Günaydin, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light: Science & Applications 7, 17141 (2018). (PMID: 10.1038/lsa.2017.141)
Goy, A., Arthur, K., Li, S. & Barbastathis, G. Low Photon Count Phase Retrieval Using Deep Learning. Physical Review Letters 121, 243902, https://doi.org/10.1103/PhysRevLett.121.243902 (2018). (PMID: 10.1103/PhysRevLett.121.2439023060874530608745)
Cherukara, M. J., Nashed, Y. S. G. & Harder, R. J. Real-time coherent diffraction inversion using deep generative networks. Scientific Reports  8, 16520, http://www.nature.com/articles/s41598-018-34525-1 (2018).
Klein, J., Peters, C., Martìn, J., Laurenzis, M. & Hullin, M. B. Tracking objects outside the line of sight using 2d intensity images. Scientific Reports 6, 32491 (2016). (PMID: 10.1038/srep32491)
O’Toole, M., Lindell, D. B. & Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 338–341 (2018). (PMID: 10.1038/nature25489)
Saunders, C., Murray-Bruce, J. & Goyal, V. K. Computational periscopy with an ordinary digital camera. Nature 565, 472–475 (2019). (PMID: 10.1038/s41586-018-0868-6)
Satat, G., Tancik, M., Gupta, O., Heshmat, B. & Raskar, R. Object classification through scattering media with deep learning on time resolved measurement. Optics Express  25, 17466–17479, https://www.osapublishing.org/oe/abstract.cfm?uri=oe-25-15-17466 (2017).
Blender Foundation. Big buck bunny, http://bigbuckbunny.org/ (2008).
Erni, R., Rossell, M. D., Kisielowski, C. & Dahmen, U. Atomic-resolution imaging with a sub-50-pm electron probe. Physical review letters 102, 096101 (2009). (PMID: 10.1103/PhysRevLett.102.096101)
Jiang, Y. et al. Electron ptychography of 2D materials to deep sub-Ångström resolution. Nature 559, 343–349 (2018). (PMID: 10.1038/s41586-018-0298-5)
Henninen, T. R., Bon, M., Wang, F., Passerone, D. & Erni, R. The Structure of Sub-nm Platinum Clusters at Elevated Temperatures. Angewandte Chemie International Edition, https://doi.org/10.1002/anie.201911068 (2019).
Mevenkamp, N. et al. Poisson noise removal from high-resolution STEM images based on periodic block matching. Advanced Structural and Chemical Imaging  1, 3 (2015-03-25).
Furnival, T., Leary, R. K. & Midgley, P. A. Denoising time-resolved microscopy image sequences with singular value thresholding. Ultramicroscopy 178, 112–124 (2017). (PMID: 10.1016/j.ultramic.2016.05.005)
Guo, S., Yan, Z., Zhang, K., Zuo, W. & Zhang, L. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1712–1722 (2019).
Lefkimmiatis, S. Universal denoising networks: a novel CNN architecture for image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3204–3213 (2018).
Ulyanov, D., Vedaldi, A. & Lempitsky, V. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 9446–9454 (2018).
Wang, P., Zhang, H. & Patel, V. M. SAR Image Despeckling Using a Convolutional Neural Network. IEEE Signal Processing Letters  24, 1763–1767, http://ieeexplore.ieee.org/document/8053792/ (2017).
Hasinoff, S. W. et al. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (TOG) 35, 192 (2016). (PMID: 10.1145/2980179.2980254)
Xie, J., Xu, L. & Chen, E. Image Denoising and Inpainting with Deep Neural Networks. In Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems 25, 341–349 (Curran Associates, Inc., 2012).
Lehtinen, J. et al. Noise2noise: Learning Image Restoration without Clean Data. arXiv preprint arXiv:1803.04189 (2018).
Batson, J. & Royer, L. Noise2self: Blind Denoising by Self-Supervision. arXiv:1901.11365 [cs, stat], http://arxiv.org/abs/1901.11365 . ArXiv: 1901.11365 (2019).
Krull, A., Buchholz, T.-O. & Jug, F. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2129–2137 (2019).
Eljarrat, A. Python code for inline holography using through-focus image series, https://github.com/AEljarrat/inline_holo (2018).
Krasin, I. et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from, https://github.com/openimages   2, 3 (2017).
Xie, S., Girshick, R., Dollàr, P., Tu, Z. & He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1492–1500 (2017).
Chollet, F. Xception: Deep Learning with Depthwise Separable Convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1800–1807 (IEEE, Honolulu, HI, 2017).
Goodfellow, I. et al. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q. (eds.) Advances in Neural Information Processing Systems 27, 2672–2680 (Curran Associates, Inc., 2014).
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, http://arxiv.org/abs/1412.6980 (2015).

Date Created: 20200403 Date Completed: 20200902 Latest Revision: 20210331

20250114

PMC7109091

10.1038/s41598-020-62484-z

32235861