Deep learning-based noise filtering toward millisecond order imaging by using scanning transmission electron microscopy

August 05, 2022

Shiro Ihara (1), Hikaru Saito (1) (3), Mizumo Yoshinaga (2), Lavakumar Avala (1), Mitsushiro Murayama (1) (4) (5)
Scientific Reports. (5 August 2022). DOI:


Microscopy, transmission electron microscopy


Application of scanning transmission electron microscopy (STEM) to in situ observation will be essential in the current and emerging data-driven materials science by taking STEM’s high affinity with various analytical options into account. As is well known, STEM’s image acquisition time needs to be further shortened to capture a targeted phenomenon in real-time as STEM’s current temporal resolution is far below the conventional TEM’s. However, rapid image acquisition in the millisecond per frame or faster generally causes image distortion, poor electron signals, and unidirectional blurring, which are obstacles for realizing video-rate STEM observation. Here we show an image correction framework integrating deep learning (DL)-based denoising and image distortion correction schemes optimized for STEM rapid image acquisition. By comparing a series of distortion corrected rapid scan images with corresponding regular scan speed images, the trained DL network is shown to remove not only the statistical noise but also the unidirectional blurring. This result demonstrates that rapid as well as high-quality image acquisition by STEM without hardware modification can be established by the DL. The DL-based noise filter could be applied to in-situ observation, such as dislocation activities under external stimuli, with high spatio-temporal resolution.

How Our Software Was Used

Dragonfly was used to train a deep learning U-Net model.

Author Affiliation

(1) Institute for Materials Chemistry and Engineering, Kyushu University, Fukuoka, 816-8580, Japan
(2) Interdisciplinary Graduate School of Engineering Sciences, Kyushu University, Fukuoka, 816-8580, Japan
(3) Pan-Omics Data-Driven Research Innovation Center, Kyushu University, Fukuoka, 816-8580, Japan
(4) Department of Materials Science and Engineering, Virginia Tech, Blacksburg, VA, 24061, USA
(5) Reactor Materials and Mechanical Design Group, Energy and Environmental Directorate, Pacific Northwest National Laboratory, Richland, WA, 99354, USA