Data Compression in Transmission Electron Microscopy
James Done, Ambarneil Saha, Jungyoun Cho, Shervin S Nia, Lucas Lee, Peter A. Ercius, Matthew H. Mecklenburg · Microscopy and Microanalysis · 2025
Electron microscopy (EM) has become an essential tool in various fields, including materials science, biology, and nanotechnology. The increasing resolution and size of EM images have led to a growing need for efficient compression algorithms to reduce storage and transmission costs. However, the complexity of EM images, which is characterized by their unique structural and textual features, poses a significant challenge to compression. In this study, Shannon entropy is used as a metric to quantify the level of complexity in EM images as seen in Figure 1 [1]. The application of Shannon entropy can be used to evaluate the efficacy of various compression algorithms than can be used in this field. Several lossless compression algorithms are investigated to compress resulting numbers of large image files. LZ4 and LZW compression can be used for TIFF files. The LZ4 compression algorithm works by finding repeated patterns in the input data and representing them as a reference to the previous occurrence, rather than storing the repeated data itself. The LZW compression is implemented similarly to LZ4 but builds its hash table dynamically. Both the LZ4 and LZW algorithms are extensions of the LZ77 algorithm [2]. In addition, this study investigates a novel compression algorithm pioneered at LBNL NCEM that is proves to be great at compressing sparse images. This algorithm saves pointers or locations of data rather than include zeroes and compress the resulting array into an HDF5 file [3]. A dataset of EM images with varying levels of complexity including images of materials with different microstructures and textures is collected [4]. Various compression algorithms are applied to each image and where Shannon entropy can be used as metric to predict the compression ratio of other images. Results show that Shannon entropy can effectively capture the complexity of EM images. Images with higher Shannon entropy values were found to be more challenging to compress, while images with lower Shannon entropy values were more easily compressed. This study demonstrates the use of Shannon entropy as a metric to quantify the level of complexity in EM images. Shannon entropy can provide a reliable metric for comparing the performance of different compression algorithms. The results of this study have important implications for the development of efficient compression algorithms for EM images. By using Shannon entropy to evaluate the complexity of EM images, researchers and developers can design more effective compression algorithms that take into account the unique features of EM images. In conclusion, this study quantifies the efficacy of compression algorithms that can be employed in EM as seen in Figure 2. The application of Shannon entropy can be used to evaluate the complexity of EM images and it can provide a reliable metric for understanding the performance of different compression algorithms. The results of this study have important implications for the development of efficient compression algorithms for EM images and highlight the need for further research in this area. For an EM image with low Shannon entropy, Shannon entropy based on a series spatial re-binning (4096×4096, 2048×2048, 1024×1024, 512×512, 256×256 and 128×128) and the respective bit density are plotted as paired points; the expected theoretical curve of Shannon entropy as a function of bit density is overlayed. Compression algorithms such as LZ4 [2] compression and LBNL NCEM (pointer based) compression (saved into the HDF5 format [3]) are compared for a series of images taken at different spot sizes examining their ratios of maximum compressed to compressed sizes.