A Distributed Storage Method Based on Elasticsearch and MinIO Cluster Federation
Weixue Lu, Cen Gao, Shihua Chen, Zhengchen Yu · 2024
This paper aims to propose a solution to address issues related to small file storage, data retrieval speed, and system resource consumption in existing technologies. The main approach involves federating Elasticsearch and MinIO clusters to form an efficient federated cluster. Based on this foundation, file upload operations and validity checks are implemented to ensure file effectiveness and security. Files that pass the checks are stored in the MinIO federated cluster to achieve efficient data storage and management. Simultaneously, the file addresses from the MinIO cluster are mapped, and the file information is synchronized with the Elasticsearch cluster to enable fast data retrieval and redundant storage. Finally, user access to files through applications is supported, ensuring that users can efficiently and conveniently obtain the required files. Through these measures, this approach not only improves the efficiency of small file storage and retrieval but also optimizes system resource usage, providing a reliable distributed storage solution. Experimental results demonstrate that this approach enables small file storage through Elasticsearch and MinIO.