Tensor-based Approach to Big Data Processing and Machine Learning
Maciej Bartoszuk, Jarosław Litwin, Mateusz Wnuk, Dominik Ślȩzak · 2022 IEEE International Conference on Big Data (Big Data) · 2022
We present an approach to tensor compression and decomposition, as well as to a design of data processing algorithms on the top of them. Our implementation uses the popular scalable data processing framework Apache Parquet to effectively store the data. This library does not directly store tensors as native data types, but we slightly changed its implementation for our purpose using its specific data storage format and extending it with additional compression. We summarize the performance of tensor storage, as well as the effectiveness of multiple machine learning methods and their hyperparameter tuning.