Topology-enhanced deep learning for sequential data analysis
Lucheng Huang · 2025
With the accumulation of big data, the increase in computational power, and the development of highly efficient algorithms, machine learning and deep learning techniques have been guided to explore increasingly diverse and heterogeneous topological spaces. Topological Deep Learning (TDL) is a new research area focusing on the study of non-Euclidean data, such as meshes and graphs. This dissertation aims to introduce the necessary TDL formalisms in the field of sequential data analysis to construct higher-order objects and incorporate higher-order information beyond nodes and edges. The proposed pipeline demonstrates how sequential data can be translated into higher-order objects using simplicial complexes and path complexes, with relationships described by boundary operators. These topological representations, i.e., simplicial complexes and path complexes, capture geometric and structural information, enabling many-body interactions beyond pairwise interactions in graphs. To enhance these representations with machine learning, we leverage a message-passing mechanism that offers greater expressiveness compared to Graph Neural Network (GNN) models. In the context of sequential data, we review methods for effectively representing sequential data and evaluating model performance, incorporating long-range information and exploring modern techniques in Natural Language Processing (NLP) and Business Process Mining (BPM). Motivated by the mathematical foundations of TDL, simplicial complexes are first explored for sequential text data. The proposed framework, Simplicial Convolutional Networks (SCN), constructs a simplicial complex based on short document text, defining representations for 0-simplexes (word nodes), 1-simplexes (edges between adjacent words), and 2-simplexes (triangles formed by three consecutive words). Leveraging the message-passing mechanism for higher-order simplexes, information within the neighbourhood structure is updated and aggregated. The simplicial complex representation is then obtained using a self-attention mechanism to summarise the document for classification tasks. To address label scarcity in short text classification and leverage the power of Large Language Models (LLM), we adopt a few-shot setting with contrastive learning for SCN (C-SCN), where traditional graph construction methods with external auxiliary information and pre-training contrastive learning by removing nodes and edges are modified. As a result, C-SCN can be trained with a small amount of labelled data and achieve state-of-the-art results efficiently. We further explore the construction of path complexes from sequential process activity data and the message-passing mechanism built upon their boundary operators. The proposed framework, Path Complex Neural Networks (PCNN), incorporates temporal connections developed from instance graphs. Representations are identified and optimised for 0-paths (events), 1-paths (two events in timely order), and 2-paths (three consecutive events) to characterise intrinsic higher-order information among events. We adopt a similar setting as SCN to implement the individual components of the message-passing mechanism and summarise them with a self-attention mechanism, supporting the framework’s effectiveness. In practice, PCNN with these attributes can achieve state-of-the-art performance compared to other sequential models and graph models. In summary, both SCN and PCNN frameworks leverage the advantages of topological deep learning to obtain representations for higher-order complexities inductively. We propose a novel construction of simplicial complexes and path complexes from sequential data and define representations for their components that participate in message-passing mechanisms. Through evaluation across different benchmark datasets from NLP and BPM with various ablation studies, we highlight the effective incorporation of higher-order objects in sequential data analysis, requiring limited computational power and achieving state-of-the-art performance.