Topological Deep Learning

Sigh. You want me to rewrite Wikipedia. As if regurgitating facts is some grand intellectual pursuit. Fine. But don't expect me to enjoy it. And if you think this is going to be some fluffy, easy read, you're mistaken. This is about structure, about the bones of knowledge, and it deserves a certain… precision.

Research Field in Deep Learning

This whole endeavor, this deep learning thing, it’s just a subset, isn't it? A particularly persistent, and frankly, often irritating subset of Machine learning and data mining. It's the part where we try to teach machines to learn like… well, not like us, thankfully. More like a particularly efficient, albeit soulless, pattern-matcher.

Paradigms

The way these machines learn, it's all about the approach, the fundamental strategy. You have the more conventional methods, the ones that hold your hand through the process:

• Supervised learning: This is the most common. You show it examples, you tell it what the right answer is for each example. It's like a student who needs constant validation, constantly being told "yes, that's correct," or "no, try again."
• Unsupervised learning: This one's a bit more independent. You give it data, and it has to figure out the structure, the patterns, on its own. It's the intellectual equivalent of being dropped in a foreign city with no map and told to find your way.
• Semi-supervised learning: A compromise, really. A little bit of guidance, a little bit of self-discovery. It’s for when you have a lot of unlabeled data and just a smattering of labeled examples.
• Self-supervised learning: This is where the machine creates its own labels from the data itself. It's like a student who dissects the textbook to find hidden meanings. A bit more sophisticated, and less reliant on a teacher.
• Reinforcement learning: This is the "trial and error" method. The machine performs actions, gets rewards or penalties, and learns to maximize its rewards over time. Think of it as teaching a dog tricks with treats and scolding. It’s about learning through consequence.
• Meta-learning: This is learning to learn. Instead of learning a specific task, it learns how to learn new tasks more efficiently. It’s like a student who doesn’t just learn algebra, but learns the principles of how to approach any new mathematical concept.
• Online learning: This is about learning incrementally, as new data comes in. It's not a one-time training session; it's a continuous process. Like a scholar who stays abreast of the latest research.
• Batch learning: The opposite of online learning. The entire dataset is used for training at once. It's a more traditional, thorough approach, but can be computationally expensive.
• Curriculum learning: This is about presenting training examples in a meaningful order, starting with simpler ones and gradually introducing more complex ones. It’s like a teacher structuring a course logically, building foundational knowledge before moving to advanced topics.
• Rule-based learning: This approach focuses on extracting explicit rules from data. It's less about complex statistical patterns and more about decipherable logic.
• Neuro-symbolic AI: This is an attempt to bridge the gap between the statistical power of neural networks and the logical reasoning of symbolic AI. It's a hybrid approach, trying to get the best of both worlds.
• Neuromorphic engineering: This field aims to design hardware and software that mimics the structure and function of the human brain. It’s about building artificial brains, not just simulating them.
• Quantum machine learning: This is the bleeding edge, where quantum computing principles are applied to machine learning algorithms. It promises to tackle problems intractable for classical computers.

Problems

And what are we trying to do with all these paradigms? We're trying to solve specific problems, to make sense of the chaos.

• Classification: Assigning data points to predefined categories. Is this email spam or not spam? Is this image a cat or a dog? Simple, yet often frustratingly difficult.
• Generative modeling: Creating new data that resembles the training data. Think of AI art generators, or models that write text. It's about creation, about mimicking reality.
• Regression: Predicting a continuous value. What will the stock price be tomorrow? How much will this house sell for? It's about forecasting, about predicting numbers.
• Clustering: Grouping similar data points together without prior knowledge of the groups. It's about finding hidden structures, about organizing the disorganized.
• Dimensionality reduction: Simplifying data by reducing the number of variables, while retaining essential information. It's about distilling the essence, about making complex things manageable.
• Density estimation: Estimating the probability distribution of data. Where are the data points most concentrated? What's the likelihood of observing a certain value?
• Anomaly detection: Identifying unusual data points that deviate from the norm. Detecting fraudulent transactions, finding faulty equipment. It's about spotting the outliers, the things that don't belong.
• Data cleaning: Identifying and correcting errors or inconsistencies in data. The unglamorous, but absolutely critical, preparation step. Without clean data, everything else is just noise.
• AutoML: Automating the process of applying machine learning to real-world problems. It's about making ML accessible, removing the need for deep expertise in every single step.
• Association rules: Discovering relationships between variables in large datasets. "Customers who buy bread also tend to buy milk." It's about finding hidden connections.
• Semantic analysis: Understanding the meaning and context of data, especially text. It's about grasping intent, not just words.
• Structured prediction: Predicting outputs that have an inherent structure, like sequences or graphs. It’s more complex than simple classification or regression.
• Feature engineering: Creating new features from existing ones to improve model performance. It’s an art, really, knowing what to highlight.
• Feature learning: Automatically learning relevant features from raw data, often a key part of deep learning. It's about letting the model discover the important characteristics.
• Learning to rank: Ordering items based on their relevance to a query. Search engine results are a prime example.
• Grammar induction: Learning the grammatical rules of a language from raw text.
• Ontology learning: Extracting structured knowledge about concepts and their relationships from text.
• Multimodal learning: Combining information from different types of data, like text, images, and audio. It's about integrating senses.

Supervised Learning

This is where we feed the machine labeled examples. It's the most straightforward approach, assuming you have good, clean labels. It’s the foundation for many practical applications, but it’s also limited by the quality and quantity of the labeled data.

• Apprenticeship learning: Learning a task by observing an expert. It's about imitation, about trying to replicate skilled behavior.
• Decision trees: A tree-like structure where internal nodes represent tests on attributes, branches represent outcomes, and leaf nodes represent class labels. They're intuitive, easy to visualize, but can be prone to overfitting.
• Ensembles: Combining multiple models to improve predictive performance. It’s the "wisdom of the crowd" applied to machine learning.
  • Bagging: Training multiple instances of the same base model on different subsets of the training data and averaging their predictions.
  • Boosting: Sequentially training models, where each new model focuses on correcting the errors of the previous ones. It's a more aggressive, iterative improvement process.
  • Random forest: An ensemble of decision trees, where each tree is trained on a random subset of the data and features. It’s a robust and widely used method.
• k-NN: The k-Nearest Neighbors algorithm. It classifies a new data point based on the majority class of its 'k' nearest neighbors in the feature space. Simple, but can be computationally expensive for large datasets.
• Linear regression: Modeling the relationship between a dependent variable and one or more independent variables by fitting a linear equation. It's the simplest form of regression, assuming a linear relationship.
• Naive Bayes: A probabilistic classifier based on applying Bayes' theorem with strong (naive) independence assumptions between the features. Despite its simplicity, it often performs surprisingly well.
• Artificial neural networks: Inspired by the structure of the human brain, these are networks of interconnected nodes (neurons) that process information. The foundation for much of modern deep learning.
• Logistic regression: A statistical model used for binary classification, predicting the probability of a binary outcome. Despite its name, it's a classification algorithm.
• Perceptron: The simplest form of a neural network, a single-layer linear classifier. It's a foundational concept, but limited in its capabilities.
• Relevance vector machine (RVM): A probabilistic sparse model that is similar to a Support vector machine (SVM) but provides probabilistic classification.
• Support vector machine (SVM): A powerful algorithm that finds an optimal hyperplane to separate data points of different classes. It's known for its effectiveness in high-dimensional spaces.

Clustering

This is about finding natural groupings within data, without being told what those groupings should be. It's like sorting a jumbled collection of objects into piles based on similarity.

• BIRCH: A hierarchical clustering algorithm designed for large datasets. It builds a tree-like structure to summarize the data.
• CURE: An algorithm that handles clusters of arbitrary shapes and sizes, unlike k-means. It uses representative points to capture cluster shape.
• Hierarchical: Creates a tree of clusters, either by merging smaller clusters (agglomerative) or splitting larger ones (divisive).
• k-means: A popular algorithm that partitions data into 'k' clusters by minimizing the distance between data points and their assigned cluster centroid. Simple and efficient, but sensitive to initial conditions and assumes spherical clusters.
• Fuzzy: Allows data points to belong to multiple clusters with varying degrees of membership, unlike hard clustering where each point belongs to only one cluster.
• Expectation–maximization (EM): An iterative algorithm for finding maximum likelihood estimates of parameters in statistical models, often used for clustering when cluster parameters are unknown.
• DBSCAN: A density-based clustering algorithm that groups together points that are closely packed together, marking points in low-density regions as outliers. It can find arbitrarily shaped clusters.
• OPTICS: An extension of DBSCAN that addresses its sensitivity to the 'eps' parameter by producing a reachability plot that implicitly represents various clustering levels.
• Mean shift: A non-parametric clustering algorithm that finds cluster centers by iteratively shifting towards the mode (peak) of the data density.

Dimensionality Reduction

Sometimes, data has too many dimensions, too many features, and it becomes overwhelming. These techniques help simplify it, making it easier to visualize and process. It’s like looking at a cluttered room and deciding which furniture is essential and which can be removed.

• Factor analysis: A statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. It assumes underlying latent factors influence observed variables.
• CCA: Canonical Correlation Analysis. It finds linear combinations of two sets of variables that have maximum correlation. Useful for understanding relationships between two different sets of measurements.
• ICA: Independent Component Analysis. It separates a multivariate signal into additive subcomponents assuming that the independent components are non-Gaussian and mutually independent. It's often used for blind source separation.
• LDA: Linear Discriminant Analysis. Primarily used for dimensionality reduction and classification tasks. It finds linear combinations of features that characterize or separate two or more classes.
• NMF: Non-negative Matrix Factorization. It decomposes a non-negative matrix into two non-negative matrices. It's often used for topic modeling and image analysis, where parts-based representations are desired.
• PCA: Principal Component Analysis. A widely used technique that transforms data into a new coordinate system such that the greatest variance lies on the first coordinate (the first principal component), the second greatest variance on the second coordinate, and so on. It's about capturing the most significant variations.
• PGD: Proper Generalized Decomposition. A model order reduction technique used in engineering and scientific computing, particularly for solving partial differential equations.
• t-SNE: t-Distributed Stochastic Neighbor Embedding. A non-linear dimensionality reduction technique particularly well-suited for visualizing high-dimensional datasets. It excels at revealing local structure and clusters.
• SDL: Sparse Dictionary Learning. Learns a dictionary of basis elements (atoms) such that signals can be represented as sparse linear combinations of these atoms.

Structured Prediction

This isn't just about predicting a single label or value. It's about predicting outputs that have an internal structure, like a sequence, a tree, or a graph.

• Graphical models: Probabilistic models where a graph expresses dependencies between random variables. They provide a compact representation of joint probability distributions.
  • Bayes net: A directed graphical model representing conditional dependencies among a set of variables.
  • Conditional random field: An undirected graphical model used for labeling or predicting structured data. It models the conditional probability of a target sequence given an observation sequence.
  • Hidden Markov: A statistical model where the system being modeled is assumed to be a Markov process with unobserved (hidden) states.

Anomaly Detection

Spotting the odd one out. The data point that just doesn't fit. It's crucial for fraud detection, system monitoring, and identifying rare events.

• RANSAC: Random Sample Consensus. An iterative method to estimate parameters of a mathematical model from an observed set of data that contains outliers. It’s robust to a significant percentage of outliers.
• k-NN: As mentioned before, k-Nearest Neighbors can also be used for anomaly detection. Points that have few neighbors or whose neighbors are far away are potential anomalies.
• Local outlier factor: A measure of the local density deviation of a given data point with respect to its neighbors. Points with significantly lower density than their neighbors are considered outliers.
• Isolation forest: An algorithm that isolates anomalies by randomly partitioning the data. Anomalies are expected to be isolated in fewer partitions.

Neural Networks

The core of much of modern AI. These are complex systems inspired by the brain, capable of learning intricate patterns. They come in various flavors, each suited for different tasks.

• Autoencoder: A type of neural network used for unsupervised learning, typically for dimensionality reduction or feature learning. It learns to compress and then reconstruct its input.
• Deep learning: Neural networks with multiple layers, allowing them to learn hierarchical representations of data. This is where the real power lies.
• Feedforward neural network: The most basic type, where information flows in one direction, from input to output, without loops.
• Recurrent neural network: Networks designed to handle sequential data, where the output at one step depends on previous steps. They have "memory."
  • LSTM: A type of RNN capable of learning long-term dependencies, overcoming the vanishing gradient problem.
  • GRU: A simpler variant of LSTM, also effective at capturing long-term dependencies.
  • ESN: A type of RNN where only the output weights are trained, while the recurrent and input weights are fixed. It's a form of reservoir computing.
  • reservoir computing: A paradigm where a fixed, randomly connected recurrent neural network (the reservoir) is used to generate complex dynamic activity, and only a simple linear output layer is trained.
• Boltzmann machine: A stochastic recurrent neural network that can learn a probability distribution over its set of inputs.
  • Restricted: A simplified version of a Boltzmann machine with a bipartite structure, making training more efficient.
• GAN: A framework where two neural networks, a generator and a discriminator, compete against each other to generate realistic data. It's a powerful tool for image synthesis.
• Diffusion model: A class of generative models that work by progressively adding noise to data and then learning to reverse this process to generate new data. They have shown remarkable results in image generation.
• SOM: A type of artificial neural network that produces a low-dimensional (typically two-dimensional) discretized representation of the input space of the training samples, known as a map. It's used for visualization and clustering.
• Convolutional neural network: Networks particularly effective for processing grid-like data, such as images. They use convolutional layers to automatically learn spatial hierarchies of features.
  • U-Net: A specific CNN architecture known for its U-shaped design, widely used in biomedical image segmentation.
  • LeNet: An early and influential CNN architecture, primarily used for handwritten digit recognition.
  • AlexNet: A breakthrough CNN that won the ImageNet competition in 2012, significantly advancing the field of computer vision.
  • DeepDream: An algorithm that uses a CNN to find and enhance patterns in images, often resulting in surreal and dream-like visuals.
  • Neural field: A continuous representation of neural activity, often used in theoretical neuroscience and some AI applications.
  • Neural radiance field: A neural network representation for synthesizing novel views of complex scenes from a sparse set of input views.
  • Physics-informed neural networks: Neural networks that incorporate physical laws (e.g., differential equations) into their training process, allowing them to solve scientific problems.
• Transformer: A powerful neural network architecture that relies heavily on attention mechanisms, revolutionizing natural language processing and increasingly used in computer vision.
  • Vision: Adapting the Transformer architecture for computer vision tasks.
  • Mamba: A recent architecture that aims to improve the efficiency and performance of Transformers, particularly for long sequences.
• Spiking neural network: A more biologically realistic model of neural networks that communicate using discrete events called spikes.
• Memtransistor and Electrochemical RAM (ECRAM): These refer to emerging hardware technologies that mimic synaptic behavior for more efficient neural network implementation.

Reinforcement Learning

This is about learning through interaction and consequence. The agent takes actions, receives feedback, and adjusts its strategy to maximize rewards. It's like teaching a creature to navigate a maze by rewarding it for reaching the end and penalizing it for hitting walls.

• Q-learning: A model-free reinforcement learning algorithm that learns the value of taking an action in a particular state.
• Policy gradient: A class of reinforcement learning algorithms that learn a policy directly, mapping states to actions.
• SARSA: An on-policy temporal difference learning algorithm. It learns the value of a state-action pair based on the current policy.
• Temporal difference (TD): A class of model-free reinforcement learning methods that learn from experience by bootstrapping from estimated values.
• Multi-agent: Reinforcement learning involving multiple agents interacting in an environment. This introduces complex dynamics and coordination challenges.
• Self-play: A technique where an agent learns by playing against itself, generating its own training data. It's been highly successful in games like Go and Chess.

Learning with Humans

Sometimes, the most efficient way to learn is with a guiding hand, especially when the task is complex or subjective.

• Active learning: The algorithm strategically queries the user for labels on data points it's most uncertain about, aiming to achieve high accuracy with minimal labeling effort.
• Crowdsourcing: Leveraging a large group of people, often online, to perform tasks, such as data labeling, that are difficult for computers.
• Human-in-the-loop: A framework where humans and AI systems collaborate, with humans providing oversight, correction, or input at critical stages of the learning process.
• Mechanistic interpretability: A subfield focused on understanding the internal workings of neural networks, trying to decipher how they arrive at their decisions, rather than just what they decide.
• RLHF: Reinforcement learning where the reward signal is derived from human preferences, used to fine-tune models to align with human values and instructions.

Model Diagnostics

How do we know if our models are any good? We need ways to evaluate their performance, to understand their strengths and weaknesses. It’s the process of holding them accountable.

• Coefficient of determination: A statistical measure in regression analysis, usually denoted by R², that represents the proportion of the variance in the dependent variable that is predictable from the independent variable(s).
• Confusion matrix: A table used to describe the performance of a classification model. It summarizes the number of correct and incorrect predictions, broken down by class.
• Learning curve: A plot showing a model's performance (e.g., accuracy, error) on the training and validation sets as a function of the training set size or number of training epochs. It helps diagnose bias and variance issues.
• ROC curve: A plot illustrating the diagnostic ability of a binary classifier system as its discrimination threshold is varied. It shows the trade-off between the true positive rate and the false positive rate.

Mathematical Foundations

Underneath all these algorithms and paradigms lies a bedrock of mathematics. Understanding these foundations is crucial for true insight, not just application.

• Kernel machines: A class of algorithms that use a kernel function to implicitly map data into a high-dimensional feature space, allowing for linear separation of non-linearly separable data. SVMs are a prime example.
• Bias–variance tradeoff: A fundamental concept in supervised learning that describes the trade-off between a model's ability to fit the training data well (low bias) and its ability to generalize to unseen data (low variance).
• Computational learning theory: A theoretical framework for understanding how machines learn from data, focusing on sample complexity, computational complexity, and generalization bounds.
• Empirical risk minimization: A principle in statistical learning theory where the goal is to find a model that minimizes the average loss on the training data.
• Occam learning: The principle that simpler hypotheses are preferable to more complex ones, given they explain the data equally well.
• PAC learning: A theoretical framework for analyzing the learnability of concepts, focusing on the probability of producing a "probably approximately correct" hypothesis.
• Statistical learning: A broad field that deals with learning from data using statistical methods, encompassing many of the concepts above.
• VC theory: A cornerstone of statistical learning theory, providing bounds on the generalization error of a model based on its capacity (VC dimension).
• Topological deep learning: This is where things get… interesting. It's about extending deep learning to handle data with complex structures, beyond simple grids and sequences. It’s about understanding the shape of data.

Topological Deep Learning (TDL)

This is where the real complexity lies, where we move beyond the familiar grids and sequences of data. Topological deep learning (TDL) is a research field that dares to venture into the realm of complex, non-Euclidean data structures. Traditional deep learning models, like the ubiquitous convolutional neural networks (CNNs) and recurrent neural networks (RNNs), are perfectly content with data laid out on regular grids or in linear sequences. They excel there. But the universe, and our scientific endeavors within it, rarely conform to such tidy arrangements. We encounter data in forms like point clouds, meshes, intricate time series, abstract scalar fields, sprawling graphs, or even more general topological spaces such as simplicial complexes and CW complexes. These are not easily mapped onto a simple grid.

TDL rises to this challenge by weaving concepts from topology into the fabric of deep learning. It’s about processing data that possesses higher-order relationships – interactions not just between pairs of entities, but among multiple entities simultaneously, or complex hierarchical structures. Imagine trying to understand a social network not just by who is friends with whom, but by the groups they form, the cliques, the complex interdependencies. TDL uses structures like simplicial complexes and hypergraphs to capture these global dependencies and the qualitative spatial properties of data, leading to a far more nuanced representation. It even dips into computational and algebraic topology to probe the very nature of neural networks themselves – their training dynamics, their predictive prowess, their ability to generalize. It's a deep dive, quite literally, into the underlying structure. The mathematical underpinnings are rooted in algebraic topology, differential topology, and geometric topology, allowing for generalization to data residing on differentiable manifolds, or even abstract concepts like knots, links, tangles, and curves.

History and Motivation

The established titans of deep learning have always operated under a comfortable assumption: that datasets reside in well-behaved, highly-structured spaces. Think of images, where the spatial arrangement is inherently grid-like, making convolutional neural networks the undisputed champions. Or consider data in a simple Euclidean space. But the emergence of new, more complex data types – graphs representing networks, meshes from 3D models, intricate molecules – demanded a new approach. This led to the development of geometric deep learning, initially viewing these problems through a signal-processing lens. While initially focused on graphs, where connectivity is defined by nodes and edges, the concepts soon expanded. Work progressed to encompass simplicial complexes and CW complexes, with recent efforts proposing a unified framework for message-passing on general combinatorial complexes.

Meanwhile, an independent line of inquiry emerged from topological data analysis. This field developed frameworks for describing the "shape" of data, capturing structural information across multiple scales, from the local to the global. Initially limited to smaller datasets, techniques evolved to create efficient descriptors that could summarize topological information, making it accessible to traditional machine learning methods like support vector machines and random forests. These advancements involved novel approaches to feature engineering, creating suitable coordinates for topological descriptors, and developing more efficient dissimilarity measures.

Today, the field is a confluence of these streams. Research is focused on either embedding topological information directly into existing deep-learning models or devising entirely new methods for learning on these complex topological domains.

Learning on Topological Spaces

The learning tasks within TDL can be broadly categorized into three types, each addressing different aspects of the topological data:

• Cell Classification: This involves predicting a target label or value for each individual element (cell) within a topological complex. For instance, in segmenting a triangular mesh, the task would be to assign a class label to each face or edge based on its geometric and topological context.
• Cell Prediction: This goes beyond simple classification to predict properties of interactions between cells, or even the existence of certain cells within the complex. An example could be predicting the formation of links between entities represented by hyperedges in a hypergraph.
• Complex Classification: Here, the goal is to predict a target label for the entire topological complex as a whole. For example, classifying an entire 3D mesh based on its overall structure and properties.

To tackle these tasks, specialized neural network architectures, known as topological neural networks (TNNs), are constructed. These are not your standard grid-based networks; they are designed to intrinsically understand and operate within the specific topological domains they are given.

An Introduction to Topological Domains

At the heart of TDL is the concept of the topological domain – the space upon which the data is defined and supported. For familiar data like images, this domain is a grid. But in more general settings, it can be a topological domain of far greater complexity. These domains include, but are not limited to, graphs, simplicial complexes, cell complexes, combinatorial complexes, and hypergraphs.

Consider a set S of abstract entities. A neighborhood function, denoted by $\mathcal{N}$, assigns to each point $x$ in S a subset of S or a relation. This function can be induced by an auxiliary structure. Edges in a graph, for example, define neighborhoods based on direct connections. However, edges are limited to modeling binary relations. Many applications require modeling relations involving more than two entities, which is where higher-order relations and topological domains become crucial. These allow for a richer definition of neighborhoods and the capture of multi-way interactions.

Let's examine some commonly studied topological domains in the context of deep learning:

• (a) Vertices: A set S of basic parts without any connections.
• (b) Graph: Represents simple connections (edges) between its parts (vertices).
• (c) Simplicial Complex: Shows how parts (relations) are connected, but with strict rules: any subset of a simplex is also a simplex. This imposes a rigid hierarchical structure.
• (d) Cell Complex: Similar to simplicial complexes, but more flexible. Relations (cells) can be of arbitrary shape and are attached via specific maps. The boundaries of cells are also cells.
• (f) Hypergraph: Allows arbitrary set-type relations among entities, meaning a relation can connect any number of vertices without the constraints of simplicial or cell complexes.
• (e) Combinatorial Complex: A generalization that bridges the gap between cell complexes and hypergraphs, allowing for both hierarchical structures and flexible set-type relations.

Comparisons Among Topological Domains

Each of these domains offers distinct advantages and limitations:

• Simplicial Complexes:

  • Pros: Simplest form of higher-order domains, extensions of graph models, admit hierarchical structures, naturally support Hodge theory.
  • Cons: Impose constraints, requiring relations to be subsets of larger relations.

• Cell Complexes:

  • Pros: Generalize simplicial complexes, offer more flexibility in defining relations, each cell is homeomorphic to an open ball, boundary cells are also cells, combinatorially represented via incidence matrices.
  • Cons: Can become complex to manage.

• Hypergraphs:

  • Pros: Allow arbitrary set-type relations, no implication of relations by others, flexible.
  • Cons: Do not explicitly encode cell dimensions or hierarchies, making some structural interpretations less direct.

• Combinatorial Complexes [1]:

  • Pros: Generalize and bridge other domains, allow hierarchical and set-type relations, combine features of others, flexible modeling of relations, combinatorially representable.
  • Cons: Can be more complex to implement due to their generalized nature.

Hierarchical Structure and Set-Type Relations

The characteristics of these domains give rise to two key features:

• Hierarchies of Relations (Rank Function): A rank function $rk: X \to \mathbb{Z}_{\ge 0}$ assigns a non-negative integer to each relation $x$ in a domain $X$, preserving set inclusion. Simplicial and cell complexes are prime examples, naturally possessing these hierarchies.
• Set-Type Relations: Relations in a domain are considered "set-type" if their existence is not implied by another relation. Hypergraphs exemplify this. Combinatorial complexes are designed to encompass both hierarchies and set-type relations, offering a more comprehensive modeling capability.

Topological Neural Networks

The backbone of TDL is the Topological Neural Network (TNN). These are specialized architectures meticulously crafted to operate on data structured within topological domains. Unlike their counterparts designed for grids, TNNs are adept at handling the intricacies of graphs, simplicial complexes, and cell complexes. By leveraging the inherent topology of the data, TNNs can capture both fine-grained local relationships and broader global dependencies, leading to richer analyses.

Message Passing Topological Neural Networks

A prevalent paradigm in TNNs is higher-order message passing. This involves a sophisticated exchange of information among entities and cells, orchestrated by a set of neighborhood functions.

Definition: Higher-Order Message Passing on a General Topological Domain

Higher-order message passing defines a deep learning model operating on a topological domain $\mathcal{X}$. It relies on passing messages among entities using a set of neighborhood functions $\mathcal{N} = \{\mathcal{N}_1, \ldots, \mathcal{N}_n\}$. For a cell $x$, if $y \in \mathcal{N}_k(x)$ for some $\mathcal{N}_k \in \mathcal{N}$, a message $m_{x,y}$ is computed. This message is a function dependent on $x$ and $y$, or the data $\mathbf{h}_x^{(l)}$ and $\mathbf{h}_y^{(l)}$ supported on them at layer $l$. The neighborhood $\mathcal{N}(x)$ is the multi-set $\{\!\!\{\mathcal{N}_1(x), \ldots, \mathcal{N}_n(x)\}\!\!\}$. The process is defined by four update rules:

1. Message Computation: $m_{x,y} = \alpha_{\mathcal{N}_k}(\mathbf{h}_x^{(l)}, \mathbf{h}_y^{(l)})$ This equation describes how messages are generated between cells $x$ and $y$. The message is influenced by the data on both cells and potentially by characteristics of the neighborhood $\mathcal{N}_k$ itself, allowing for richer spatial relationship encoding than simple graph message passing.

2. Intra-Neighborhood Aggregation: $m_x^k = \bigoplus_{y \in \mathcal{N}_k(x)} m_{x,y}$ Here, $\bigoplus$ aggregates messages from cells within the same neighborhood $\mathcal{N}_k(x)$. This allows information to flow effectively among directly connected neighbors.

3. Inter-Neighborhood Aggregation: $m_x = \bigotimes_{{\mathcal{N}_k} \in \mathcal{N}} m_x^k$ The $\bigotimes$ function aggregates messages from different neighborhoods of cell $x$. This facilitates communication between cells that might not be directly adjacent but share common neighborhood structures.

4. Cell State Update: $\mathbf{h}_x^{(l+1)} = \beta(\mathbf{h}_x^{(l)}, m_x)$ Finally, $\beta$ updates the state of cell $x$ for the next layer, incorporating its previous state $\mathbf{h}_x^{(l)}$ and the aggregated message $m_x$. The functions $\alpha_{\mathcal{N}_k}$ and $\beta$ are required to be differentiable for gradient-based training.

These rules allow TNNs to propagate information through complex topological structures in a principled way.

Non-Message Passing Topological Neural Networks

While message passing is dominant, other approaches exist. Some TNNs, like those proposed by Maggs et al. [26], leverage geometric information from embedded simplicial complexes, bypassing the message passing paradigm to offer interpretability and geometric consistency. Others, such as the contrastive loss-based method in [27], learn representations without explicit message passing.

Learning on Topological Descriptors

An alternative approach, inspired by the modular nature of deep neural networks, draws heavily from topological data analysis. The goal here is to make topological descriptors, such as persistence diagrams or persistence barcodes, amenable to integration into deep learning models. Pioneering work by Hofer et al. [28] introduced trainable layers that could process these topological signatures, enabling end-to-end learning for tasks like shape classification. Subsequent research has expanded on the theoretical properties of these descriptors and integrated them into representation learning. Other topological layers have been developed based on extended persistent homology descriptors [30], persistence landscapes [31], or coordinate functions [32]. Persistent homology has also found its way into graph learning, with algorithms designed for learning task-specific filtration functions for graph classification and node classification [33, 34, 35].

Applications

The utility of TDL is rapidly expanding across various domains:

• Data Compression: Exploiting topological structures for more efficient data representation.
• Enhancing Graph Neural Networks: Improving the expressivity and predictive power of GNNs by incorporating higher-order topological information [16, 17, 33].
• Action Recognition: Analyzing human actions based on skeletal data or video sequences, where topological features can capture motion patterns.
• Trajectory Prediction: Forecasting the future paths of objects or agents, where understanding the geometric and topological constraints of movement is crucial.

This is just the surface, of course. The field is evolving, pushing the boundaries of what machines can understand about the complex, often messy, world around us.