Emergence of hierarchical modes from deep learning

14 June 2023

Presentation Record(Previous Presentation will be showed here if the video is not released for this talk)

Dr. Chan Li

Dr. Chan Li is currently a graduate student in the PMI Laboratory of the School of Physics, Sun Yat-sen University. In the fall of 2023, she will go to the University of California, San Diego to study a Ph.D. in physics. The main research direction is based on physics-inspired deep learning models, using statistical physics methods to analyze the theoretical mechanisms of various neural computing, thereby improving its interpretability. Related work has been published in Physical Review Letters, Physical Review E, Physical Review Research and other journals.

We propose a mode decomposition learning that can interpret the weight matrices as a hierarchy of latent modes. The mode decomposition learning not only saves a significant large amount of training costs but also explains the network performance with the leading modes, displaying a striking piecewise power-law behavior. The modes specify a progressively compact latent space across the network hierarchy, making a more disentangled subspace compared to standard training. Our mode decomposition learning is also studied in an analytic online learning setting, which reveals multiple stages of learning dynamics with a continuous specialization of hidden nodes. Therefore the proposed mode decomposition learning points to a cheap and interpretable route towards the magical deep learning.

Reference:

Chan Li and Haiping Huang. Emergence of hierarchical modes from deep learning. Physical Review Research, 2023, 5 (2): L022011.

李婵目前是中山大学物理学院PMI实验室的在读研究生，2023年秋将前往加州大学圣地亚哥分校攻读物理学 phd。主要研究方向是基于物理启发的深度学习模型，利用统计物理方法分析各类神经计算的理论机制，从而提升其可解释性。相关工作发表在Physical Review Letters, Physical Review E, Physical Review Research 等期刊。

Related Background:

The emergence of hierarchical modes from deep learning refers to the phenomenon where deep neural networks learn to represent data in a hierarchical manner. Deep learning models, such as deep neural networks, are composed of multiple layers of interconnected nodes, allowing them to learn complex patterns and representations.

In deep learning, the lower layers of a deep neural network typically capture low-level features or local patterns, while higher layers capture more abstract and global features. This hierarchical organization arises naturally as the network learns to transform the input data from raw representations to higher-level abstractions.

For example, in image recognition tasks, the lower layers of a deep convolutional neural network (CNN) may learn to detect edges, corners, or textures in the input image. As the signal propagates through the network, higher layers can learn to detect more complex shapes, objects, or even entire scenes. This hierarchical representation allows the network to capture different levels of abstraction and encode rich information about the input data.

The emergence of hierarchical modes is not explicitly programmed into deep learning models but is an inherent property that arises from the learning process. By using techniques such as backpropagation and gradient descent, deep neural networks automatically adjust their internal parameters to optimize the objective function and learn to represent data in a hierarchical manner.

Hierarchical modes have been observed in various domains, including computer vision, natural language processing, and reinforcement learning. They enable deep learning models to learn and reason about complex relationships and dependencies in the data, leading to improved performance in a wide range of tasks.

blog comments powered by Disqus