A Path toward AGI Extractive Summarization as Feature Selection

• Title：MAIB-class-011: A Path toward AGI Extractive Summarization as Feature Selection
• Date：10:00pm US East time, 04/01/2023
• Date：10:00am Beijing time, 04/02/2023
• Zoom ID：933 1613 9423
• Zoom PWD：416262
• Zoom: https://uwmadison.zoom.us/meeting/register/tJcudu-prTIuGNda1MsF8PKyRQlnGn06TP2E

• Momiao Xiong, Ph. D, Professor in Department of Biostatistics snd Data Science , University of Texas, School of Public Health. Dr. Xiong graduated from the Department of Statistics at the University of Georgia in 1993. From 1993 to 1995, Dr. Xiong was postdoctoral fellow at the University of Southern California working with Michael Waterman.

• Research Interest： Causal Inference, Artificial Intelligence , Manifold Learning, Statistic Genetics and Bioinformatics .

• https://theaisummer.com/diffusion-models/#:~:text=Diffusion%20models%20are%20a%20new,to%20train%20large%2Dscale%20models

• https://towardsdatascience.com/understanding-graph-convolutional-networks-for-node-classification-a2bfdb7aba7b

• https://medium.com/red-buffer/implementation-and-understanding-of-graph-neural-networks-gnn-54084c8a0e24

Background

• 1.通用人工智能（AGI）的追求在于更强的泛化能力。泛化能力越强，智能水平越高。

• 2.压缩就是泛化。对于一个数据集最好的无损压缩，就是对于数据集之外的数据最佳泛化。

• 3.GPT预测下一个token的训练任务，等同于对训练数据进行无损压缩。GPT是目前最好的数据无损压缩算法，因此具备最强的智能。压缩即泛化，泛化即智能, 大模型

Summarization as a New Paradigm for Data Reduction

• Extractive Summarization Approach to Feature Selection

• Abstract Summarization Approach to Dimension Reduction

• Protein and DNA Language Models are Extremely Important to Genetics, Population Genetics, Molecular Biology, Clinics and Drug Development.

作为一个人工智能模型，GPT确实具备强大的泛化能力。泛化能力是指一个模型能够从训练数据中学到普遍规律，并能够将这些规律应用到新的、之前没有见过的数据上。这是实现通用人工智能（AGI）的关键能力之一。

压缩确实是一种形式的泛化。通过无损压缩一个数据集，可以从中提取出其中的规律和模式，从而更好地理解这个数据集。因此，无损压缩可以被看作是一种数据的泛化方法。

GPT模型的主要训练任务是对给定文本序列中下一个单词的预测。这个任务可以被看作是对给定数据集进行无损压缩的过程。GPT通过学习语言中的规律和模式来预测下一个单词，从而能够理解文本的含义和结构。因此，GPT确实具备很强的智能能力。

总的来说，压缩确实可以被看作是一种泛化方法，而GPT作为一种数据无损压缩算法，具备很强的智能能力。

The pursuit of AGI is focused on achieving stronger generalization capabilities, which increases intelligence levels. Compression is equivalent to generalization, where the best lossless compression for a dataset is the best generalization for data outside of the dataset. GPT’s task of predicting the next token is equivalent to lossless compression of the training data, making it the best data compression algorithm and therefore possessing the strongest intelligence. Summarization is a new paradigm for data reduction, with extractive summarization serving as a feature selection approach and abstract summarization serving as a dimension reduction approach. Protein and DNA language models are critical in genetics, population genetics, molecular biology, clinics, and drug development.

Replicating the ChatGPT training process requires access to a large dataset of text, high-performance computing resources, and expertise in machine learning and natural language processing. The training process is also highly proprietary and specific to OpenAI’s technology stack, which may not be fully available to the public.

However, there are several open source deep learning frameworks and libraries available that can be used to build and train language models. Some popular options include TensorFlow, PyTorch, and Keras.

To replicate the ChatGPT training process, you would need to:

Acquire a large dataset of text. This could include web pages, news articles, books, and other sources of text. The quality and diversity of the data is critical to the success of the language model.

Preprocess the data to prepare it for training. This includes tokenizing the text, normalizing it, and encoding it in a format that can be used by the deep learning framework.

Choose a deep learning framework and set up a high-performance computing environment to train the model. This may involve using GPU-accelerated hardware, cloud computing resources, or a cluster of machines.

Build a language model architecture based on the transformer architecture used in ChatGPT. This involves designing the model architecture, including the number of layers, attention mechanisms, and other hyperparameters.

Train the model on the dataset using the chosen deep learning framework. This may involve using techniques such as gradient descent, backpropagation, and regularization to optimize the model.

Evaluate the model to determine its accuracy and performance. This may involve using metrics such as perplexity, BLEU score, and human evaluation to assess the model’s ability to generate coherent and meaningful text.

Overall, replicating the ChatGPT training process requires significant expertise in machine learning and natural language processing, as well as access to large amounts of data and computing resources. While there are open source tools and frameworks available, reproducing the ChatGPT model in its entirety may be challenging for most researchers and developers.

我们正在经历人工智能的巨大革命性变化。人们逐渐认识到语言模型和其他人工智能工具实际上是模拟人的智能活动。ChatGPT 不仅能理解自然语言，写作，而且能解答数学问题，编写和执行程序。人工智能一定会改变我们生活和工作的方式，提供解决科学和工程技术问题的新思路，改变从小学，中学到大学，研究生的所有教学内容和教学方式。本次讲课除了介绍一些大型语言模型的一些知识外，主要讨论我们在科学和工程技术方面工作的朋友们如何迎接人工智能的巨大挑战。

相关背景

1. Meta Language Model: 元语言模型，是一种能够学习在多种语言或风格中生成文本的语言模型，或者学习适应不同领域或任务的模型。它还可以指学习从先前的任务中学习以提高未来任务表现的模型。

2.Automatic Multi-step Reasoning: 自动多步推理，是一种机器或人工智能系统能够使用逻辑或概率方法进行多次推断或推理以得出结论的类型。

3.Hierarchically Organized Modules of Thought as a General Framework of Data Analysis: 思维模块的层次结构组织作为数据分析的一般框架，提出人类思维过程由层次结构组织的模块相互作用以产生复杂行为的理论框架。此框架已用于建模复杂的数据分析任务。

4.Embedding for Tabular Values: 表格数值嵌入，是将表格数据（如电子表格或数据库）转换为可用作机器学习模型输入的数字格式的技术。嵌入表示表格中的每个值都是一个数字向量，该向量捕捉它与表格中其他值的关系。

5.Text Summarization as Feature Selection: 将文本摘要作为特征选择，是一种通过从中提取最重要或相关信息来减少文本数据复杂性的技术。这可以通过选择对于给定任务最具信息量的特征（如单词或短语）来实现。

6.Multi-Task Text Summarization and Multi-Omics Data Integration: 多任务文本摘要和多组学数据集成，是一种涉及训练模型同时执行多个任务的机器学习方法，例如文本摘要和整合多种生物数据（如基因组学、蛋白质组学等）。

7.Diffusion Variational Autoencoder: 扩散变分自编码器，是一种使用扩散过程来建模潜在变量分布的生成模型（即捕捉数据的潜在结构的隐藏变量）。这种技术可以用于生成图像或语音等任务。

8.Diffusion GAN: 扩散生成对抗网络，是一种使用扩散过程生成图像的生成对抗网络技术。这种技术可以用于生成具有逼真细节和纹理的高质量图像。

课程中相关的潜在讨论

1. 元语言模型是一种强大的自然语言处理技术，可以用于多语言文本生成、对话系统、文本摘要等应用。与此同时，元语言模型还可以用于多任务学习和迁移学习，使得模型可以适应不同的领域和任务。

2.自动多步推理技术可以用于逻辑推理、概率推理、数据分析等领域。在自然语言处理领域中，多步推理可以用于自然语言推理、知识图谱构建等任务。同时，多步推理还可以结合元语言模型进行多模态推理，使得模型可以更好地理解和处理不同的语言、视觉、声音等信息。

3.思维模块的层次结构组织作为数据分析的一般框架可以用于多种数据分析任务，包括文本摘要、基因组学、蛋白质组学等领域。在文本摘要任务中，思维模块可以用于特征选择，选择最具信息量的特征，以便更好地理解和总结文本。在基因组学和蛋白质组学领域中，思维模块可以用于数据集成和特征选择，以便更好地理解和分析生物数据。

4.嵌入技术是一种将非结构化数据转换为结构化数据的技术，在自然语言处理、图像处理等领域中得到广泛应用。在自然语言处理领域中，嵌入可以用于词向量表示，捕捉词语之间的关系。在数据分析领域中，嵌入可以用于表格数值嵌入，捕捉表格中数值之间的关系。

5.扩散变分自编码器（Diffusion Variational Autoencoder）和扩散生成对抗网络（Diffusion GAN）是一种新兴的生成模型技术，可以用于图像、语音等任务。这些模型使用扩散过程来建模潜在变量分布，从而可以生成具有逼真细节和纹理的高质量图像。这些模型还可以与元语言模型和自动多步推理技术结合使用，以便更好地理解和处理复杂的自然语言和图像信息。

Meta Language Model: A meta language model is a type of language model that can learn to generate text in multiple languages or styles, or learn to adapt to different domains or tasks. It can also refer to models that learn from previous tasks to improve future task performance.

Automatic Multi-step Reasoning: Automatic multi-step reasoning is a type of machine or artificial intelligence system that can use logical or probabilistic methods to make multiple inferences or deductions to reach conclusions.

Hierarchically Organized Modules of Thought as a General Framework of Data Analysis: The hierarchical organization of modules of thought as a general framework of data analysis proposes a theoretical framework that suggests that human thinking processes are organized into interacting modules arranged in a hierarchical structure, which give rise to complex behavior. This framework has been used for modeling complex data analysis tasks.

Embedding for Tabular Values: Embedding for tabular values is a technique for converting tabular data (such as spreadsheets or databases) into a numerical format that can be used as input for machine learning models. The embedding represents each value in the table as a numerical vector that captures its relationships with other values in the table.

Text Summarization as Feature Selection: Text summarization as feature selection is a technique that reduces the complexity of text data by extracting the most important or relevant information. This can be achieved by selecting the most informative features (such as words or phrases) for a given task.

Multi-Task Text Summarization and Multi-Omics Data Integration: Multi-task text summarization and multi-omics data integration is a machine learning approach that involves training models to perform multiple tasks simultaneously, such as text summarization and integrating multiple types of biological data (such as genomics, proteomics, etc.).

Diffusion Variational Autoencoder: A diffusion variational autoencoder is a generative model that uses a diffusion process to model the distribution of latent variables (i.e., hidden variables that capture the underlying structure of data). This technique can be used for tasks such as generating images or speech.

Diffusion GAN: Diffusion generative adversarial network is a generative adversarial network technique that uses a diffusion process to generate high-quality images with realistic details and textures.

Potential discussion topics in the course:

Meta language models are powerful natural language processing techniques that can be used for multilingual text generation, dialogue systems, text summarization, and other applications. At the same time, meta language models can also be used for multi-task learning and transfer learning, enabling models to adapt to different domains and tasks.

Automatic multi-step reasoning techniques can be used in logic reasoning, probabilistic reasoning, data analysis, and other fields. In natural language processing, multi-step reasoning can be used for tasks such as natu