NLPHotTopic
This week i get a summary knowledge of NLP, and learn some direction for further learning. And in this blog, i will record what i learned this weak by searching some information on Internet, the content is organized as follows: the Preparatory knowledge which need to be master in the following years, and some direction in NLP areas from model sides, application sides and the scene task, and some paper and learning resource recording.
0. Preparatory knowledge
- Probability& Statistics
-
Machine Learning
-
Text Mining
- NLP
1. Model sides
1.1. Transformers and pre-trained language models
- “Attention is all you need” (Vaswani et al., 2017)
- “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (Devlin et al., 2018)
Theory-proving side: (Shi et al., 2020; Brunner et al., 2020; Yun et al., 2019; Cordonnier et al., 2019).
improving the task performances of Transformers and pre-trained language models:(Wang et al. 2019; Lee et al., 2019).
Reducing the size of models or the time of training:(Wu et al., 2020; Lan et al., 2019; Kitaev et al., 2020; Clark et al., 2020; Rae et al., 2019; Fan et al., 2019; You et al., 2019).
- Model Compression/Pruning (Zhang et al., 2019)
1.2. Multilingual/Cross-lingual tasks:
- (Karthikeyan et al., 2019; Berend 2020; Cao et al., 2020; Wang et al., 2019)
- Multimodal models(Su et al., 2019)
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- On the Relationship between Self-Attention and Convolutional Layers
1.3. Reinforcement learning and NLP
Session 4：The Machine Learning in NLP
Learning Sparse Sharing Architectures for Multiple Tasks
Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
Shapley Q-value: A Local Reward Approach to Solve Global Reward Games
Measuring and relieving the over-smoothing problem in graph neural networks from the topological view
Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning
Neural Snowball for Few-Shot Relation Learning
Multi-Task Self-Supervised Learning for Disfluency Detection
Constructing Multiple Tasks for Augmentation: Improving Neural Image Classification With K-means Features
Graph-propagation based correlation learning for fine-grained image classification
End-to-End Bootstrapping Neural Network for Entity Set Expansion
2. Application sides
2.1. Natural language generation
- Generation of realistic, rhymed and theme based poetry (creative writing)
- Generation of theme based short stories (creative writing)
- Generation of theme based novels (creative writing)
- Generation of news / short articles based on numerical / audio / video data
- Generation of research papers based on a topic.
2.2. Natural language understanding
- Sentiment Analysis
Deriving sentiments in sentences (positive, negative, neutral), and also in articles (though that will be more appropriate like bag of sentence sentiments). The future is to include emotions (attributes) in that, like the attributes now on Facebook posts - Love, Like, Angry, Surprised, Sad, Hilarious. These attributes make a lot more sense for sentiments going forward.
- Text Summarization（汇总）
Summarizing a single or many articles according to a particular theme.
- Textual entailment（语篇蕴涵）
Inferring directional causal relationships between textual fragments. This can be challenging in a long article.
- Towards Building a Multilingual Sememe Knowledge Base: Predicting Sememes for BabelNet Synsets
- Multi-Scale Self-Attention for Text Classification
- Learning Multi-level Dependencies for Robust Word Recognition
- Information Extraction or Relationship Extraction or Knowledge Graph
Find structured information from unstructured data, like entities, relationships, co-reference resolution. This at a basic level is very useful for algorithmic trading. An extension of this is a global form of extracting logic structures (first order and higher order).
- Topic Segmentation
Topic Extraction (with regions). Normally, there will be overlapping regions.
- Question Answering or NLP-based voice assistant
Answer the questions to both closed (specific) and open questions (subjective). Answers to subjective questions is the main challenge for the likes of realistic Virtual Assistants.
- Modeling Fluency and Faithfulness for Diverse Neural Machine Translation
- Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
- Neural Machine Translation with Joint Representation
- Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context
- A pre-training based personalized dialogue generation model with persona-sparse data
- Knowledge Graph Grounded Goal Planning for Open-Domain Conversation Generation
- Parsing
Parsing natural language generally in the form a tree. This involves hierarchical segmentation of the language involving the grammar rules.
- Prediction
Given a short text, predict what happens next. The prediction problem is beginning to be targeted in vision, but it has never ever gained paths for realistic products. For closed and deterministic prediction (not innovative else that would fall under the paradigm of creative writing), this can be a useful task for prediction of future events based on past evidences and analysis. This can be then very useful for finance sectors.
- Part of Speech Tagging(词性标注)
Tagging words whether they are nouns, verbs or adjectives.
- Translation
Translate one language to another. This can be very challenging given the nature of the language, and the grammar. Normally, under probabilistic models, this assumes that the underlying grammar is mostly the same, and thus, models normally fail for Sanskrit.
- Query Expansion
Expand query in possible ways for making the search results more meaningful. This is normally an issue with search engines, where people do not know what all keywords (or query sentences) to include to cover the entire gamut of relevancy.
- Argumentation Mining(论证分析挖掘）
Evolving field of NLP, where one wants to analyse discussions and arguments.
- Interestingness(趣味性挖掘）
2. 3. NLP and CV
- Visual Question Answering
- Automated Image Captioning（自动图像字幕）
- OCR
- DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
- Storytelling from an Image Stream Using Scene Graphs
2.4. Voice and NLP
- speech to text
Analysts predict speech recognition technologies will be substantially improved in the near future thanks to natural language processing. This will involve minimization of errors, recognition of what several individuals are saying despite different accents and a noisy environment.
3. Scene task
-
Integrated Chatbot
-
Human-to-machine Interaction
conversing with a machine is as simple as conversing with a human.
- Company monitoring
Banks and other monetary organizations can utilize NLP to find and parse client sentiment by checking social media and analyzing discussions about their services and strategies. With the capacity to get to significant, separated data, financial services analysts can compose increasingly definite reports and give better advice to customers and internal decision makers.
- Business intelligence
getting business intelligence from raw business information, including product information, marketing and sales information, customer service, brand notoriety and the present talent pool of a company. This implies NLP will be the way to moving numerous legacy organizations from data-driven to intelligence-driven platforms, helping humankind rapidly get the insights to make decisions.
搜索是NLP技术最早得到大规模应用的技术，例如百度搜索、知乎话题搜索以及各大互联网公司的query搜索技术，都涉及到语义匹配或文本分类技术。此外，大型的搜索引擎，知识图谱的搭建是必须的。
推荐系统在一定层面来说是跟搜索场景相反的。搜索是基于用户的意图，在文本库中寻找匹配项；推荐则相反，通常基于积累的用户信息，给用户推荐可能感兴趣的内容。推荐系统常常涉及用户画像、标签定义等过程，需要一定程度的依赖NLP技术。
聊天机器人是目前NLP技术应用最多的场景，基于NLP技术构建一个能够替代客服、销售、办公文员是这一任务的终极目标。目前，聊天机器人已经以各种形态出现在人们面前，有站在银行门口迎接顾客的迎宾机器人，有放在卧室床头的智能音箱，有呆在各个APP首页的助手机器人等等。在聊天机器人中，运用了文本分类、语义匹配、对话管理、实体识别等大量的NLP技术。要做好是一件难度大、超复杂的任务。
知识图谱是AI时代一个非常重要基础设施，大规模结构化的知识网络的搭建，能够重塑很多的智能场景。
4. Paper & Relative Article
4.1. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
About: In this paper, researchers from Carnegie Mellon University and Google Brain proposed a novel neural architecture known as Transformer-XL that enables learning dependency beyond a fixed-length without disrupting temporal coherence. According to the researchers, TransformerXL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation.
4.2. Bridging The Gap Between Training & Inference For Neural Machine Translation
About: This paper is one of the top NLP papers from the premier conference, Association for Computational Linguistics (ACL). This paper talks about the error accumulation during Neural Machine Translation. The researchers addressed such problems by sampling context words, not only from the ground truth sequence but also from the predicted sequence by the model during training, where the predicted sequence is selected with a sentence-level optimum. According to the researchers, this approach can achieve significant improvements in multiple datasets.
4.3. BERT: Pre-training Of Deep Bidirectional Transformers For Language Understanding
BERT by Google AI is one of the most popular language representation models. Several organisations, including Facebook as well as academia, have been researching NLP using this transformer model. BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations from the unlabeled text by jointly conditioning on both left and right context in all layers. The model obtained new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5%, MultiNLI accuracy to 86.7%, and much more.
4.4. Emotion-Cause Pair Extraction: A New Task To Emotion Analysis In Texts
Emotion cause extraction (ECE) is a task that is aimed at extracting the potential causes behind certain emotions in text. In this paper, researchers from China proposed a new task known as emotion-cause pair extraction (ECPE), which aims to extract the potential pairs of emotions and corresponding causes in a document. The experimental results on a benchmark emotion cause corpus that prove the feasibility of the ECPE task as well as the effectiveness of this approach.
4.5. Improving Language Understanding By Generative Pre-Training
This paper is published by OpenAI, where the researchers talked about natural language understanding and how it can be challenging for discriminatively trained models to perform adequately. The researchers demonstrated the effectiveness of the approach on a wide range of benchmarks for natural language understanding. They proposed a general task-agnostic model, which outperformed discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon state-of-the art in 9 out of the 12 tasks studied.
4.6. Neural Approaches To Conversational AI
This research paper by Microsoft Research surveys neural approaches to conversational AI that have been developed in the last few years. In this paper, the researchers grouped conversational systems into three categories, which are question answering agents, task-oriented dialogue agents, and chatbots. For each category, a review of state-of-the-art neural approaches is presented, drawing the connection between them and traditional approaches, as well as discussing the progress that has been made and challenges still being faced, using specific systems and models as case studies.
Session 1：翻译、对话与文本生成
(1) Modeling Fluency and Faithfulness for Diverse Neural Machine Translation
(2) Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
(3) Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context
(4) A pre-training based personalized dialogue generation model with persona-sparse data
(5) Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
(6) SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Bases
(7) Knowledge Graph Grounded Goal Planning for Open-Domain Conversation Generation
(8) Neural Machine Translation with Joint Representation
Session 2：文本分析与内容挖掘
(9) Multi-Scale Self-Attention for Text Classification
(10) Learning Multi-level Dependencies for Robust Word Recognition
(11) Towards Building a Multilingual Sememe Knowledge Base: Predicting Sememes for BabelNet Synsets
(12) Cross-Lingual Low-Resource Set-to-Description Retrieval for Global E-Commerce
(13) Integrating Relation Constraints with Neural Relation Extractors
(14) Capturing Sentence Relations for Answer Sentence Selection with Multi-Perspective Graph Encoding
(15) Replicate, Walk, and Stop on Syntax: an Effective Neural Network Model for Aspect-Level Sentiment Classification
(16) Cross-Lingual Natural Language Generation via Pre-Training
Session 3：知识理解与NLP应用
(17) Hyperbolic Interaction Model For Hierarchical Multi-Label Classification
(18) Multi-channel Reverse Dictionary Model
(19) Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement
(20) Logo-2K+: A Large-Scale Logo Dataset for Scalable Logo Classification
(21) DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialog
(22) DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
(23) Storytelling from an Image Stream Using Scene Graphs
(24) Draft and Edit: Automatic Storytelling Through Multi-Pass Hierarchical Conditional Variational Autoencoder
【NLP】GPT：第一个引入Transformer的预训练模型
【NLP】XLnet：GPT和BERT的合体，博采众长，所以更强
【每周NLP论文推荐】 NLP中命名实体识别从机器学习到深度学习的代表性研究
【文本信息抽取与结构化】目前NLP领域最有应用价值的子任务之一
【NLP-ChatBot】搜索引擎的最终形态之问答系统（FAQ）详述
【知识图谱】人工智能技术最重要基础设施之一，知识图谱你该学习的东西