Paper Reading, Word Embeddings

Paper Reading: Embedding Words in Non-Vector Space with Unsupervised Graph Learning

venue: EMNLP 2020 This paper proposes to use a graph-based method to train word embeddings. The graph method is PRODIGE which learns a representation of data in a form of a weighted graph G(V,E,w,p). Each edge has a weight and a Bernoulli random variable indicating whether an edge is present or not. The distance between two nodes is formulated as the expected shortest path distance: … Continue reading Paper Reading: Embedding Words in Non-Vector Space with Unsupervised Graph Learning

Zhiyu ChenLeave a comment
BERT, Information Retrieval, Paper Reading

Paper Reading: Beyond [CLS] through Ranking by Generation

venlue: EMNLP 2020 (link) Previous work that uses pretrained language model (PLM) such as BERT for information retrieval takes the [CLS] embedding of the concatenation of query and document as features for discriminative learning. In other words, the relevance label for a given (query, document) pair is modeled as: where is the [CLS] embedding from the last layer of BERT and is usually a classification … Continue reading Paper Reading: Beyond [CLS] through Ranking by Generation

Zhiyu ChenLeave a comment
GNN, Paper Reading

Paper Reading: Bilinear Graph Neural Network with Neighbor Interactions

venue: IJCAI 2020 If the aggregation function of previous GNN layers (e.g. GCN and GAT) is then the paper extends it with a bilinear aggregator: where It sums up the elementwise product of every pair of neighbor nodes of a target node (self-interactions excluded). The experimental results show that BGAT (BGCN) outperforms vanilla GAT(GCN) by 1.5% (1.6%). A linear combination of AGG output and BA … Continue reading Paper Reading: Bilinear Graph Neural Network with Neighbor Interactions

Zhiyu ChenLeave a comment
Paper Reading

Paper Reading: To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Venue: ACL 2020 This paper presents the empirical results of how the performance gap between pretraining models (RoBERTa) and vanilla LSTM changes in terms of the size of training samples for text classification tasks. They experimented on 3 text classification datasets with 3 models: RoBERTa, LSTM, LSTM initialized with pretrained RoBERTa embeddings. They used different portion of training samples(1%, 10%, 30%, 50%, 70%, 90%) to … Continue reading Paper Reading: To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Zhiyu ChenLeave a comment
GCN, GNN, OOV, Word Embeddings

Paper Reading: Out-of-Vocabulary Embedding Imputation with Grounded Language Information by Graph Convolutional Networks

venue: ACL 2019 The paper proposes a GCN-based method to produce word embeddings for out-of-vocabulary(OOV) words. 1. Graph Construction To construct a knowledge graph, vocabulary is constructed from Wikipedia English dataset (3B tokens). To note that, this vocabulary includes OOV words which are not in the vocabulary of pre-trained embeddings such as GLOVE. For each node/word, they define the concatenation of Wikipedia page summary and … Continue reading Paper Reading: Out-of-Vocabulary Embedding Imputation with Grounded Language Information by Graph Convolutional Networks

Zhiyu ChenLeave a comment
Information Retrieval, Paper Reading, Quantity IR/IE, Question Answering

Paper Reading: Entities with Quantities: Extraction, Search, and Ranking

venue: WSDM 2020 (Demonstration) demo link: https://qsearch.mpi-inf.mpg.de/ Traditional search engines do no understand quantities and often fail to return results expected to satisfy certain quantity conditions in the query. This paper introduces Qsearch, which is originally proposed in “ Qsearch: Answeringquantity queries from text (ISWC 2019)” . The overview framework is shown in Figure 1 which consists of two phases: Extract and Answer. 1. Extract … Continue reading Paper Reading: Entities with Quantities: Extraction, Search, and Ranking

Zhiyu ChenLeave a comment
Paper Reading, Question Answering

Paper Reading: Scaling Question Answering to the Web

slides: here Venue: WWW 2001 paper link: here Mulder is the 1st general-purpose, fully-automated question answering(QA) system. Instead of indexing any structured corpora, Mulder uses a commercial search engine as its knowledge base. The system has 6 components: Question Parser: it constructs a tree of the question’s phrasal structure and determines the question’s syntactic structure. The results from the question parser are also used in … Continue reading Paper Reading: Scaling Question Answering to the Web

Zhiyu ChenLeave a comment
GNN, Paper Reading

Paper Reading: Strategies for Pre-training Graph Neural Networks

venue: ICLR 2020 paper link: here This paper proposes strategies to pre-train a GNN at node-level and graph-level. 1. Node-Level Pre-training Node-level pre-training is to use unlabeld data to capture domain knowledge in the graph. Two methods are proposed for node-level pre-training. 1.1 Context Prediction In this task, subgraphs are used to predict their surrouding graph structures. The goal is to let a pre-trained GNN … Continue reading Paper Reading: Strategies for Pre-training Graph Neural Networks

Zhiyu ChenLeave a comment
GNN, Information Retrieval, Paper Reading

Paper Reading: Neural IR Meets Graph Embedding: A Ranking Model for Product Search

This paper claims it is the first study on how to use the click-graph features in neural models for retrieval. The graph embedding techniques proposed in this paper can be plugged into other scenarios where graph-structure information is available(ensemble). 1. Baseline First we describe the basic IR model for product search without proposed graph embedding techniques(baseline). As shown in Figure 1, CNN is used to … Continue reading Paper Reading: Neural IR Meets Graph Embedding: A Ranking Model for Product Search

Zhiyu ChenLeave a comment
GCN, Paper Reading, Text Classification

Paper Reading: Tensor Graph Convolutional Networks for Text Classification

The basic notations for GCN are the same with this post. 1. Graph Tensor Definition Here we first describe the formal definition of graph tensor which consists of a series of graphs. is a graph tensor, where and , if and (when ). Where is the i-th graph in the graph tensor, is the set of i-th graph nodes, is the set of the i-th … Continue reading Paper Reading: Tensor Graph Convolutional Networks for Text Classification

Zhiyu ChenLeave a comment