Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6708–6718July 5 - 10, 2020. c©2020 Association for Computational Linguistics

6708

Document Modeling with Graph Attention Networks for Multi-grainedMachine Reading Comprehension

Bo Zheng1∗, Haoyang Wen1, Yaobo Liang2, Nan Duan2,Wanxiang Che1†, Daxin Jiang3, Ming Zhou2, Ting Liu1

1Harbin Institute of Technology, Harbin, China2Microsoft Research Asia, Beijing, China

3STCA NLP Group, Microsoft, Beijing, China{bzheng,hywen,car,tliu}@ir.hit.edu.cn

{yalia,nanduan,djiang,mingzhou}@microsoft.com

Abstract

Natural Questions is a new challenging ma-chine reading comprehension benchmark withtwo-grained answers, which are a long answer(typically a paragraph) and a short answer (oneor more entities inside the long answer). De-spite the effectiveness of existing methods onthis benchmark, they treat these two sub-tasksindividually during training while ignoringtheir dependencies. To address this issue, wepresent a novel multi-grained machine read-ing comprehension framework that focuses onmodeling documents at their hierarchical na-ture, which are different levels of granularity:documents, paragraphs, sentences, and tokens.We utilize graph attention networks to obtaindifferent levels of representations so that theycan be learned simultaneously. The long andshort answers can be extracted from paragraph-level representation and token-level represen-tation, respectively. In this way, we can modelthe dependencies between the two-grained an-swers to provide evidence for each other. Wejointly train the two sub-tasks, and our exper-iments show that our approach significantlyoutperforms previous systems at both long andshort answer criteria.

1 Introduction

Machine reading comprehension (MRC), a taskthat aims to answer questions based on a givendocument, has been substantially advanced by re-cently released datasets and models (Rajpurkaret al., 2016; Seo et al., 2017; Xiong et al., 2017;Joshi et al., 2017; Cui et al., 2017; Devlin et al.,2019; Clark and Gardner, 2018). Natural Questions(NQ, Kwiatkowski et al., 2019), a newly releasedbenchmark, makes it more challenging by introduc-ing much longer documents than existing datasets

∗Work was done while this author was an intern at Mi-crosoft Research Asia.

†Email corresponding.

Figure 1: An example from NQ dataset.

and questions that are from real user queries. Be-sides, unlike conventional MRC tasks (e.g. Ra-jpurkar et al.,2016), in NQ, answers are providedin a two-grained format: long answer, which is typ-ically a paragraph, and short answers, which aretypically one or more entities inside the long an-swer. Figure 1 shows an example from NQ dataset.

Existing approaches on NQ have obtainedpromising results. For example, Kwiatkowski et al.(2019) builds a pipeline model using two sepa-rate models: the Decomposable Attention model(Parikh et al., 2016) to select a long answer, andthe Document Reader model (Chen et al., 2017) toextract the short answer from the selected long an-swer. Despite the effectiveness of these approaches,they treat the long and short answer extraction astwo individual sub-tasks during training and failto model this multi-grained characteristic of thisbenchmark, while we argue that the two sub-tasksof NQ should be considered simultaneously to ob-tain accurate results.

According to Kwiatkowski et al. (2019), a validlong answer must contain all of the information re-quired to answer the question. Besides, an accurateshort answer should be helpful to confirm the longanswer. For instance, when humans try to find thetwo-grained answers in the given Wikipedia pagein Figure 1, they will first try to retrieve paragraphs

6709

(long answer) describing the entity bowling hallof fame, then try to confirm if the location (shortanswer) of the asked entity exists in the paragraph,which helps to finally decide which paragraph is thelong answer. In this way, the two-grained answerscan provide evidence for each other.

To address the two sub-tasks together, insteadof using conventional documents modeling meth-ods like hierarchical RNNs (Cheng and Lapata,2016; Yang et al., 2016; Nallapati et al., 2017;Narayan et al., 2018), we propose to use graphattention networks (Velickovic et al., 2018) andBERT (Devlin et al., 2019), directly model repre-sentations at tokens, sentences, paragraphs, anddocuments, the four different levels of granularityto capture hierarchical nature of documents. Inthis way, we directly derive scores of long answersfrom its paragraph-level representations and obtainscores of short answers from the start and end posi-tions on the token-level representations. Thus thelong and short answer selection tasks can be trainedjointly to promote each other. At inference time,we use a pipeline strategy similar to Kwiatkowskiet al. (2019), where we first select long answersand then extract short answers from the selectedlong answers.

Experiments on NQ dataset show that our modelsignificantly outperforms previous models at bothlong and short answer criteria. We also analyzethe benefits of multi-granularity representationsderived from the graph module in experiments.

To summarize, the main contributions of thiswork are as follows:

• We propose a multi-grained MRC modelbased on graph attention networks and BERT.

• We apply a joint training strategy where longand short answers can be considered simulta-neously, which is beneficial for modeling thedependencies of the two-grained answers.

• We achieve state-of-the-art performance onboth long and short answer leaderboard of NQat the time of submission (Jun. 25th, 2019),and our model surpasses single human per-formance on the development dataset at bothlong and short answer criteria.

We will release our code and models at https://github.com/DancingSoul/NQ_BERT-DM.

Figure 2: System overview. The document fragmentsof one document are fed into our model independently.The outputs of graph encoders are merged and sent intothe answer selection module, which generates a longanswer and a short answer.

2 Preliminary

2.1 Natural Questions Dataset

Each example in NQ dataset contains a questiontogether with an entire Wikipedia page. The mod-els are expected to predict two types of outputs: 1)long answer, which is an HTML span containingenough information for a reader to completely inferthe answer to the question. It can be a paragraph,a table, a list item, or a whole list. A long answeris selected in a list of candidates, or a “no answer”should be given if no candidate answers the ques-tion; 2) short answer, which can be “yes”, “no” ora list of entities within the long answer. Also, a“no answer” should be given if there is no suitableshort answer.

2.2 Data Preprocessing

Since the average length of the documents in NQ istoo long to be considered as one training instance,we first split each document into a list of documentfragments with overlapping windows of tokens,like in the original BERT model for the MRC tasks(Alberti et al., 2019b; Devlin et al., 2019). Then wegenerate an instance from a document fragment byconcatenating a “[CLS]” token, tokenized question,a “[SEP]” token, tokens from the content of the doc-

6710

Token-Level Self-Attention

Sentence-LevelSelf-Attention

Paragraph-LevelSelf-Attention

Doc

Graph Initialization

BERT Encoder

Graph Integration

Add & Norm

Feed-Forward

Add & Norm

Output Layer

N×

Concatenate

Figure 3: Inner structure of our graph encoder.

ument fragment and a final “[SEP]” token. “[CLS]”and “[SEP]” follow the definitions from Devlinet al. (2019). We tag each document fragment withan answer type as one of the five labels to constructa training instance: “short” for instances that con-tain all annotated short spans, “yes” and “no” foryes/no annotations where the instances contain thelong answer span, “long” when the instances con-tain the long answer span, but there is no short oryes/no answer. In addition to the above situations,we tag a “no-answer” to those instances.

We will explain more details of the data prepro-cessing in the experiment section.

3 Approach

In this section, we will explain our model. Themain idea of our model lies in multi-granularitydocument modeling with graph attention networks.The overall architecture of our model is shown inFigure 2.

3.1 Input & Output Definition

Formally, we define an instance in the training setas a six-tuple

(c, S, l, s, e, t).

Suppose the instance is generated from the i-thdocument fragment Di of the corresponding ex-ample, then c = ([CLS], Q1, ..., Q|Q|, [SEP], Di,1

, ..., Di,|Di|, [SEP]) defines the document fragmentDi along with a question Q of the instance, |Q|+|Di|+ 3 = 512 corresponding to the data prepro-cessing method. S denotes the set of long answercandidates inside the document fragment. l ∈ S

Document Fragment

Paragraph

Sentence

Token

Figure 4: The graph on the left is an illustration of thegraph integration layer. The graph on the right showsthe incoming information when updating a paragraphnode. The solid lines represent the edges in the hierar-chical tree structure of a document while the dash linesstand for the edges we additionally add.

is the target long answer candidate among the can-didate set S of this instance. s, e ∈ {0, 1, ..., 511}are inclusive indices pointing to the start and endof the target answer span. t ∈ {0, 1, 2, 3, 4} isthe annotated answer type, corresponding to thefive labels. For instances containing multiple shortanswers, we set s and e to point to the leftmostposition of the first short answer and the rightmostposition of the last short answer, respectively.

Our goal is to learn a model that identifies a longanswer candidate l and a short answer span (s, e)in l and predicting their scores for evaluation.

3.2 Multi-granularity Document1 Modeling

The intuition of representing documents in multi-granularity is derived from the natural hierarchicalstructure of a document. Generally speaking, a doc-ument can be decomposed to a list of paragraphs,which can be further decomposed to lists of sen-tences and lists of tokens. Therefore, it is straight-forward to treat the document structure as a tree,which has four types of nodes, namely token nodes,sentence nodes, paragraph nodes, and a documentnode. Different kinds of nodes represent informa-tion at different levels of granularity. Since longanswer candidates are paragraphs, tables, or lists,information at paragraph nodes also represents theinformation for long answer candidates.

The hierarchical tree structure for a documentcontains edges that are between tokens and sen-tences, between sentences and paragraphs, and be-tween paragraphs and documents. Besides, wefurther add edges between tokens and paragraphs,between tokens and documents, between sentencesand the document to construct a graph. All these

1For brevity, the word “document” refers to documentfragment in the rest of our paper.

6711

edges above are bidirectional in our graph repre-sentation. Hence information between every twonodes can be passed through no more than twoedges in the graph. In the rest of this section, wewill present how we utilize this graph structure topass information between nodes with graph atten-tion networks so that the two-grained answers canpromote each other.

3.3 Graph EncoderFigure 3 shows the inner structure of our graphencoder. Each layer in our graph encoder consistsof three self-attention layers, a graph integrationlayer, and a feed-forward layer. The self-attentionlayers are used for interactions among nodes withthe same granularity, while the graph integrationlayer aims at gathering information from other lev-els of granularity with graph attention networks.Figure 4 is an illustration for the graph integra-tion layer. Since self-attention is a special case ofgraph attention networks, where the graph is fullyconnected, we only introduce the general form ofgraph attention networks, which can be generalizedto the self-attention mechanism.

3.3.1 Graph Attention NetworksWe apply graph attention networks (Velickovicet al., 2018) to model the information flow betweennodes, which can further improve the representa-tions of nodes by attention mechanism over fea-tures from its neighbors. In this way, the interactionbetween the two-grained answers can be enhanced.Instead of other graph-based models, we use graphattention networks to keep consistency with themulti-head attention module in the BERT model.We will describe a single layer of our graph atten-tion networks in the following.

We define a graph G = (V, E , X) that is com-posed of a set of nodes V , node features X =(h1, ...,h|V|) and a list of directed edge set E =(E1, ..., EK) where K is the number of edges. Eachi ∈ V has its own representation hi ∈ Rdh wheredh is the hidden size of our model.

We use the multi-head attention mechanism inour graph attention networks following Vaswaniet al. (2017). We describe one of the m attentionheads. All the parameters are unique to each atten-tion head and layer. If there is an edge from node jto node i, the attention coefficient eij is calculatedas follows:

eij =

(hiW

Q) (

hjWK)T

√dz

. (1)

We normalize the attention coefficients of node iby using the softmax function across all the neigh-bor nodes j ∈ Ni. Especially, there is a self-loopfor each node (i.e. i ∈ Ni) to allow it update itself.This process can be expressed as:

αij = softmaxj(eij) =exp(eij)∑

k∈Niexp(eik)

.

Then the output of this attention head zi is com-puted as a weighted sum of linear transformed inputelements:

zi =∑j∈Ni

αijhjWV. (2)

In the above equations, WQ,WK and WV ∈Rdh×dz are parameter matrices, dz is the output sizeof one attention head, we use dz ×m = dh.

Finally we get the multi-head attention resultz′i ∈ Rdh by concatenating the outputs of m indi-vidual attention heads:

z′i =m

‖k=1

zki .

3.3.2 Self-Attention LayerThe self-attention mechanism is equivalent to thefully-connected version of graph attention net-works. To make interactions among nodes with thesame granularity, we utilize three self-attention lay-ers, which are token-level self-attention, sentence-level self-attention, and paragraph-level self-attention. Since the four types of nodes are essen-tially heterogeneous, we separate the self-attentionlayer from the graph integration layer to distinguishinformation from nodes with the same granularityor different ones.

3.3.3 Graph Integration LayerWe use graph attention networks on the graph pre-sented in Figure 4, this layer allows information tobe passed to nodes with different levels of granu-larity. Instead of integrating information only onceafter the graph encoder, we put this layer rightafter every self-attention layer inside the graph en-coder, which means the update brought by the self-attention layer will also be utilized by the nodeswith other levels of granularity. This layer helpsto model the dependencies of the two-grained an-swers. We concatenate the input and output ofthe graph integration layer and pass it to the feed-forward layer.

6712

3.3.4 Feed-Forward LayerFollowing the inner structure of the transformer(Vaswani et al., 2017), we also utilize an addi-tional fully connected feed-forward network at theend of our graph encoder. It consists of two lin-ear transformations with a GELU activation in be-tween. GELU is Gaussian Error Linear Unit ac-tivation (Hendrycks and Gimpel, 2016), and weuse GELU as the non-linear activation, which isconsistent with BERT.

3.3.5 Relational EmbeddingInspired by positional encoding in Vaswani et al.(2017) and relative position representations inShaw et al. (2018), we introduce a novel relationalembedding on our constructed graph, which aimsat modeling the relative position information be-tween nodes on the multi-granularity documentstructure. We make the edges in our documentmodeling graph to embed relative positional infor-mation. We modify equation 1 and 2 for eij and zi

to introduce our relational embedding as follows:

eij =

(hiW

Q) (

hjWK)T

+ hiWQ(aKij

)T√dz

,

zi =∑j∈Ni

αij

(hjW

V + aVij

).

In above equations, the edge between node iand node j is represented by learnable embeddingaKij , aV

ij ∈ Rdz . The representation can be sharedacross attention heads. Compared to previous workwhich encodes positional information in the em-bedding layer, our proposed relational embeddingis more flexible, and the positional information canbe taken into consideration in each graph layer. Forexample, relational embedding between two nodesof the same type represents the relative distance be-tween them in the self-attention layer. In the graphintegration layer, relational embedding between asentence and its paragraph represents the relativeposition of the sentence in the paragraph, and it isthe same for other types of edges.

3.3.6 Graph InitializationSince the BERT model can only provide token-level representation, we use a bottom-up average-pooling strategy to initialize the nodes other thantoken-level nodes. We use oi ∈ {0, 1, 2, 3} to in-dicate the type of node i, representing token node,sentence node, paragraph node and document node

respectively. The initialized representation is cal-culated as follows:

h0i = average

j∈Ni,oj+1=oi

{h0j + aij

}+ boi ,

where aij , boi ∈ Rdh represent the relational em-bedding and node type embedding in the graphinitializer.

3.4 Output Layer

The objective function is defined as the negativesum of the log probabilities of the predicted dis-tributions, averaged over all the training instances.The log probabilities of predicted distributions areindexed by the true start and end indices, true longanswer candidate index, and the type of this in-stance:

L(θ) =− 1

N

N∑i

[log p(s, e, t, l | c, S)]

=− 1

N

N∑i

[log ps(s | c, S)

+ log pe(e | c, S) + log pt(t | c, S)+ log pl(l | c, S)],

where ps(s | c, S), pe(e | c, S), pl(l | c, S) andpt(t | c, S) are the probabilities for the start andend position of the short answer, probabilities forthe long answer candidate, and probabilities for theanswer type of this instance, respectively. One ofthe probability, ps(s | c, S), is computed as follow,and the others are similar to it:

ps(s | c, S) = softmax(fs(s, c, S; θ)),

where fs is a scoring function, derived from the lastlayer of graph encoder. Similarly, we derive scorefunctions at the other three levels of granularity.For instances without short answers, we set thetarget start and end indices to the “[CLS]” token.We also make “[CLS]” markup as the first sentenceand paragraph, and the paragraph-level “[CLS]”will be classified as long answers for the instanceswithout long answers. At inference time, we get thescore of a document fragment g(c, S), long answerscore g(c, S, l) and short answer score g(c, S, s, e)

6713

Long Answer Dev Long Answer Test Short Answer Dev Short Answer TestP R F1 P R F1 P R F1 P R F1

DocumentQA 47.5 44.7 46.1 48.9 43.3 45.7 38.6 33.2 35.7 40.6 31.0 35.1DecAtt + DocReader 52.7 57.0 54.8 54.3 55.7 55.0 34.3 28.9 31.4 31.9 31.1 31.5BERTjoint 61.3 68.4 64.7 64.1 68.3 66.2 59.5 47.3 52.7 63.8 44.0 52.1+ 4M synthetic data 62.3 70.0 65.9 65.2 68.4 66.8 60.7 50.4 55.1 62.1 47.7 53.9

BERT-syn+Model-III 72.4 73.0 72.7 - - - 60.1 54.1 56.9 - - -+ ensemble 3 models 74.2 73.6 73.9 73.7 75.3 74.5 64.0 54.9 59.1 62.6 55.3 58.7

Single Human 80.4 67.6 73.4 - - - 63.4 52.6 57.5 - - -Super-annotator 90.0 84.6 87.2 - - - 79.1 72.6 75.7 - - -

Table 1: Results of our best model on NQ compared to the previous systems and to the performance of a singlehuman annotator and of an ensemble of human annotators. The previous systems include DocumentQA (Clarkand Gardner, 2018), DecAtt + DocReader (Parikh et al., 2016; Chen et al., 2017) , BERTjoint and BERTjoint + 4Msynthetic data (Alberti et al., 2019a).

as follows:

g(c, S) =ft(t > 0, c, S; θ)

− ft(t = 0, c, S; θ);

g(c, S, l) =fl(l, c, S; θ)

− fl(l = [CLS], c, S; θ);

g(c, S, s, e) =fs(s, c, S; θ) + fe(e, c, s; θ)

− fs(s = [CLS], c, S; θ)

− fe(e = [CLS], c, S; θ).

We use the sum of g(c, S, l) and g(c, S) to se-lect a long answer candidate with highest score.g(c, S) is considered as a bias term for documentfragments. Then we use g(c, S, s, e) to select thefinal short answer within the selected long answerspan. We rely on the official NQ evaluation scriptto set thresholds to separate the predictions to posi-tive and negative on both long and short answer.

4 Experiments

In this section, we will first describe the data prepro-cessing details, then give the experimental resultsand analysis. We also conduct an error analysis andtwo case studies in the appendix.

4.1 Data Preprocessing DetailsWe ignore all the HTML tags as well as tokens notbelonging to any long answer candidates. The av-erage length of documents is approximately 4, 500tokens after this process. Following Devlin et al.(2019) and Alberti et al. (2019b), we first tokenizequestions and documents using a 30, 522 word-piece vocabulary. Then we slide a window of acertain length over the entire length of the docu-ment with a stride of 128 tokens, generating a listof document fragments. There are about 7 para-graphs and 18 sentences on average per document

fragment. We add special markup tokens at thebeginning of each long answer candidate accordingto the content of the candidate. The special tokenswe introduced are of the form “[Paragraph=N]”,“[Table=N]” and “[List=N]”. According to Albertiet al. (2019b), this decision was based on the obser-vation that the first few paragraphs and tables in thedocument are more likely to contain the annotatedanswer. We generate 30 instances on average perNQ example, and each instance will be processedindependently during the training phase.

Since the fact that only a small fraction of gen-erated instances are tagged as positive instanceswhich contains a complete span of long or shortanswer, and that 51% of the documents do not con-tain the answers for the questions, We downsampleabout 97% of null instances to get about 660, 000training instances in which 350, 000 has a long an-swer, and 270, 000 has short answers.

4.2 Experimental Settings

We use three model settings for our experiments,which are: 1) Model-I: A refined BERT baselineon the basis of Alberti et al. (2019b); 2) Model-II: A pipeline model with only graph initializa-tion method to get representation of sentence, para-graph, and document; 3) Model-III: Adding twolayers of our graph encoder on the basis of Model-II.

Model-I improves the baseline in Alberti et al.(2019b) in two ways: 1) When training an instancewith a long answer only, we ignore the loss ofpredicting the short answer span to “no-answer”because it would introduce distraction to the model.2) We sample more negative instances.

We use three BERT encoders to initialize ourtoken node representation: 1) BERT-base: a

6714

Model LA. F1 SA. F1

BERT-base+Model-I 63.9 51.0BERT-base+Model-II 67.7 50.9BERT-base+Model-III 68.9 51.9

BERTjoint 64.7 52.7BERT-large+Model-I 66.0 52.9BERT-large+Model-II 70.3 53.2BERT-large+Model-III 70.7 53.8

BERT-syn+Model-I 67.8 56.1BERT-syn+Model-II 72.2 56.7BERT-syn+Model-III 72.7 56.9

Table 2: Comparison of different models with differentBERT models on the development dataset.

BERT-base-uncased model finetuned on SQuAD2.0; 2) BERT-large: a BERT-large-uncasedmodel finetuned on SQuAD 2.0; 3) BERT-syn:Google’s BERT-large-uncased model pre-trainedon SQuAD2.0 with N-Gram Masking and Syn-thetic Self-Training.2 Since the Natural Questiondataset does not provide sentence-level informa-tion, we additionally use spacy (Honnibal and Mon-tani, 2017) as the sentence segmentor to get theboundaries of sentences.

We trained the model by minimizing loss L fromSection 3.4 using the Adam optimizer (Kingma andBa, 2015) with a batch size of 32. We trained ourmodel for 2 epochs with an initial learning rateof 2 × 10−5, and we use a warmup proportion of0.1. The training of our proposed model is con-ducted on 4 Tesla P40 GPUs for approximately 2days. For each setting, the results are averagedover three models initialized with different randomseeds to get a more solid comparison, which alsosuggests the improvements brought by our methodsare relatively stable. The hidden size, the numberof attention heads, and the dropout rate in our graphencoder are equal to the values in the correspondingBERT model.

4.3 Comparison

The main results are shown in Table 1. The re-sults show that our best model BERT-syn+Model-III(ensemble 3 models) have gained improvementover the previous models by a large margin. Ourensemble strategy is to train three models with dif-ferent random seeds. The scores of answer candi-

2This model can be downloaded at https://bit.ly/2w7nUQK.

Model LA.F1 SA.F1

0-layer 67.7 50.91-layer 68.8 51.22-layer 68.9 51.93-layer 68.9 51.94-layer 68.9 51.7

Table 3: Influences of graph layer numbers on the de-velopment set.

dates are averaged over these three models. At thetime of submission (Jun. 25th, 2019), this modelhas achieved the state-of-the-art performance onboth long answer (F1 score of 74.5%) and shortanswer (F1 score of 58.7%) on the public leader-board3. Furthermore, our model surpasses singlehuman performance at both long and short answercriteria on the development dataset.

The comparison of different models with differ-ent BERT models is illustrated in Table 2. Theresults show that our approach significantly outper-forms our baseline model on both the long answerand the short answer. For the BERT-base setting,our Model-II with a pipeline inference strategy out-performs our baseline by 3.8% on long answer F1score while our Model-II with two graph layers fur-ther improves the performance by 1.2% and 1.0%.For the BERT-syn setting, the Model-III benefitsless from the graph layers because the pretrainingfor this model is already quite strong. Our Model-III with BERT-large, compared to previously pub-lic model (BERTjoint) also using BERT-large, im-proves long answer F1 score by 6.0% and shortanswer F1 score by 1.1% on the development set.

From Table 1 and Table 2, we can see that the en-semble of human annotators can lead to a massiveimprovement at both long and short answer criteria(from 73.4% to 87.2%, 57.5% to 75.7%). However,the improvement of ensembling our BERT-basedmodel is relatively smaller (from 72.7% to 73.9%,56.9% to 59.1%). This suggests that the diversityof human annotators is a lot better than the samemodel structure with different random seeds. Howto improve the diversity of the deep learning mod-els for the open-domain datasets like NQ remainsas a hard question.

3Since we can only make 10 submissions on the testdataset, we only submit and report the result of our best model.Due to the official attempts on the test dataset are given 24hours. We can only ensemble 3 models at most.

6715

Model LA. F1 SA. F1

BERT-base+Model-III 68.9 51.9

-Graph module 63.9 51.0-Long answer prediction 65.1 51.4-Short answer prediction 68.2 --Relational embedding 68.8 51.7-Graph integration layer 68.3 51.1-Self-attention layer 68.4 51.2

Table 4: Ablation study on the development set.

4.4 Ablation Study

We evaluate the influence of layer numbers, whichis illustrated in Table 3. We can see the increase inthe performance of our models when the numberof layers increases from 0 to 2 (The 0-layer settingmeans that only the graph initialization module isused to obtain the graph representations). Thenthe model performance does not improve with thenumber of network layers increasing. We attributeit to the fact that the information between every twonodes in our proposed graph can be passed throughin no more than two edges, and that increasing thesize of randomly initialized parameters may not bebeneficial for BERT fine-tuning.

To evaluate the effectiveness of our proposedmodel, we conduct an ablation study on the de-velopment dataset on the BERT-base setting. Theresults are shown in Table 4. First, we discussthe effect of the joint training strategy. We cansee that the removal of either sub-task goals willbring decreases on both tasks. It suggests that thetwo-grained answers can promote each other withour multi-granularity representation. Then we re-move the whole graph module, which means theinference process depends on the score of shortanswer spans because long answer candidates can-not be scored. We can see the decrease of bothlong and short answer performance by 5.0% and0.9%, respectively, indicating the effectiveness ofour proposed graph representations.

Finally, we investigate the effect of componentsin our graph encoder. In Table 4, we can see thatwithout relational embedding, the performance onthe long answer and short answer both slightly de-crease. When removing the graph integration layer,the performance of long answer and short answerboth become worse by 0.6% and 0.8%. At last,we remove the self-attention layer in the graph en-coder, the performance of long answer and short

answer both become worse by 0.5% and 0.7%. Theablation study shows the importance of each com-ponent in our method.

5 Related Work

Machine reading comprehension has been widelyinvestigated since the release of large-scale datasets(Rajpurkar et al., 2016; Joshi et al., 2017; Lai et al.,2017; Trischler et al., 2017; Yang et al., 2018).Lots of work has begun to build end-to-end deeplearning models and has achieved good results (Seoet al., 2017; Xiong et al., 2017; Cui et al., 2017;Devlin et al., 2019; Lv et al., 2020). They normallytreat questions and documents as two simple se-quences regardless of their structures and focus onincorporating questions into the documents, wherethe attention mechanism is most widely used. Clarkand Gardner (2018) proposes a model for multi-paragraph reading comprehension using TF-IDFas the paragraph selection method. Wang et al.(2018) focuses on modeling a passage at word andsentence level through hierarchical attention.

Previous work on document modeling is mainlybased on a two-level hierarchy (Ruder et al., 2016;Tang et al., 2015; Yang et al., 2016; Cheng andLapata, 2016; Koshorek et al., 2018; Zhang et al.,2019). The first level encodes words or sentences toget the low-level representations. Moreover, a high-level encoder is applied to obtain document repre-sentation from the low-level. In these frameworks,information flows only from low-level to high-level.Fernandes et al. (2018) proposed a graph neural net-work model for summarization and this frameworkallows much complex information flows betweennodes, which represents words, sentences, and en-tities in the graph.

Graph neural networks have shown their flexibil-ity in a variant of NLP tasks (Zhang et al., 2018c;Marcheggiani et al., 2018; Zhang et al., 2018b;Song et al., 2018). A recent approach that be-gan with Graph Attention Networks (Velickovicet al., 2018), which applies attention mechanismsto graphs. Wang et al. (2019) proposed knowledgegraph attention networks to model the informa-tion in the knowledge graph, (Zhang et al., 2018a)proposed gated attention networks, which use aconvolutional sub-network to control each atten-tion head’s importance. We model the hierarchicalnature of documents by representing them at fourdifferent levels of granularity. Besides, the rela-tions between nodes are represented by different

6716

types of edges in the graph.

6 Conclusion

In this work, we present a novel multi-grainedMRC framework based on graph attention net-works and BERT. We model documents at differentlevels of granularity to learn the hierarchical na-ture of the document. On the Natural Questionsdataset, which contains two sub-tasks predictinga paragraph-level long answer and a token-levelshort answer, our method jointly trains the twosub-tasks to consider the dependencies of the two-grained answers. The experiments show that ourproposed methods are effective and outperform thepreviously existing methods by a large margin. Im-proving our graph structure of representing the doc-ument as well as the document-level pretrainingtasks is our future research goals. Besides, the cur-rently existing methods actually cannot process along document without truncating or slicing it intofragments. How to model long documents is still aproblem that needs to be solved.

Acknowledgments

This work was supported by the National NaturalScience Foundation of China (NSFC) via grant61976072, 61632011 and 61772153.

ReferencesChris Alberti, Daniel Andor, Emily Pitler, Jacob De-

vlin, and Michael Collins. 2019a. Synthetic QA cor-pora generation with roundtrip consistency. In Pro-ceedings of the 57th Conference of the Associationfor Computational Linguistics, ACL 2019, Florence,Italy, July 28- August 2, 2019, Volume 1: Long Pa-pers.

Chris Alberti, Kenton Lee, and Michael Collins. 2019b.A BERT baseline for the Natural Questions. arXivpreprint arXiv:1901.08634.

Danqi Chen, Adam Fisch, Jason Weston, and AntoineBordes. 2017. Reading wikipedia to answer open-domain questions. In Proc. of ACL.

Jianpeng Cheng and Mirella Lapata. 2016. Neural sum-marization by extracting sentences and words. InProc. of ACL.

Christopher Clark and Matt Gardner. 2018. Simpleand effective multi-paragraph reading comprehen-sion. In Proc. of ACL.

Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang,Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehen-sion. In Proc. of ACL.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: pre-training ofdeep bidirectional transformers for language under-standing. In Proc. of NAACL.

Patrick Fernandes, Miltiadis Allamanis, and MarcBrockschmidt. 2018. Structured neural summariza-tion. arXiv preprint arXiv:1811.01824.

Dan Hendrycks and Kevin Gimpel. 2016. Bridgingnonlinearities and stochastic regularizers with gaus-sian error linear units. CoRR, abs/1606.08415.

Matthew Honnibal and Ines Montani. 2017. spacy 2:Natural language understanding with bloom embed-dings, convolutional neural networks and incremen-tal parsing. To appear.

Mandar Joshi, Eunsol Choi, Daniel Weld, and LukeZettlemoyer. 2017. TriviaQA: A large scale dis-tantly supervised challenge dataset for reading com-prehension. In Proc. of ACL.

Diederik P. Kingma and Jimmy Ba. 2015. Adam:A method for stochastic optimization. In Proc. ofICLR.

Omri Koshorek, Adir Cohen, Noam Mor, Michael Rot-man, and Jonathan Berant. 2018. Text segmentationas a supervised learning task. In Proc. of NAACL.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhine-hart, Michael Collins, Ankur Parikh, Chris Alberti,Danielle Epstein, Illia Polosukhin, Matthew Kelcey,Jacob Devlin, et al. 2019. Natural Questions: abenchmark for question answering research.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang,and Eduard Hovy. 2017. RACE: Large-scale ReAd-ing Comprehension dataset from Examinations. InProc. of EMNLP.

Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang,Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang,Guihong Cao, and Songlin Hu. 2020. Graph-basedreasoning over heterogeneous external knowledgefor commonsense question answering. In Proceed-ings of the Thirty-Fourth Conference on Associa-tion for the Advancement of Artificial Intelligence(AAAI).

Diego Marcheggiani, Joost Bastings, and Ivan Titov.2018. Exploiting semantics in neural machine trans-lation with graph convolutional networks. In Proc.of NAACL.

Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.Summarunner: A recurrent neural network based se-quence model for extractive summarization of docu-ments. In Proc. of AAAI.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata.2018. Ranking sentences for extractive summa-rization with reinforcement learning. In Proc. ofNAACL.

6717

Ankur P. Parikh, Oscar Tackstrom, Dipanjan Das, andJakob Uszkoreit. 2016. A decomposable attentionmodel for natural language inference. In Proc. ofEMNLP.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. SQuAD: 100,000+ questionsfor machine comprehension of text. In Proc. ofEMNLP.

Sebastian Ruder, Parsa Ghaffari, and John G Breslin.2016. A hierarchical model of reviews for aspect-based sentiment analysis. In Proc. of EMNLP.

Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, andHannaneh Hajishirzi. 2017. Bidirectional attentionflow for machine comprehension. In Proc. of ICLR.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.2018. Self-attention with relative position represen-tations. In Proc. of NAACL.

Linfeng Song, Yue Zhang, Zhiguo Wang, and DanielGildea. 2018. N-ary relation extraction using graph-state LSTM. In Proc. of EMNLP.

Duyu Tang, Bing Qin, and Ting Liu. 2015. Documentmodeling with gated recurrent neural network forsentiment classification. In Proc. of EMNLP.

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har-ris, Alessandro Sordoni, Philip Bachman, and Ka-heer Suleman. 2017. Newsqa: A machine compre-hension dataset. In Proceedings of the 2nd Work-shop on Representation Learning for NLP.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Proc. of NIPS.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova,Adriana Romero, Pietro Lio, and Yoshua Bengio.2018. Graph attention networks. In Proc. of ICLR.

Wei Wang, Ming Yan, and Chen Wu. 2018. Multi-granularity hierarchical attention fusion networksfor reading comprehension and question answering.In Proc. of ACL.

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, andTat-Seng Chua. 2019. Kgat: Knowledge graph atten-tion network for recommendation. In Proceedingsof the 25th ACM SIGKDD International Conferenceon Knowledge Discovery & Data Mining (KDD).

Caiming Xiong, Victor Zhong, and Richard Socher.2017. Dynamic coattention networks for questionanswering. In Proc. of ICLR.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio,William Cohen, Ruslan Salakhutdinov, and Christo-pher D Manning. 2018. Hotpotqa: A dataset for di-verse, explainable multi-hop question answering. InProc. of EMNLP.

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He,Alex Smola, and Eduard Hovy. 2016. Hierarchicalattention networks for document classification. InProc. of NAACL.

Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jian-jun Xu, Ji Wang, Ming Gong, and Ming Zhou.2019. Pretraining-based natural language gen-eration for text summarization. arXiv preprintarXiv:1902.09243.

Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, IrwinKing, and Dit-Yan Yeung. 2018a. Gaan: Gated at-tention networks for learning on large and spatiotem-poral graphs. In Proc. of UAI.

Yue Zhang, Qi Liu, and Linfeng Song. 2018b.Sentence-state LSTM for text representation. InProc. of ACL.

Yuhao Zhang, Peng Qi, and Christopher D. Manning.2018c. Graph convolution over pruned dependencytrees improves relation extraction. In Proc. ofEMNLP.

Appendix

A Error Analysis

We provide an error analysis for our proposed mod-els. We divide the results for instances in develop-ment dataset into five cases:

• Case 1: The question has a long (short) an-swer, and the predicted score is above thethreshold.

• Case 2: The question does not have a long(short) answer, and the predicted score is be-low the threshold.

• Case 3: The question has a long (short) an-swer, and prediction is wrong.

• Case 4: The question has a long (short) an-swer, and the predicted score is below thethreshold.

• Case 5: The question does not have a long(short) answer, and the predicted score isabove the threshold.

The analysis results are shown in Table 5. ForBERT-base+Model-III, we can see it outperformsother BERT-base models in the first four cases onthe long answer and gets comparable results onCase 5. For the short answer, the improvementof our proposed model mainly comes from Case1 and Case 4, which suggests that our approach

6718

Long Answer Short AnswerCase1 Case2 Case3 Case4 Case5 Case1 Case2 Case3 Case4 Case5

BERT-base+Model-I 38.2 28.4 9.7 10.9 12.8 20.2 48.5 7.7 16.2 7.3BERT-base+Model-II 40.8 28.6 8.4 9.7 12.5 20.0 49.0 7.7 16.4 6.9BERT-base+Model-III 41.8 28.6 8.1 9.0 12.6 20.9 48.2 8.0 15.3 7.7

BERT-syn+Model-I 40.0 30.0 7.9 11.0 11.1 22.6 49.3 7.4 14.1 6.6BERT-syn+Model-II 42.8 30.7 6.6 9.5 10.4 23.3 48.9 7.6 13.2 7.0BERT-syn+Model-III 43.0 30.9 6.2 9.7 10.2 23.9 48.2 8.1 12.2 7.7

Table 5: Percentage of five categories for both long answer and short answer.

Question: what ’s the dog ’s name on tom and jerry

Long Answer: Tom ( named “ Jasper ” in his debut appear-ance ) is a grey and white domestic shorthair cat . “ Tom ” isa generic name for a male cat . He is usually but not always, portrayed as living a comfortable , or even pampered life ,while Jerry ...

Long Answer: Spike , occasionally referred to as Butch orKiller , is a stern but occasionally dumb American bulldogwho is particularly disapproving of cats , but a softie whenit comes to mice ( though in his debut appearance , DogTrouble , Spike goes after both Tom and Jerry ) ...

Short Answer: Jasper Short Answer: Spike , occasionally referred to as Butch orKiller

Question: when is a spearman correlation meant to be used instead of a pearson correlation

Long Answer: This method should also not be used incases where the data set is truncated ; that is , when theSpearman correlation coefficient is desired for the top Xrecords ( whether by pre-change rank or post-change rank, or both ) , the user should use the Pearson correlationcoefficient formula given above .

Long Answer: The Spearman correlation between two vari-ables is equal to the Pearson correlation between the rankvalues of those two variables ; while Pearson ’s correla-tion assesses linear relationships , Spearman ’s correlationassesses monotonic relationships ( whether linear or not ) ...

Short Answer: where the data set is truncated Short Answer: assesses monotonic relationships ( whetherlinear or not )

Table 6: Case studies from the development dataset. The results of directly predicting short answer span are shownon the left, and the results on the right are predicted by a pipeline strategy.

helps the model do well in cases that have a shortanswer. Comparing Model-I and Model-III, we cansee the significant improvement of our model liesin the long answer on Case 1 (From 38.2%, 40.0%to 41.8%, 43.0%, respectively).

For Case 2 and Case 5, our Model-III does nothave significant improvement compared to Model-I. The reason is that, for instances with no answeror no apparent answers, fine-grained informationis more crucial. Therefore, using the score of shortanswer spans might be more accurate than the longanswer score from paragraph nodes, which arecoarse-grained. Overall, our Model-III is betterthan the baseline Model-I, especially for exampleswith long or short answers.

B Case Study

We report two case studies on the developmentdataset shown in Table 6. In the first case, theformer prediction finds a wrong short answer“Jasper” where the word-level information in ques-tion “name” and “tom” is captured within a min-imal context. Our pipeline strategy can consider

the context of the whole paragraph, leading to amore accurate long answer along with its short an-swer. For the second case, the former predictionfailed to capture the turning information while ourpipeline model sees the whole context in the para-graph, which leads to the correct short answer. Inboth two cases, short answers on the left both havea larger score than those on the right. This suggeststhat for a model that learns a strong paragraph-levelrepresentation, we can prevent errors from short an-swers by constraining it to the selected long answerspans.