Structure-sensitive Dependency Learning in Recurrent Neural Networks

Recent technological advances have made it possible to train recurrent neural networks (RNNs) on a much larger scale than before. These systems, which have proved very useful in natural language engineering applications, are sequence models that lack explicit formal linguistic representations. Their success in learning language from a corpus could have profound cognitive implications: is the computational problem of language acquisition easier than we thought? Do neural networks have the potential to serve as models of human linguistic competence? Before answering these questions in the affirmative, we need to better understand the linguistic capabilities of the networks: success in a practical task may be a sufficient standard for engineering purposes, but careful analysis using tools from psycholinguistics and theoretical linguistics is necessary if we are to draw any cognitive conclusions.

This project evaluates the ability of RNNs to learn structure-sensitive syntactic dependencies, taking English subject-verb number agreement as a case study. We focus on specific sentence types that are indicative of the network's syntactic abilities; our tests use both naturally occurring sentences and constructed sentences from the experimental psycholinguistics literature. We analyze the internal representations of the network to explore the sources of its ability (or inability) to approximate sentence structure. Finally, we compare the errors made by the RNNs to agreement attraction errors made by humans.

RNNs were able to approximate certain aspects of syntactic structure very well, but only in common sentence types and only when trained specifically to predict the number of a verb (as opposed to a standard language modeling objective). In complex sentences their performance degraded substantially; they made many more errors than human participants. These results suggest that stronger inductive biases are likely to be necessary to eliminate errors altogether. More broadly, our work illustrates how methods from linguistics and psycholinguistics can help us understand the abilities and limitations of "black-box" artificial intelligence systems.

Please find more information here:

https://linguistics.stonybrook.edu/events/colloquium/2017/09/29/tal.linzen