Description:

Recent technological advances have made it possible to train recurrent neural networks (RNNs) on a much larger scale than before. These networks have proved effective in applications such as machine translation or speech recognition. These engineering advances are surprising from a cognitive point of view: RNNs do not have the kind of explicit structural representations that are typically thought to be necessarily for syntactic processing. In this talk, I will discuss two studies that go beyond standard engineering benchmarks and examine the syntactic capabilities of contemporary RNNs using established cognitive and linguistic diagnostics. The first study shows that RNNs are able to compute subject-verb agreement relations with considerable success, although their error rate increases in complex sentences. The second study compares the detailed pattern of agreement errors made by the RNNs to those made by humans in a behavioral experiment, and reveals some similarities (attraction errors, number asymmetry) but also some differences (relative clause modifiers increase the probability of attraction errors in RNNs but decrease it in humans). Overall, RNNs can develop fairly sophisticated syntactic representations despite the lack of an explicit hierarchical bias, but differ from humans in their behavior in important ways.