Computers Are Learning to Read-But They're Still Not So Smart | WIRED - 0 views
www.wired.com/...-but-theyre-still-not-so-smart
research artificial intelligence pattern recognition
shared by johnsonel7 on 20 Oct 19
- No Cached
-
computers still weren’t very good at understanding the written word. Sure, they had become decent at simulating that understanding in certain narrow domains, like automatic translation or sentiment analysis (for example, determining if a sentence sounds “mean or nice,” he said). But Bowman wanted measurable evidence of the genuine article: bona fide, human-style reading comprehension in English. So he came up with a test
-
The machines bombed. Even state-of-the-art neural networks scored no higher than 69 out of 100 across all nine tasks: a D-plus, in letter grade terms. Bowman and his coauthors weren’t surprised. Neural networks — layers of computational connections built in a crude approximation of how neurons communicate within mammalian brains
-
It produced a GLUE score of 80.5. On this brand-new benchmark designed to measure machines’ real understanding of natural language — or to expose their lack thereof — the machines had jumped from a D-plus to a B-minus in just six months.
- ...4 more annotations...
-
The only problem is that perfect rulebooks don’t exist, because natural language is far too complex and haphazard to be reduced to a rigid set of specifications.
-
Researchers simply fed their neural networks massive amounts of written text copied from freely available sources like Wikipedia — billions of words, preformatted into grammatically correct sentences — and let the networks derive next-word predictions on their own. In essence, it was like asking the person inside a Chinese room to write all his own rules, using only the incoming Chinese messages for reference.“The great thing about this approach is it turns out that the model learns a ton of stuff about syntax,”
-
The nonsequential nature of the transformer represented sentences in a more expressive form, which Uszkoreit calls treelike. Each layer of the neural network makes multiple, parallel connections between certain words while ignoring others — akin to a student diagramming a sentence in elementary school. These connections are often drawn between words that may not actually sit next to each other in the sentence. “Those structures effectively look like a number of trees that are overlaid,” Uszkoreit explained.
-
But instead of concluding that BERT could apparently imbue neural networks with near-Aristotelian reasoning skills, they suspected a simpler explanation: that BERT was picking up on superficial patterns in the way the warrants were phrased.