Group items tagged

“There is a huge value in learning with instant feedback,” Dr. Agarwal said. “Students are telling us they learn much better with instant feedback.”

Les Perelman, has drawn national attention several times for putting together nonsense essays that have fooled software grading programs into giving high marks. He has also been highly critical of studies that purport to show that the software compares well to human graders. “My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman, a retired director of writing and a current researcher at M.I.T.

Two start-ups, Coursera and Udacity, recently founded by Stanford faculty members to create “massive open online courses,” or MOOCs, are also committed to automated assessment systems because of the value of instant feedback. “It allows students to get immediate feedback on their work, so that learning turns into a game, with students naturally gravitating toward resubmitting the work until they get it right,” said Daphne Koller, a computer scientist and a founder of Coursera.

Mark D. Shermis, a professor at the University of Akron in Ohio, supervised the Hewlett Foundation’s contest on automated essay scoring and wrote a paper about the experiment. In his view, the technology — though imperfect — has a place in educational settings. With increasingly large classes, it is impossible for most teachers to give students meaningful feedback on writing assignments, he said. Plus, he noted, critics of the technology have tended to come from the nation’s best universities, where the level of pedagogy is much better than at most schools. “Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could,” Dr. Shermis said. “There seems to be a lack of appreciation of what is actually going on in the real world.”

A copy of that internal presentation, which Junn repeatedly emphasized was preliminary, was obtained this week by Inside Higher Ed from the California Faculty Association. According to the preliminary presentation, 74 percent or more of the students in traditional classes passed, while no more than 51 percent of Udacity students passed any of the three courses.

The courses were also put together in a rush. That’s apparently because of the timing of the deal with Udacity. The pilot project was announced a fortnight before classes started. (Like other similar deals, it was also the result of a no-bid contract.) The deal came together at the highest levels: On June 16, 2012, Brown e-mailed and called Thrun to talk about how Udacity could help California's higher education systems. “We need your help,” Brown said, according to Thrun. But, because of the haste, faculty were building the courses on the fly. Not only was this a “recipe for insanity,” Junn said, but faculty did not have a lot of time to watch how students were doing in the courses because the faculty were busy trying to finish them. It took about 400 hours to build a course, though the courses are designed to be reused.

Another factor in the disappointing outcomes may have been the students themselves. The courses included at-risk students, high school students and San Jose State students who had already failed a remedial math course.

Student performance data from the San Jose State/Udacity courses are expected to be released in coming weeks.