June 27, 2014 bySteve Kolowich

In December 2013 a group of academics gathered during a Texas snowstorm and began the second phase of a discussion about massive open online courses. They were not terribly impressed by the hype the courses had received in the popular media, and they had set out to create a better body of literature about MOOCs—albeit a less sensational one.

The MOOC Research Initiative, backed by the Bill & Melinda Gates Foundation, had given many of those academics research grants to study what was going on in the online courses. Now the organization has posted preliminary findings from some of those research projects.

The findings have not yet been peer-reviewed and should not be generalized, but they do represent some of the most rigorous analysis to date on MOOCs. Following is a synopsis of the more interesting findings. For wonkier interpretations of the data, you can find the researchers’ own summaries here.

1. If you are isolated, poor, and enamored of the prestigious university offering the MOOC you’re taking, you are less likely to complete it.

Researchers at Columbia University’s Teachers College asked students to rate their reasons for registering for “Big Data in Education,” a MOOC the university offered through Coursera. Students who said they were “geographically isolated from educational institutions,” “cannot afford to pursue a formal education,” and were motivated because the “course is offered by a prestigious university” were less likely than others to finish the course.

2. Coaching students to have a healthier mindset about learning may not help in a MOOC.

Carol S. Dweck, a professor of psychology at Stanford University, famously coined the terms “growth mindset” and “fixed mindset” to describe opposing ways people think about intelligence. People with a fixed mindset believe their thinking skills are set in concrete; those with a growth mindset think those skills can grow and improve in the right circumstances.

Staging “mindset interventions” can help students become better learners, writes Daniel Greene, a doctoral student in Stanford’s Graduate School of Education.

But when Mr. Greene studied the effects of such interventions on students in one of Stanford’s MOOCs, “Introduction to Logic,” he found that they did not significantly affect how students behaved or how well they performed in the course. His research noted an important caveat, however: The students Mr. Greene studied had unusually strong growth mindsets to begin with. He speculated that students who sign up for MOOCs, and stick around long enough to be studied, might have less to learn about learning.

3. Paired with the right incentives, MOOCs can help prepare at-risk students for college-level work.

After a high-profile pilot program involving Udacity and San Jose State University failed to produce impressive results, it became conventional wisdom that MOOCs have nothing to offer underprivileged high-school students. But researchers at the University of California at Irvine found that MOOCs might help underprepared students become college-ready. All it takes is the right incentive.

The university offered a MOOC, “Preparation for Introductory Biology,” and identified a subset of students who wanted to study biology at Irvine but did not score high enough on the mathematics section of the SAT to declare it as a major. The university told the students that if they completed the MOOC with distinction, and then did well on other prerequisite courses, then they would be allowed to transfer into the biology major sooner than otherwise.

The incentive worked. “Low-preparation students with an incentive from the university to do well performed better than high-preparation students in the MOOC,” wrote the Irvine researchers. “The findings suggest that MOOCs have the potential to help prepare students, especially at-risk learners entering college, for challenging STEM courses.”

4. Discussion forums in MOOCs are healthy places for the few students who use them.

Anyone familiar with the bluster of most Internet comment boards might assume that a MOOC discussion forum is no place for open minds. But researchers at Duke University found that the discussion forums in two introductory-level courses, one in English and the other in chemistry, “generally contribute[d] positively to the learning environment.”

The pattern was truer in the English course than in the chemistry course. In the English one, the researchers determined that 62 percent of posts contributed to “learning gains,” compared with only 38 percent in the chemistry course. However, in both cases, even posts that did not result in learning gains tended to be well meaning and emotionally neutral. The researchers found that a staggering 90 percent of the posts on the forum contained constructive feedback, as opposed to 2 percent that contained nonconstructive feedback. They noted that forum posts tended to be more helpful even than direct feedback on peer-reviewed assignments, which was often flattering but not necessarily constructive.

The catch is that MOOC discussion forums are usually dominated by a small group of participants. In this case, the discussants were particularly elite. “We specifically looked for challenges faced by students, such as the following: lack of time or energy; less academically prepared; and less or not self-directed,” wrote the researchers, but “these barriers did not come up in the forum threads that we coded in either class.”

5. We still do not know if doing well in MOOCs will help underprivileged learners become upwardly mobile.

One of the biggest questions around free online courses has been whether they provide any of the benefits of higher education to students who cannot afford a college degree. Researchers at the University of Michigan at Ann Arbor’s School of Information set out to answer that question. They analyzed data from six MOOCs offered by Michigan through Coursera, focusing on a group of learners who stated in a pre-course survey that they “cannot afford to pursue a formal education.”

The researchers were surprised to find that, although the students in the target group completed the course at lower rates than did those who had signed up for reasons not related to affordability, those who did complete often did so with distinction.

But it is still unclear whether those underprivileged, high-achieving MOOC learners will be able to improve their station in life as a result. “Our findings suggest that our participants perceived MOOCs as being a way to increase their economic mobility,” write the Michigan researchers, who are still working to analyze the interviews they conducted with some learners. “However, there was very limited evidence for employment as an outcome to taking or completing MOOCs.”