Assessing the Value of an Undergrad Degree
A new study from Academically Adrift‘s co-author seeks to evaluate a broad range of student experiences and outcomes as well as new ways to measure learning.
A broad and ambitious attempt to understand what students get out of higher education, both experientially and intellectually, took an important step this week at the University of California, Irvine.
Richard Arum coauthored the high-profile book Academically Adrift, which evaluated the critical thinking skills students gain in college, with unflattering results for higher education. Arum is now dean of the university’s education school, where he is leading a project to survey students’ experiences in college as well as learning outcomes that have in the past eluded measurement. Some of the study’s 1,000 randomly selected students began the survey process Wednesday.
The project is meant to incorporate a complicated array of student data, ranging from answers to short experience surveys — where students report what they are doing at the moment, whom they are with and how engaged they are — to behavior on learning management systems, grades and measures of civic reasoning.
“We’ll have the ability to identify student experiences inside and outside the classroom, as well as undergraduate outcomes, in a way that’s more rich, complex and deep than anything that’s occurred previously in the field,” Arum said.
The goals of the project, he said, are to identify tools other institutions can use to do these measurements, identify where students are flourishing and struggling, and use the collected data to improve instruction.
“Nationally, 40 percent of students who start college don’t finish it. And there are not systematic efforts underway to address and systematically put in place support so that all students are learning and developing trajectories so that they can attain their goals in life,” Arum said. “You need to have the scientific basis, the high-quality measurement systems in place, in order for those decisions to be informed, evidence based and data driven.”
The project was designed to be a first stage and to inform a later effort to develop a similar large-scale study for multiple colleges and universities. At UC Irvine, the study’s sample of 1,000 randomly selected students includes 500 incoming, first-year students, 250 junior transfer students and 250 continuing juniors.
Natasha Jankowski, director of the National Institute for Learning Outcomes Assessment, voiced support for the project, but she has concerns around equity in its future application to other institutions. Because the project is focused on highly selective universities and advised by staff at those institutions, she said, it should not be used to create norms for students at other colleges.
“My concern is that something that’s made for a particular type of institution, for a particular type of traditional student,” she said, “being applied outside of that into a larger student-success conversation at community colleges, minority-serving institutions and institutions with predominantly adult student populations — that fundamentally will not fit.”
For example, holding adult learners, with likely very different experiences from the study sample, to a learning or behavior standard not designed for them could be detrimental to their success and make them feel like they do not belong in higher education, she said.
Arum said the project ideally will track students after they graduate and evaluate their success in the labor market as well as their civic engagement.
New tools from Educational Testing Service will be employed to evaluate difficult-to-measure outcomes, such as a student’s ability to understand different perspectives, collaborate and problem solve — skills Arum says are not only essential for labor-market success, but are relevant to creating a democratic citizenry.
“It’s not easy. These are complicated things to measure well,” he said. “We’re not content with taking off-the-shelf measures.”
Outcomes and Soft Skills
George Kuh, founding director of the National Institute for Learning Outcomes Assessment, said the most exciting and likely difficult part of the project would be measuring these soft skills.
“I’ll be curious to see what they develop to tap these — sometimes people call them more esoteric outcomes,” he said. “If you look at what makes for a successful person postcollege, some of these lesser-talked-about proficiencies are very, very important, especially in the long term and as we think about what’s going to matter after college, what kinds of outcomes are going to portend people’s success in jobs that don’t exist today.”
Academically Adrift, Arum’s 2011 book, analyzed students’ scores on a tool called the Collegiate Learning Assessment, which measured critical thinking skills. When the study found that 45 percent of students made little to no progress in their scores after two years of college, commentators took to both criticizing the research and declaring an existential crisis in higher ed.
“One of the criticisms of [Academically Adrift] was the measurement being not particularly deep and sophisticated,” Arum said. “I’ve been devoted professionally to responding to that criticism, to improving the measurement in higher education.”
Arum said he would have liked to have assessed students on discipline-specific learning goals, for subjects like history and biology, but didn’t yet have the resources.
While he said the new research, which is backed by the Andrew W. Mellon Foundation, is critical for “demonstrating the value of undergraduate education,” he also emphasized that the project is not about confirming pre-existing beliefs.
John Gagliardi, assistant vice president for strategy, policy and analytics at the City University of New York, Lehman College, and an editor of the book The Analytics Revolution in Higher Education, said that because earnings and employment are easier student outcomes to measure, they can often be the focus of attempts to pin down the value of a degree.
“It’s tough to thread the needle between ensuring that folks have those tools to help them have a long life, and a great career and happiness, while at the same token making sure that they’ve got the skills that they need to get a job immediately,” he said. “I think that this project at its heart offers promise to institutions to be able to do just that.”
Doris Zahner, chief scientist at the Council for Aid to Education, which makes the Collegiate Learning Assessment, said the project is an important step in the general process of improving institutions and student outcomes. She pointed to a gap between the skills students are graduating with and the skills employers want. The goal, generally and for the council, is to measure that gap and remedy it.
“It’s not about, at the end of the day, saying things are good or bad. It’s about, how do you inform the institutions to be able to improve how they are supporting students?” Arum said. “We’ve asked K-12 institutions to do this for decades. We have to bring the same rigor, the same approach to our own institutions to fulfill their promise … to restore the public’s faith in these institutions.”
0 Comments