This is the third in a series of posts about the data that are likely to appear in the Obama administration’s proposed college-ratings system. The first post looked at graduation rates. The second post examined net price.

The most contentious data that might find their way into the Obama administration’s college-ratings plan are conspicuously missing from the Education Department’s current data-collection tools: postgraduate wages.

At a symposium held by the department several weeks ago, experts were told that they weren’t debating the “if” of the proposed Postsecondary Institution Ratings System, just the “how.” Nevertheless, many still questioned whether it was appropriate to use postgraduate wage data to compare colleges in the first place.

“We do not believe that income alone is an adequate outcome measure of bachelor’s-degree recipients,” said Christine M. Keller, associate vice president for academic affairs at the Association of Public and Land-Grant Universities. “We think that a combination of employment and enrollment in advanced degree programs is a better metric.”

Ms. Keller was speaking on behalf of a consortium of six higher-education associations that have proposed an alternative to the Obama administration’s ratings system. They aren’t the only ones with reservations about using postgraduate wage data to compare institutions and, ultimately, as part of determining eligibility for federal funding, as President Obama has recommended the ratings would do by 2018.

“We’re on the record saying that wage outcomes should not be used to measure the effectiveness of an institution or program,” Tod R. Massa, director of policy research and data warehousing for the State Council of Higher Education for Virginia, told The Chronicle.

Institutions should instead be rated on whether they teach strong core values, instill analytical- and critical-thinking skills, and provide their graduates with proficiency in literacy and numeracy, he said.

That’s not to say postgraduate wage data do not have value. If prospective students understand what their job prospects are, Mr. Massa said, they might make better decisions about whether to take on student-loan debt and how much debt is more than they’ll be able to pay off.

“If students are thinking about graduate or professional school, they need to understand debt or earnings, to have an idea of what the next four, eight, or 12 years of their life might look like,” he said.

Others caution about reading too much into wage data. Brandon Busteed, executive director of Gallup Education, a division of the polling company, said decades of survey research have shown that salary is a relatively poor measure of a person’s well-being.

“We have overinvested in the idea that how much money one makes is the ultimate outcome variable,” he said. “It’s actually less important, if we were to rank-order it, than how engaged they are in their work or to what degree they have a high quality of life.”

It may be that other metrics could be more useful in helping students pick the college that will meet their needs and goals. In collaboration with Purdue University and with significant funding from the Lumina Foundation, Gallup is embarking on an initiative to track quality-of-life measures for 150,000 college graduates over the next five years. Mr. Busteed said he hopes the Gallup-Purdue Index, as it’s called, will eventually be used by colleges—alongside salary, if not in place of it—to measure their students’ outcomes.

Not if, but How

Despite those concerns, if the White House’s College Scorecard is any indication of the Obama administration’s idea of the best data points for comparing colleges, then it’s likely that wages will play some kind of role in the college-ratings system. But the Scorecard, a kind of predecessor to the ratings system, also hints at the challenges that lie ahead before reliable, comparable earnings data for college graduates are available.

Those challenges are reflected by the blank briefcase icon in the employment section of the College Scorecard, alongside a note that reads, in part, “The U.S. Department of Education is working to find information about the average earnings of former undergraduate students.”

Why is it so hard to find data on graduates’ earnings? Some say it’s because Congress won’t allow the Education Department to collect student-level data.

But no such ban exists at the state level, and Virginia has collected student-level data since 1992, said Mr. Massa, allowing it to publish detailed reports about the wages of graduates from different majors and programs at each college and university in the state. The data go back a couple of decades. The key to replicating this on a federal level, Mr. Massa said, is making both the wage and educational records anonymous.

The devil, of course, is in the details, and it appears unlikely that the ban on federal collection of such unit-record data is going to be lifted in time for the rollout of the college-ratings system.

Perhaps the most likely alternative so far comes from a PayScale, a relative newcomer to the higher-education scene.

PayScale is a company that collects crowdsourced employment data, which it then sells to employers as a means to set competitive salaries. The company gathers salary data through monthly web surveys of about 200,000 people. The surveys often gather information on where respondents went to college and what they majored in, said Lydia Frank, director of editorial and marketing at PayScale.

For the past several years, PayScale has used the data to produce rankings that show which colleges and majors provide the best return on investment. (Disclosure: The Chronicle bought institution-level graduate salary data from PayScale for its College Reality Check microsite.) The company has been in talks with the Education Department about its college salary data. Although neither party has committed to using the data in the college ratings, PayScale wants to be part of the conversation on college outcomes, Ms. Frank said.

“We feel that the PayScale data around career outcomes for college graduates is the most complete set of data that exist without passing some type of legislation,” she said, referring to the unit-record ban. “So yes, if there’s some opportunity for consumers to better understand about how to make a smarter educational decision on the way into college, then we would like to be part of that, absolutely.”

Other Factors That Affect Salary

PayScale’s overall salary report “favors schools with strong engineering programs,” according to its website, because of the higher wages earned by engineering professionals. That’s why Harvey Mudd College, a Southern California institution focused on science, technology, engineering, and mathematics, tops the list, while nearby Pomona College, an equally selective liberal-arts institution, doesn’t even break the top 50. That’s also why the company produces reports on the colleges with the best salaries for graduates with specific majors.

Most experts agree that if wage data are to be used in the college-ratings system, it should compare salaries by major or program, not by institution, because engineering and business degrees are likely to lead to higher-salary jobs than degrees in education or social work.

This may sound straightforward, but it can get challenging pretty quickly. PayScale’s crowdsourced data make it hard to obtain a large-enough sample of students from smaller majors at smaller colleges—art-history graduates at Pomona, for example—to allow for a meaningful comparison with graduates of similar programs at other institutions.

Even with unit-record data, though, the picture of postgraduate wages can be blurry. Most states that report graduates’ salaries use unemployment-insurance data to determine the average salaries of people in certain industries. But that approach conflates occupation and industry, and may not give an accurate picture of postgraduate wages, said Mark S. Schneider, vice president of the American Institutes for Research, at last month’s Education Department symposium.

Wage statistics derived from unemployment-insurance data use industry codes, so “you know what industry a person is in, but you don’t know what their occupation is within it,” Mr. Schneider said. “This is a serious problem. The difference between occupation and industry is fundamental.” Such data would miss the difference between an accountant and a software engineer who both work at a technology company, for example, and would fail to reflect the differing wages among people with the same occupation in different industries.

That is just one of the difficulties in controlling postgraduate salary data for the many factors that affect a person’s wages, aside from just his or her alma mater.

Another problem: location. Mr. Schneider and several other experts said the data must control for location in order to take into account the wide variation in cost of living across the country, or even within a single state.

The Education Department will also use salary data to its own detriment if it does not control for how much a person was making before he or she entered college, said Braden J. Hosch, vice president for institutional research, planning, and effectiveness at the State University of New York at Stony Brook.

When he was director of policy and research for the Connecticut Board of Regents for Higher Education, the now-defunct Connecticut Department of Higher Education used to publish data on the wages of college graduates across the state. The data would invariably show that associate-degree recipients earned more within several months of graduating than bachelor’s-degree recipients did, he said.

The reason? Community-college graduates tend to be older than bachelor’s recipients, and they often have already spent time in the work force, so they enter college already earning a higher salary, Mr. Hosch said. But those reports obscured the fact that, even if their initial gross salaries were higher, the overall long-term value added for associate-degree holders was lower than for bachelor’s-degree recipients.

“We really need to encourage more institutions, states, and even the federal government to do more research on earnings outcomes of graduates,” Mr. Hosch said. “But because we don’t understand it well now, it’s not recommended for inclusion in a system that rates the effectiveness of institutions.”

While excluding salary data from the college-ratings system may not be an option, there’s still time for the Education Department to heed Mr. Hosch’s warnings, and those of other experts, about the pitfalls of postgraduate earnings data and how to make sure what’s presented will actually be useful to students and institutions alike.

Return to Top