Ok guys, I’m going to take the rest of the week to nerd out about performance-based funding (PBF) indicators, since clearly this is all anyone here in Ontario is going to be talking about for the next few months. I’m going to start with the issue of what indicators are going to be used—and fair warning: this is going to be long.

(Reminder to readers: I actually do this stuff for a living. If you think your institution needs help understanding the implications of the new PBF, please do get in touch).

Last week, the provincial government briefed institutions on the indicators it would like to use in its new performance-based funding formula, which in theory is meant to eventually govern 60% of all provincial grants to institutions. It won’t actually be that much, for a variety of reasons, but let’s stick to the government script for the moment. Currently, the government still has literally no idea about how it is going to turn data on these ten indicators into actual dollar amounts for institutions -I’ll talk more about how that might work later this week. But we can still make some useful observations about the indicators on their own in three respects. First, with respect to the conceptual validity of the proposed measure, second, with respect to any potential measurement problems, and third with respect to any perverse incentives the indicator may create.

Are you sitting comfortably? Then I’ll begin.

According to some briefing papers handed out by the Ontario Ministry of Training, Colleges and Universities last week (which someone very kindly provided me, you know who you are, many thanks) the new PBF is supposed to be based on ten indicators: six related to “skills and job outlooks” and four related to “economic and community impact”. One of the latter sets of indicators is meant to be designed and measured individually by each university and college (in consultation with the ministry), which is a continuation of practices adopted in previous strategic mandate agreements. One of the “economic” indicators is a dual indicator – a research metric for universities and an “apprenticeship-related” metric for colleges which is “under development” (i.e. the government has no clue what to do about this), so I won’t touch the college one for now. And finally, one of the skills indicators is some kind of direct measurement of skills, presumably using something like the tests HEQCO performed as part of its Postsecondary and Workplace Skills project (which I wrote about back here) –I will deal with separately with that one tomorrow because it’s a huge topic all on its own.

So that leaves us with eight indicators, which are:

Graduation Rate. Assuming they stick with current practice, this will be defined as “percentage of first-time, full-time undergraduate university students who commenced their study in a given Fall term and graduated from the same institution within 6 years.”. Obvious problems include what to do about transfer students (aren’t we supposed to be promoting pathways?)

Graduate Employment. The government is suggesting measuring not employment rates but the “proportion of graduates employed full-time in fields closely or partly related to one’s field of study”. This question is currently asked on both the university and college versions of Ontario Graduate Surveys but is not currently published at an institutional level. It is a self-report question, which means the government does not have to define what a “related” or “partly related” means.

Graduate Earnings. This is currently tracked by the ministry through the Ontario Graduate Surveys, but the government appears to want to switch to using Statistics Canada’s new Educational and Labour Market Longitudinal Platform (ELMLP), through which graduate incomes can be tracked through the tax system as a means of measurement. This is mostly to the good, in the sense that response rates will be high and valid and graduates can be tracked for longer (though it is not clear what the preferred time frame is here), but they will lose the ability to exclude graduates who are enrolled in school for a second degree.

Experiential Learning. The Government’s briefing document indicates it wants to use the “number and proportion” of graduates in programs with experiential learning (this is confusing because it is actually two indicators) for colleges, but substitutes the word “courses” for “programs” when it comes to universities. I have no idea what this means and suspect they may not either. Possibly, this is a complicated way of saying they want to know what proportion of graduates have had a work-integrated learning experience.

Institutional Strength/Focus. This is a weird one. The government says it wants to measure “the proportion of students in an area of strength” at each institution. I can’t see how any institution looking at this metric is going to name anything other than their largest faculty (usually Arts) as their area of strength. Or how OCAD isn’t just going to say “art/design” and get a 100% rating. Maybe there’s some subtlety here that I’m missing but this just seems pointless to me.

Research Funding and Capacity (universities only): Straight up, this is just how much tri-council research funding each institution receives, meaning it could be seen as a form of indirect provincial support to cover overhead on federal research. This seems clear enough, but presumably there will be quite some jostling about definitions, in particular: how is everyone supposed to count money for projects that have investigators at multiple institutions? Should it use the same method as the federal indirect research support program, or some other method? Over how many years will the calculation be made? A multiple-year rolling average seems best, since in any given year the number can be quite volatile at smaller institutions.

“Innovation”. Simply, they mean funding from industry sources (for universities, this is specified as “research income”). The government claims it can get this data from the Statscan/CAUBO Financial Information of Universities and Colleges Survey, although I’m 99% sure that’s not something that gets tracked specifically. Also, important question: do non-profits count as “industry”? Because particularly in the medical field, that’s a heck of a big chunk of the research pie.

“Community/Local Impact”. OK, hold on to your hats. Someone clearly told the government they should have a community impact indicator to make this look like “not just a business thing”, but of course community impact is complex, diffuse, and difficult to measure consistently. So, in their desperation to find a “community” metric which was easy to measure, they settled on…are you ready?…institution size…divided by….community size. No, you’re not misreading that and yes, it’s asinine. I mean, first of all it’s not performance. Secondly, it’s not clear how you measure community; for instance, Confederation College has campuses in five communities, Waterloo has three, etc., so what do you use as a denominator? Third: What? WHAT? Are you KIDDING ME? Set up a battle of wits between this idea and a bag of hammers and the blunt instruments win every time. This idea needs to die in a fire before this process goes any further because it completely undermines the idea of performance indicators. If the province needs a way to quietly funnel money to small town schools (helloooo, Nipissing!) then do it through the rest of the grant, not the performance envelope.

OK, so that is eight indicators. Two of these (community impact, institutional strength) are irretrievably stupid and should be jettisoned at the first opportunity. The “research” and “innovation” measures are reasonable provided sensible definitions are used (multi-year averages, the indirect funding method of counting tri-council income, inclusion of non-profits) and would be non-controversial in most European countries. The experiential learning one is probably OK, but again much depends on the actual definitions chosen.

That leaves the three graduation/employment metrics. There are some technical issues with all of them. The graduation rate definition is one-dimensional, and in most US states a simple grad rate is now usually accompanied by other metrics of progress beyond completion (e.g. indicators for successfully bringing transfer students to degree, or indicators for getting students to complete 30/60/90 credits). The graduate employment “in a related field” is going to make people scream (it might be a useful metric for professional programs, but in most cases degrees aren’t designed to link to occupations and even where they are, people shift occupations after a few years anyway) and in any case it is to be measured through a survey with notoriously low completion rates, which will matter at small institutions. The graduate income measure is technically OK but doesn’t work well as an indicator in some types of PBF systems because it does not scale with institution size (I’ll deal more with this in Thursday’s blog).

But the bigger issue with all three of these is that they conceivably set up some very bad incentives for institutions. In all three of them, institutions could juice their scores by dumping humanities or fine arts programs and admit only white dudes, because that’s who does best in the labour market. I’m not saying they would do this – institutions do have ethical compasses – but it is quite clearly a dynamic that could be in play at the margin. As it stands, there is a strong argument here that these measures have the potential to be anti-diversity and anti-access.

There is, I think, a way to counter this argument. Let’s say the folks in TCU do the right thing and consign those two ridiculous indicators to the dustbin: why not replace them with indicators which encourage broadening participation? For instance, awarding points to institutions which are particularly good at enrolling students with disabilities, Indigenous students, low-income students, etc. The first two are measured already through the current SMA process; the third could be measured through student aid files, if necessary. That way, any institution which tries to win points by being more restrictive in its intake would lose points on another (hopefully equally weighted) indicator, and the institutions which do best would be those that are both open access and have great graduation/employment outcomes. Which, frankly, is as it should be.

Tomorrow: Measuring skills.

Yesterday,  I critiqued most of the indicators being suggested for the new Ontario PBF system. But I left one out because I thought it was worth a blog all on its own, and that is the indicator related to “skills and competencies”. It’s the indicator that is likely to draw the most heat from the higher education traditionalists, and so it is worth drilling into.

In principle, measuring the ability of institutions to provide students more of the skills that allow them to succeed in the work force is perfectly sound. Getting a better job is not the only reason students attend university or college, but repeated surveys over many decades show it is the most important one. Similarly, while labour force improvement is not the only reason governments fund universities, it is the most important one (it pretty much is the only reason they fund colleges). So, it is in no way wrong to try to find out whether institutions are doing a good job at this. And if you don’t like simple employment or salary measures (both of which are fairly crude), then it makes sense to look at something like skills, and the contribution each institution makes towards improving them.

Problem is, there isn’t exactly a consensus on how to do that.

In some countries – Brazil and Jordan, for instance – students take national subject-matter examinations. Nobody seems to think this is a good idea in OECD countries, partly this is because there tends to be a lot of curriculum diversity, making the setting of such national tests hilariously fraught. But mostly, it’s because in many fields, there is no particularly straight line between disciplinary knowledge and the kinds of skills and competencies required in the labour market. One is not necessarily superior to the other, but they are just different things. And so was born the idea to try to measure these skills directly and more importantly, to measure their change over time.

The dominant international approach is a variation of the Collegiate Learning Assessment (CLA), pioneered about 20 years ago by the Council for Aid to Education (a division of RAND Corporation). The approach is, essentially, give the same exam to a (hopefully randomly-selected) group of students during their first and final year of studies at an institution. Assuming there has been no major change in the qualities of the entering student body (which can be done in the US by looking at average SAT scores, but is harder to do in Canada), this creates a “synthetic cohort” and obviates having to offer a test to incoming students and then waiting three years to test them again, which would be tedious. You mark the scores on both tests, and you see the change in scores over time. Not all of the growth in skills/competencies can be attributed to the institution, of course (22 year-olds, on the whole, are smarter than 18 year-olds, even if they don’t go to school), so the key is then to compare the change in results at one institution against those in the others. Institutions where the measured “gain” is very high are hence deemed “good” at imparting skills and those where the gain is small are deemed “poor”. By measuring learning gain rather than just exiting scores, institution can’t just win by skimming the best students; they actually have to do some imparting of skills.

This is all reasonable, methodologically speaking. You can quibble about whether synthetic cohorts are good substitutes for real ones, but most experts think they are. And many years of testing suggest the results are reliable. The question is, are they valid? And this is where things get tricky.

What CLA does is designed to test students’ ability to think critically, reason analytically, solve problems, and communicate clearly and cogently. It does this by asking a set of open-ended questions about a document (for some sample questions, see here) and then scoring answers on a scoring rubric (see here). This isn’t an outlandish approach, but there are questions to be asked about how well the concepts of “thinking critically” and “reasoning analytically” are measured. As this quite excellent review of CLA methodology suggests, it seems to have straddled three separate and not-entirely compatible definitions of “critical thinking”; but while it seems to capture the nature of what is taught in humanities and social sciences, it is not clear that this is the same thing as “critical thinking skills” in other fields.

(A similar problem exists with tests that claim to measure “problem-solving”.  Turns out that results of exams of problem-solving actually are highly correlated with measures of IQ. This is not the end of the world if you use the CLA synthetic cohort approach because you would be comparatively measuring an increase in IQ rather than IQ itself, which might not be such a terrible thing to the extent one could prove that the competencies IQ measures are valued in the work place. But it’s not quite as advertised.)

Now, we don’t know exactly how Ontario plans to measure skills. The Higher Education Quality Council of Ontario (HECQO) – which is presumably the agency which will be put in charge of these initiatives – has been experimenting with a couple of different tests of skills: the HEIghten Critical Thinking assessment developed by the American company ETS (some sample questionshere), and the Education and Skills Online (ESO) test, which is a variant of the test the OECD uses in its Programme for the International Assessment of Adult Competencies (methodology guide here). The former is a one-off test of graduating students, and (though I am not a psychometrician) just seems like a really superficial version of CLA or OCED’s AHELO; the ESO test uses the synthetic cohort strategy but uses a test which is much heavier on simple literacy and numeracy skills and lighter on critical thinking (technically, it tries to measure “problem-solving in a digital environment” which may come close to IQ testing, but again not necessarily a problem if you are looking at “gain”). That probably means EASI has higher levels of validity (because literacy and numeracy are pretty well-studied and firmly conceptualized), though again there has to be some doubt about the degree to which these specific skills represent what is valued/rewarded in the workplace.

The smart money is that Ontario is planning on using some variant of EASI (final report here, my take thereon here) in order to measure institutional outcomes. Which means the focus will be on literacy and numeracy, which are not ridiculous things to want post-secondary graduates to have and hence not ridiculous outcomes for which to hold institutions accountable. The problem is really twofold.

First, HEQCO (or whoever is eventually put in charge of this) is going to have to pay a lot of attention to sampling. In the first initial test done in 2017, they basically got whoever they could to take the test. Now that actual money is on the table, the randomization of the sample is incredibly important. And it’s going to be hard to enforce genuine randomization unless some students all agree to take a test, which, you know, might be tricky since the government can’t actually force anyone to do so. Second, it’s not clear how well the synthetic cohort strategy will work without a standardized measure like SAT scores to back it up: currently, there’s no good way to tell how alike the entering and exiting cohorts actually are. I don’t think it’s beyond HEQCO’s wit to devise something (obvious possibility: use individual students’ entering grades, normalized by whatever means Ontario institutions use to equalize scores from high schools with different marking schemes), but they need something better than what they have now.

There will be some who say it is a grave perversion of the purpose of university to measure graduates on anything other than their knowledge of subject matter. Hornswoggle. If universities and college aren’t focussing on enhancing literacy and numeracy skills as part of their development of subject expertise, something is desperately wrong. Some will claim it will force institutions to change their curriculum. No, it won’t: but it will probably get them to change their methods of assessment to focus more on these transversal skills than on pure subject matter. And that’s a good thing.

What it boils down to is the following: students and governments both pay universities and colleges on the assumption that an education there improves employability. Employability is at least plausibly related to advanced literacy and numeracy skills. We can measure literacy and numeracy, and subject to some technical improvements noted above, we can probably measure the extent to which institutions play a role in enhancing those skills. This is – subject to caveats about not getting too excited about measures of “critical thinking” and paying attention to some important methodological issues – generally a good idea. In fact, compared to some of the other performance measures the government of Ontario is considering, I would say it is among the better ones.

But do pay attention to those caveats. They matter.