Vinculación con el medio

Measuring community engagement: The limitation of metrics

Thomas Farnell  14 May 2022

Discussions about the societal impact of universities and their engagement with various societal actors are now commonplace around the world, notably in the United Kingdom, Australia and the United States, but also Latin America.

In most of the European Union, however, the emergence of these debates has been more gradual and fragmented. EU policies supporting universities’ broader societal role were framed for a long time in purely economic terms, focusing on innovation, technology transfer and cooperation with industry.

This is now changing: recent EU documents refer for the first time to civic universities and to ‘service to society’ as a ‘fourth mission’ of higher education.

Assuming that this newly emerging priority will remain on the policy agenda in Europe, policy-makers will soon be asking the inevitable follow-up question: “How do we measure it?”

To tackle this question, a consortium of universities, university networks, experts and local councils from eight EU countries have been working since 2018 on the EU-funded project Towards a European Framework for Community Engagement in Higher Education (TEFCE).

Defining ‘community’ engagement

Throughout the project, we have framed the discussion about universities’ interactions with society through the term ‘community engagement’. There are dozens of other terms used to describe such interactions: ‘public’, ‘civic’ and ‘regional’ engagement; ‘knowledge exchange’; or the ‘civic university’.

In the TEFCE project, we chose to define the term in a purposefully broad way: community engagement is about how universities address societal needs in partnership with their communities – a term that can include public authorities, businesses, schools, civil society and citizens.

According to this definition, community engagement can take place through virtually all university activities and the TEFCE project proposed seven thematic dimensions of community engagement: teaching and learning, research, service and knowledge exchange, students, management (partnerships and openness), management (policies and support structures) and supportive peers.

In practice, community engagement includes activities such as community-based learning, participatory research, projects with external organisations, policy advocacy, student volunteering and providing access to university facilities.

Measuring without metrics

When trying to assess how an institution is performing, many rush towards developing metrics and rankings. The latest Times Higher Education Impact Rankings does precisely this: it ranks universities according to how well they contribute to the United Nations’ Sustainable Development Goals.

Another recent initiative by a consortium of universities from the UK, the US and Australia also proposed a ranking methodology for university engagement.

We analysed and discussed at length the merits and shortcomings of different forms of performance assessment in higher education and how these could be applied to community engagement.

Our conclusions were that developing meaningful metrics for community engagement would probably be impossible, and even if it were possible, would not be valuable.

We reasoned the following:

• Community engagement is notoriously difficult to measure: community engagement is context specific – which communities are engaged with and what kind of engagement takes place depends on a range of factors including geography, socio-economic development, institutional missions and academic disciplines. None of the previous attempts to develop community engagement metrics have successfully addressed this challenge.

• Assessment of community engagement should be an institutional learning journey rather than a narrow performance assessment: the ultimate purpose of assessing community engagement should be to allow universities to identify community engagement activities, help them understand how they perform (by demonstrating the value and mutual benefits generated by such activities) and assist them in eventual improvement. It is difficult to see how a metrics-based approach could lead to such outcomes.

An alternative approach: the TEFCE Toolbox

The TEFCE project therefore adopted an alternative approach and developed an institutional self-reflection framework for community engagement in higher education: the ‘TEFCE Toolbox’.

The toolbox begins with mapping the range of community engagement activities that are carried out across the university. It then provides a framework allowing universities to critically reflect on their community engagement. The result is a ‘heatmap’ indicating areas in which the university performs best, and areas which are in most need of further development.

The heatmap is structured according to the seven dimensions of community engagement and to five criteria: authenticity of engagement, range of societal needs addressed, diversity of communities engaged with, extent of institutional spread of community engagement and institutional sustainability of community engagement.

There are many other international tools to assess community engagement in higher education, such the US’s Carnegie Classification for Community Engagement in Higher Education and the UK’s EDGE Tool.

The TEFCE Toolbox builds upon these to do something slightly different. Firstly, it is centred around a critical approach to determining authenticity of engagement, ie the extent to which engagement is mutually beneficial and the extent to which communities are able to co-determine and co-create activities.

Secondly, it is participatory: the findings are validated through open discussions within the university and with external partners.

Thirdly, it is the first tool that aims to be applied by universities across different countries, irrespective of socio-economic, cultural or institutional contexts.

Leading the debate on community engagement

As part of the TEFCE project, four universities with diverse institutional profiles and from four different countries (Croatia, Germany, Ireland and the Netherlands) successfully piloted the Toolbox and confirmed its value.

Users particularly valued the fact that the Toolbox raises the visibility of the value of community engagement, supports the intrinsic motivation of community-engaged staff, students and external partners and facilitates a learning journey rather than tools that focus on compliance or competition.

A follow-up EU project, Steering Higher Education for Community Engagement (SHEFCE), will see four more universities from Austria, Belgium and Spain applying the TEFCE Toolbox.

The new project consortium includes a range of international stakeholders who will discuss how to further develop a European framework for community engagement in higher education, including the European University Association, the European Students’ Union and the OECD, as well as national partners such as the National Co-ordinating Centre for Public Engagement (NCCPE UK) and Campus Engage (Ireland).

Recent developments have shown that community engagement is likely to remain on the policy agenda in Europe in the next decade.

In the latest ministerial communiqué of the European Higher Education Area, ministers stressed that higher education institutions “must engage with their communities to undertake mutually beneficial and socially responsible joint activities”, and one of the 10 European Higher Education Area principles and guidelines for the social dimension of higher education is that “higher education institutions should ensure that community engagement in higher education promotes diversity, equity and inclusion”.

Through the European Universities Initiative, the European Commission also aims to further optimise the vision of European Universities “to address big societal challenges, become true engines of development for cities and regions and promote civic engagement”.

The aforementioned frameworks could therefore provide a structure for supporting community engagement in higher education in Europe and beyond. Ultimately, our hope in developing these frameworks is that, when the question is asked, universities will be ready to provide a robust, qualitative answer to how they engage with society rather than resorting to metrics.

Thomas Farnell is a higher education policy expert at the Institute for the Development of Education, a think tank for higher education based in Zagreb, Croatia. He will be giving a presentation on a new vision for community engagement at the UNESCO World Higher Education Conference which takes place in Barcelona, Spain from 18-20 May 2022. This article is based on a text originally published on the LSE Impact Blog.