Effectiveness

Attending school or university does not always translate into as much academic learning as one would hope for. The OECD Programme for International Student Assessment (PISA) has shown that attending school may actually lead to very different levels of learning outcomes across countries. While there is no similar evidence at the higher education level as yet, this is arguably the same at that level too. One of the key promises of smart technologies is to enhance the effectiveness of teaching and learning for better student learning.

In the classroom, applications that directly support student learning show early promise. Personalised learning aims to provide all students with the appropriate curriculum or task, and scaffold them within a task, based on a diagnosis of their knowledge and knowledge gaps. This is not only done at the academic level, focusing on the “what”, but increasingly takes into account how students learn and factors such as self-regulation, motivation or effort (Molenaar, 2021[8]). Engagement is key for learning, and solutions to keep students engaged within digital or physical learning environments are being developed to identify their affective states during learning and nudge them towards re-engagement when they seem to disengage (D’Mello, 2021[9]). Social robots perform similar tasks in different ways: they can use adaptive learning to tutor students with natural language, but they can also teach, or motivate them to learn by playing the role of a peer student. They support teachers by enabling the implementation of different types of teaching and learning strategies (Belpaeme and Fumihide, 2021[10]). Finally, smart technologies give students with impairments and special needs access to curriculum materials and allow those students to participate in learning activities to an extent that was not possible before, here again increasing the effectiveness of education (Good, 2021[11]).

Those solutions can be used and remain helpful outside of the classroom too, either for homework, as automated private tutoring or practice solutions, and for lifelong learning. In fact, the largest market for educational technology companies is the consumer market targeting students and parents directly, either for recreational learning activities or for tutoring or test preparation.

A second promise of learning effectiveness comes from classroom analytics that support teachers in providing more effective teaching. This is still a work in progress but many applications already show how a variety of solutions could support teachers in better using their time in class, for example, by suggesting when it is a good time to shift to the next teaching or learning activity, who would require their attention the most, how they could engage the whole class in collaborative learning activities. While classroom orchestration solutions can help teachers in real time, they also provide feedback on their own practice, for example, how much they talk, to whom, or how they divide their time between different types of activities (Dillenbourg, 2021[12]). Both real-time and post-hoc feedback are akin to personal professional learning opportunities for the teacher in question, and have the significant advantage of being about the specific teacher who was (digitally) observed rather than about theoretical or general teaching practice. In that sense, smart technology has real potential to improve the teaching practice of all individual teachers, and subsequently the learning outcomes of their students.

At the organisational and system levels, smart technologies also hold promise in making education more effective. While this remains relatively rare (Ifenthaler, 2021[13]), smart technologies can be integrated in most dimensions of school activities, providing administrators, teachers and learners with feedback to manage school resources as well as improve the effectiveness of teaching and learning (see Box 1.1). The rise of a new generation of assessments powered by AI also opens new avenues for recognising and evaluating competences that were hard to assess through paper and pencil tests. This could accompany most education systems in their shift towards emphasising skills (in addition to the traditional emphasis on knowledge). Game-based assessments and simulations allow assessments to be designed to be more realistic but also to assess skills such as complex problem solving, creativity or collaboration in new ways (Buckley et al., 2021[14]).

Finally, the emergence of longitudinal education data systems that follow students through the course of their studies also allows for more effective policy and organisational interventions and a better design of educational offerings. For example, in the United States, the analysis of community college graduation rates, the success of their student placement strategies in “remedial courses” and students’ study patterns as part-time or full-time enrolment led to revisiting what the educational experience of community college students actually is and “redesigning” community colleges (Bailey, Smith-Jaggars and Jenkins, 2015[15]). As in other sectors (OECD, 2019[16]), the use of data supports policy design and interventions.

Equity

Smart technologies can help education systems provide more equitable learning opportunities. In this respect, smart technologies are more ambivalent. One the one hand, they clearly do or could help reduce inequity both by increasing access to learning opportunities for all and improving learning effectiveness for those who need it the most. On the other hand, without a widespread and equitable availability of smart technologies, inequity could also rise. They may also leave achievement gaps unchanged or even widened, depending on their differential impact on different learners.

Let us start with the difficulties. There are at least two reasons why technology may have a negative effect on equity. The first, obvious reason lies in the difference in access to devices and connectivity by students from different groups, notably students from lower socio-economic backgrounds. These students may not have the devices, the connectivity or the resources that allow accessing and using smart technologies either at the school they attend or at home. A second reason is that, if technology (e.g. personalised learning) works the same for everyone, those who start with stronger prior knowledge can maintain their advantage or even make faster progress than those with less prior knowledge. In spite of supporting students with less prior knowledge, it is thus possible that technology could help more advanced students more. This would widen rather than reduce the achievement gap.

There are also many reasons to believe that smart technologies can advance the equity agenda.

First, learning technology can expand access to learning opportunities. Educational platforms proposing open educational resources (Orr, Rimini and van Damme, 2015[17]) or massive open online course (MOOC) platforms are good examples. They allow learners to access learning materials with a quality that may be superior to what they can access locally. While many studies have shown that this increased access has not decreased inequity at scale given the low take-up and the fact that most users are already well educated, a recent systematic review of their effect on equity provides a more optimistic perspective, notably for non-English MOOCs or open educational resources (Lambert, 2020[18]).

As importantly, smart technologies can reduce inequity by facilitating the inclusion of students with special needs and by adapting learning to different learning styles. Technology has, for example, made it much easier to support the diagnosis of learning difficulties such as dysgraphia, and remedial digital responses have also been developed. A variety of smart technologies applied to learning solutions also make it easier for blind or visually impaired students as well as deaf or hard-of-hearing students to access learning materials and easily perform the educational tasks required from other students. Artificial intelligence enabling speech to text (and vice versa) or automatic subtitles are the most obvious examples. Learning technologies also tackle more difficult issues and support the socio-emotional (and thus the subsequent academic) learning of autistic children. They increasingly propose ways to help children with attention deficit hyperactivity disorder (ADHD) to self-regulate and better benefit from their schooling. One caveat here is that inclusion is not just about the individual “fitting in” but also for society to be more inclusive and open to differences (Good, 2021[11]). Technology encourages that by enabling students with special needs to study in a traditional (and inclusive) learning environment, which also changes peoples’ view on disability and special needs.

Second, solutions such as early warning systems are entirely focused on reducing inequity by helping students at risk of dropping out from high school (or university) to graduate – students who drop out typically come from disadvantaged and minority backgrounds. Early warning systems also allow designing appropriate interventions by identifying the factors or indicators most likely to predict dropout (Bowers, 2021[19]). Some use of learning analytics within institutions, for example, to monitor student engagement or redesign study programmes, could also have the same effects, should the educational institution pay particular attention to inequity (Ifenthaler, 2021[13]).

Third, the use of learning analytics as exemplified by personalisation at the individual level, be it using intelligent tutoring systems or learning analytics to keep students engaged in learning, all hold promise in reducing inequity, notably by supporting students with less prior knowledge to learn at the right pace. Box 1.3 gives an example of an online solution that reduced the learning gap between the strongest and weakest students in mathematics at the beginning of the intervention. There is, however, little evidence that adaptive learning generally reduces achievement gaps between students. Classroom analytics can also give feedback to teachers on how they could improve their teaching; specifically how and when to pay more attention to different group of students in their class, based on their academic level, gender, ethnicity, etc. Adaptive learning technology can also help students practice and make progress at home, outside of the classroom, supported by intelligent tutoring systems. This may be particularly important for students coming from households where parents can support their students less effectively with their school work, be it directly or indirectly.

Efficiency

In most other sectors than education, smart technologies are used as a tool to enhance the cost-efficiency of operations, notably by automating a number of tasks and processes, making services faster and often cheaper (OECD, 2019[16]). While education might be behind most other sectors in this respect, digitalisation is also making many educational processes more efficient as interactions between stakeholders and educational institutions become increasingly automated. As noted above, also in teaching and learning, some degree of automation is gaining ground. To what extent will digitalisation allow for enhanced cost-efficiency and productivity in education?

Any discussion of cost-efficiency should keep in mind that technology incurs investment and maintenance costs, which have to be compared with the costs of current arrangements. Digital technology has not always delivered on its cost-efficiency promises in the past because one has sometimes forgotten that, beyond the initial investment, it needs to be continuously upgraded, maintained, etc.

Nevertheless, as in other sectors, there are good reasons to believe that smart technologies could increase cost-efficiency in education.

One example lies in the application (and admission) process to educational institutions. They are increasingly undertaken through digital platforms, especially in higher education, where a “matching” (or selection) process is often necessary. In cases of open admission institutions, when no selection is required, implementing seamless automated processes is even easier. The implementation of the National Education Information System (NEIS) in Korea, an e-government system allowing, inter alie, for the digital transfer of students’ academic records from one school to the other (as well as from school to university) was estimated to save USD 237 million a year in 2010 (KERIS, 2010[22]).

A second area where digitalisation could possibly lead to cost-efficiency is the provision of verifiable degrees and other credentials on blockchain. The gradual development of an infrastructure for digital credentials and the adoption of open standards will gradually lead to a different way of certifying and holding degrees, with individuals being able to manage their qualifications themselves. This is one of the strongest and most immediate arguments of cost-efficiency in the different digital solutions examined in this book.

A third area where cost-efficiency is underway is the collection of system-level statistical information. While statistical information often relied on the establishment of statistical panels (of representative samples of individuals or institutions) and often involved multiple handlings of the same data, the use of administrative data combined with the interoperability of diverse systems has made it much easier to get statistical information from operational services in almost real time (González-Sancho and Vincent-Lancrin, 2016[23]; n.d.[24]).

At the end of the day, a cost-benefit analysis comparing the benefits of smart technology, including the non-financial ones mentioned above, to that of an existing solution, will determine how cost-efficient it is for a given service (or educational goal).

Smart technologies as socio-technical systems

Whether they are already available or under development, most of the smart education technologies for teaching and learning covered in this book do not aim to replace teachers or human beings. They were in fact developed with the current education model in mind. One common thread of all analyses in the book is that most teaching and learning solutions are designed as hybrid human-AI systems and require teacher-student interactions and human oversight of the machine at different points. Molenaar (2021[8]) offers a model to better understand the continuum between fully automated and teacher-only education. Most advanced personalisation solutions require teacher intervention or alert teachers when they should intervene, for example, because students are still struggling or they need to move on to another step of their learning process. Most solutions to support classroom orchestration are also hybrid solutions that merely scaffold teachers in implementing rich learning scenarios for their students. As Dillenbourg (2021[12]) puts it, “there is a teacher in the loop” and classroom analytics are designed to support teachers in orchestrating the teaching and learning of students in the class and in providing them effectively with rich learning scenarios – not to replace them.

Contrary to how robots are often presented in other sectors, the social robots presented by Belpaeme and Tanaka (2021[10]) are not meant to replace teachers either, but to support their students for specific learning tasks, in the same spirit as personalisation tools. Not that it could not be possible one day, in a distant future. As of 2021, social robots are mainly effective in accomplishing narrowly defined tasks. They play the role of a teacher assistant, as computers do in their different way. As for telepresence robots, they enable human teachers to be present from a distance. Good (2021[11])provides an excellent case showing how smart technologies for students with special needs may actually create new social relationships between learners and the humans in charge of providing them with appropriate learning tasks – rather than suppress them.

At the system and organisational level, the use of smart technologies follows the same pattern. Early warning systems help predict dropout, but they require a human intervention for “at risk” students not to drop out (Bowers, 2021[19]). Other types of learning analytics used within educational institutions to support decision-making also provide information that needs to be acted upon; they do not make final decisions in the place of administrators and teachers (Ifenthaler, 2021[13]).

This is not to say that smart technologies never make decisions or are not designed for full automation. Personalisation systems, classroom analytics, and early warning systems all make some decisions to enact their next step or recommend one to human beings. But they typically only provide input to decisions. Game-based standardised assessments do more than provide a suggestion: they automatically score the test-takers and assess their skills – as is already the case with traditional computer-based standardised assessments. Blockchain technology does not make decisions, it just records trustfully what a variety of (usually) human actors have done, building on different social processes: accrediting institutions, awarding a qualification or credential, storing the credential on some blockchain, sharing the credential with other parties, verifying the authenticity of the credential, etc. (Smolenski, 2021[25]). But both cases also highlight the relevance of thinking of smart technologies as socio-technical systems, that is, systems in which social and technical features interact and are shaped together.

One of the challenges of game-based standardised assessments will be to create social acceptance, if not full trust, as was the case for traditional standardised assessments. Buckley et al. (2021[14]) note that building valid, reliable, and fair game-based assessments is considerably more complex and challenging than designing traditional standardised tests. While it solves a problem with a clear level of efficacy, one challenge to the widespread use of blockchain for credentialing also relates to social change and legal adaptations: social processes to certify credentials already exist and not everyone may be willing to change those social habits – or cope with the uncertainties attached to any new solution.

There are different ways to acknowledge the fact that most smart technologies are hybrid AI-human systems or that, more generally speaking, smart technologies are better designed and understood as socio-technical systems. One is to clearly communicate that while technology could play a bigger role in the future, it currently needs to be supplemented and controlled by human actions in most cases. The ways to recognise those current realities would be as follows:

• Involving teachers, students and other end users as co-designers in the research and development process would ensure the usefulness and use of smart digital solutions. It also help the people involved in understanding and shaping the social context in which smart education technologies would best be used (the classroom, home, etc.). This should be an aim even when it is challenging to involve end users, for example, students with special needs.
• Public-private partnerships between government, technology researchers within universities and companies, and the education technology industry should be a key characteristic of most research and development projects in this area. This research should go beyond the technology functionalities to analyse how it is used in context and also work on the social and legal adjustments that would be required for their widespread adoption.

Algorithm accuracy

Smart technologies have made and are making very rapid progress, and this book illustrates their potential benefits in a wide range of educational areas both in terms of teaching and learning, and administrative efficacy. Smart technologies often outperform traditional data analysis and technology thanks to more powerful algorithms. Game-based assessments allow skills to be assessed that are difficult to assess through traditional computer-based or paper-and-pencil tests. Personalisation adapts to learner characteristics in ways that pre-existing personalisation methods could not permit – and they possibly do it as well as human teachers. The new algorithms in early warning systems outperform traditional regressions in terms of predictive power and bring visibility to dropout patterns that were not traditionally acknowledged by human administrators or teachers.

Nevertheless, many of the smart technologies presented in the book are not fully mature yet. For example, while some early warning systems now approach good predictive power, Bowers (2021[19])shows that most early warning systems rely on predictors that are no better than a random guess. In the areas of student engagement, D’Mello (2021[9]) points to new approaches that are developed to better measure students’ engagement in learning using facial image analysis and other ways but also notes the inaccuracy of many of the measures used in the field of learning engagement. In the area of classroom analytics, some solutions manage to identify whether learners are working individually or in groups with a very high level of accuracy (90%) but identifying the type of teaching and learning activity remains more challenging (67% of accuracy). Those are just three examples, which are encouraging as accuracy levels can be very high, but show that this is not guaranteed for any AI-powered education application.

In the current state of technology, one policy challenge is to ensure that the developed technology solutions perform their tasks with accuracy – or to get a clear sense of that level of accuracy. In spite of the very rapid advancement of smart technologies, computers and smart education technologies still remain imperfect – though not necessarily more imperfect than humans. Some of those imperfections are expected to be rapidly solved while it may take much longer for others. The contrary would be surprising when looking at technologies considered to be at the education technology frontier. But it is even possible that for some tasks, smart technologies will never become perfectly accurate and remain confounded by false positives, etc. However, the real question is how they compare to humans. After all, human beings perform those tasks with some levels of imperfection too (when they do) as the problems that they are addressing are usually complex. Whether full accuracy should be the expected standard is an open question. Probably this should depend on the social stakes related to the task. In many cases, an “accurate enough” diagnosis or decision should be sufficient.

Given that smart technologies are not fully mature yet and still have some intrinsic limitations, it is important for users and governments to remain aware of those limitations without preventing those technologies from continuing to improve and grow thanks to their actual use. Some possible policy pointers to mitigate the limits of smart technologies while embracing their potential are as follows:

• While smart education technologies can be useful before they are fully accurate (or even without being fully accurate), they should demonstrate a certain level of accuracy in their predictions and diagnoses when they support decision-making – or just be effective enough when they have a different role. Education technology companies could be asked to demonstrate the level of accuracy or effectiveness of their technology solutions with different accuracy requirements depending on the stakes of the supported decision (when they support a decision). Those accuracy requirements should ideally be compared with the current performance of human teachers and administrators.
• When still imperfect in terms of accuracy, smart education technologies should merely inform human decision-making and reflection rather than make fully automated decisions or support a decision process that will rarely deviate from their recommendations, especially for high-stakes solutions. Those technology requirements should have a risk-mitigation rather than no-risk policy, acknowledging that smart technologies can be beneficial even when they are not fully accurate. This may just imply keeping humans in control in the final stages when social stakes are high.

Designing for use

Sometimes, education technology solutions are designed and proposed because they are possible rather than because they are useful and provide clear benefits to end users in education. Most education technology products are mere educational derivatives of solutions designed for other sectors. Even when technology applications are useful and beneficial, some teachers, learners and users may have no interest in using them. Instances of lack of use and lack of usefulness of education technology have given rise to several critiques of education technology (Cuban, 1986[26]; Reich, 2020[27]) even though the increased use of technology in classroom instruction represents one of the biggest changes in classrooms of the 2010s (Vincent-Lancrin et al., 2019[28]).

How can one overcome this problem? Several chapters in this book discuss smart technology solutions that may not be sufficiently useful to be used at scale (given stakeholders’ usual ways of working). One reason for this lack of use lies in the design of the smart technology solutions or in an insufficient understanding of how teachers can use them in their professional practice in ways that support rather than distract them. For example, classroom analytics are useful when they make visible to teachers what is either invisible or not easy to see (either in real time or after class) and when they provide information that they can act upon and interpret (Dillenbourg, 2021[12]).

One aspect of smart technologies that makes them more or less useful to their users relates to how they display the final information to end users. The interface between technology and humans is essential. For example, research shows that different types of dashboards can be more or less effective to support teachers and learners, or more appropriate in certain contexts than others (Molenaar, 2021[8]; Dillenbourg, 2021[12]). Dashboards typically display the final output of the analytics: they can take different forms (e.g. centralised, distributed, ambient) and use different display devices. While the mere appearance of social robots does not seem to matter much, social robots work better than virtual agents as users may relate to them in different ways than with a virtual image or a computer (Belpaeme and Fumihide, 2021[10]). The effectiveness of some solutions such as the ECHOES learning environment, which scaffolds autistic children’s exploration and learning of social communication skills, partly lies in a setting that fosters communication between the autistic child and the adult monitoring the software and learning environment. This important dimension was actually discovered as the tool was tested and improved rather than as a preconceived use case, showcasing the importance in designing and adapting technology solutions with end users (Good, 2021[11]).

In some cases though, the usefulness of education technology may go beyond offering a technical solution to a specific problem. This is why usefulness and algorithmic accuracy are not always related: a solution with an algorithm accurately performing its task may not be so useful, whereas algorithms imperfectly performing their task may be. Changing the stakeholders’ mindset or catalysing some broader change within an institution or an education system may be its usefulness. Generally speaking, innovation is a driver of professional learning and change (Avvisati et al., 2013[29]; Vincent-Lancrin, 2016[30]; Vincent-Lancrin et al., 2019[28]). Smart technologies play the same role.

Ifenthaler (2021[13]) shows that many universities and institutions introduce learning analytics at the institution-wide level in order to change their organisational culture or processes, and perhaps sometimes to foster new collaborations and ways of working between different stakeholders within the institution. Providing a solution to a specific problem or automating their processes may only be a secondary objective. Regardless of their effectiveness in reducing student dropout, one of the benefits that early warning systems (and the related research) has already delivered lies in a better and broader understanding of the circumstances leading students to drop out.

Bowers (2021[19]) shows that the traditional conception of students at risk of dropping out (as students with low and declining grades who do not like school) corresponds to only 38% of actual dropouts in the United States – so that traditional interventions miss out on the majority of students who actually drop out. Beyond providing real-time information, several of the functionalities of classroom analytics provide feedback to teachers on what happened in their class and one of their virtues can be to trigger professional reflection and learning, hopefully followed by behavioural change and improved teaching proficiency (Dillenbourg, 2021[12]).

Beyond design, including the final display of information to learners and other users, cost-benefit analysis is a final way to think about this “use” problem: if cheaper alternatives are available, in budget, time, cognitive load (or whatever relevant metric), new smart technology solutions may remain unattractive compared to existing human or older technology solutions. This is easily understandable when learning analytics predict, diagnose or act with little accuracy, but this can also be true for digital solutions with fully accurate or effective algorithms. Many of the digital tools that support teachers and administrators help them to solve very specific problems and sometimes the existing solutions may outperform the new ones. For example, at the system level, game-based assessments and simulations are likely to supplement rather than replace traditional standardised assessments based on batteries of questions: they are much more expensive to design, have less generalisability, and are only better suited to assess complex competences that are more difficult (or impossible) to assess through cheaper traditional alternatives (Buckley et al., 2021[14]). Smolenski (2021[25]) shows that blockchain has the potential to make the credentialing process more cost-efficient and simple for individuals. The technology is superior to some other technologies when fraud is possible, but not necessarily an adequate solution in all situations.

One last important point on usefulness relates to the affordability of smart technologies for public establishments and individuals. In the context of education systems, digital technology usually has to be very affordable to be bought – and thus be useful and have a chance to be actually used. As noted by Good (2021[11]) in the case of students with special needs, smart technologies should be designed to run on low-cost and widely available platforms (or devices). This is not always the case, but remains one of the conditions for them to be used – and to make their benefits widely accessible. Smolenski (2021[25]) also highlights the importance of open standards in the case of blockchain, partly as a way to make long term costs lower and ensure the solution is more sustainable for the end users (both institutions and individuals). This is true with many other technologies: open standards allow for greater interoperability, more sustainability over time, more competition among vendors, and often lower user costs. Technology solutions that run on widely available platforms are also more affordable and useable than specialised devices.

Several key messages emerge in terms of enhancing the usefulness and usage of smart technologies in education:

• Cost-benefit analysis should typically guide the design and adoption of smart digital solutions for different types of problems, acknowledging that benefits and costs are not just pecuniary.
• While the benefits of any solution may go beyond immediate student academic learning gains, and the costs, beyond merely financial ones, both the expected costs and benefits of smart technologies should be clearly identified and estimated either through research evidence when available and possible or a good theory of action (or theory of change).
• The display (or communication) of the information provided by learning analytics and other technology matters in making smart technologies useful to students, teachers and decision makers. More broadly speaking, the design of the interface between human and smart education is often a key aspect of the useability and impact of the digital solutions on learning or other targeted goals.
• Smart technology solutions should aim to be low cost and run on widely available platforms/devices to be as affordable as possible, possibly using open standards (and interoperability standards). Governments can support the development of those standards, preferably at the international level. Attention to affordability is essential to making smart technologies accessible to all so as not to reinforce the digital divide. Ensuring smart technology solutions can benefit all learners or institutions is key to equity and inclusiveness.

Smart technology and data governance: transparency, fairness and ethics

One key element of any socio-technical system is the broader social context in which the system operates, including its values and principles. Because they rely on large amounts of education data, including, sometimes, personal data such as biological markers, face recognition or expression, etc., or require a permanent monitoring and tracking of learners, classrooms or institutions, common concerns about the development and use of smart technologies relates to data protection and privacy, but also to ethical and political concerns. Could (or should) education establishments and systems become a new version of “Big Brother” for the sake of improved learning outcomes? Can governments and other parties be trusted to use this information for the mere sake of educational improvement – and to enforce strong data protection regimes? What could be the adverse consequences of this use, if not done properly, either in the present or in the future. Could data-rich education technologies, for example, perpetuate and reinforce biases and inequity? Embracing smart technologies implies some trust in how they are used, credible safeguards, and some level of understanding and acceptance of their processes and outputs.

Most OECD countries have strong data protection regulation that ensures that personal education data cannot be shared with (or used by) third parties beyond the educational processes for which they are collected unless certain privacy conditions are met. This is the case in the Europe Union with the General Data Protection Regulation (GDPR) and in the United States with the Family Educational Rights and Privacy Act (FERPA), which have both influenced many other data protection laws across countries. Much of the data concerns administrative micro-data (González-Sancho and Vincent-Lancrin, 2016[23];n.d.[24]). The data protection regime usually extends to vendors providing technology solutions to schools and education administrations. It is noteworthy that the enforcement and implementation of data protection regulations can vary from one country to the other (or even one place to the other within a country). Given these arguably strong safeguards, the fact that privacy and data protection issues remain central in public discussions may point to a lack of trust in how the data are used (or could be used) within the education system and beyond.

Data protection is just one aspect of data governance though. One important question relates to the relationship between governments, the data subjects (who are usually the users of (sometimes mandatory) education services), and the private sector that usually develops the smart education technologies. This relates to questions of data ownership and of competition policy in the digital world: how should administrative and other education data be shared across education technology companies and public researchers to allow for progress and for a sufficient amount of competition in the sector of smart education technologies? How can the interests of learners and other individuals be best preserved in this context? The different solutions proposed for other sectors than education could probably be adapted to education (OECD, 2019[31]; 2019[1]).

Ethical discussions should normally concern what is not regulated within a country and thus aspects for which individuals and authorities have more freedom of action. This is how we interpret the question of “ethics” in educational AI.

The regulation of algorithms is usually not as strong as regulation for data protection. A major concern about algorithms is that they could be biased and have an undesirable social impact for some population groups (based on gender, ethnicity, socio-economic status, etc.) – but also that they could be flawed or just reinforce past human biases rather than reflect current societal values. A usual requirement is to ensure they are transparent and open, and that their “decisions” can be explained and challenged when automated. In the case of the EU General Data Protection Regulation, the regulatory language about algorithms is ambiguous (articles 13-15, 22, and recital 73) and lawyers are still debating what those articles imply in terms of “right of explanation” (transparency) and of the possibility to opt out of or challenge “automated decisions” for citizens. In the European Union, only the French and Hungarian laws have an explicit law about the “right of explanation” (with French law requiring both ex-ante and ex-post explanation in an intelligible way, and Hungarian law, some level of explanation) (Malgieri, 2019[32]). In the United States, there is no regulation about algorithms and their requirements as part of FERPA (or other regulation). Most OECD countries do not have clear regulatory requirements about them as of 2021.

Because algorithms based on machine learning are trained with historical data, many observers are concerned they will reproduce past biased (human) practices, as has apparently been the case in some countries in domains other than education (finance, justice, etc.) (O’Neil, 2016[33]). Several guidelines have been developed to avoid these pitfalls, which can happen at different steps of the process: measurement (data collection or labelling), model learning (when machine learning is involved), and action (when the algorithms detect, diagnose and act, for example). Different possible measures of fairness are also possible, which makes the issue even more complex (Kizilcec and Lee, 2020[34]; Baker and Hawn, 2021[35]). Ifenthaler (2021[13]) mentions several check lists of good practice and ethics for learning analytics. Bowers (2021[19]) points to the “open algorithm” movement in the area of early warning systems and notably to two overlapping sets of criteria to ensure transparency, verification and replicability of algorithms: the AAAA (accurate, accessible, actionable and accountable) and FAIR (findable, accessible, interoperable and reproducible) principles. Molenaar (2021[8]) also points to the importance of transparency to govern learning analytics and algorithms. As was shown in 2019 with some difficulties around exams and grade assignment for university admission, when it comes to high-stakes automated decisions, transparency is also about initiating an early dialogue about the criteria, expected social outcomes, relevance and acceptability of smart algorithms with diverse stakeholders from experts through to final users and other social bodies (Box 1.3). In some cases, the algorithms can be human-coded rather than involve AI techniques.

For example, given that accurate predictors for early warning systems can rely on minimal information that does not include information about gender, race and socio-economic status (Bowers, 2021[19]), it may not be considered ethical (or even necessary) to include these kinds of indicators to diagnose dropout in early warning systems unless they improve significantly the performance of the algorithms. On the other hand, results that do not include any personal information about learners may still lead to biased or socially/politically undesirable outcomes. Ethical concerns should therefore include a verification and discussion of the effects of smart technologies on different groups and ensure they are aligned with countries’ social and political principles. As few people are, in practice, able to verify the effects and impacts of algorithms, some independent groups of stakeholders may be responsible for or even assigned this task. While anyone should be allowed to do it in the frame of an open algorithm culture (at least when algorithms lead to a decision or a quasi-decision), education researchers, non-governmental organisations, but also, possibly, independent governmental agencies, could play an enhanced role in this area.

In the case of the most advanced applications of learning analytics based on a continuous monitoring of individuals (e.g. engagement, self-regulation, classroom orchestration, game-based assessments), another question is whether stakeholders feel comfortable with some aspects of the applications even if they are legal. While the tracking and data collection necessary to power learning analytics focusing on student engagement, self-regulation or classroom orchestration have to comply with domestic data protection regulations (and algorithm regulation, if any), the question is how to make them compatible with the political values of the country where they are implemented. This may require some imagination in terms of data protection arrangements (such as deleting immediately the data once processed). As in the exam case mentioned above, this also requires a social negotiation with all stakeholders including transparency about the data collection and how they are used. This is not just a matter of regulation or even perceived ethics. Even within the same country, what can be acceptable in some communities may not be in others depending on how the smart technologies were introduced (Box 1.5).

Another pragmatic and ethical issue relates to the use of the information generated by data analytics about teachers and other staff. Typically, staff within an organisation do not benefit of the same data protection and privacy regulation as students and other users. While smart technologies and learning analytics have the potential to provide feedback and support to teachers and other education stakeholders to make better decisions and improve their professional practices, they could also be used against them and unintentionally lead to undesirable social behaviours. Classroom analytics can be used to monitor teachers’ professional behaviour, and sometimes identify shortcoming in how they orchestrate learning in their classroom (Dillenbourg, 2021[12]). Should this information be used to sanction or support them? One possible intervention for students at risk of dropping out also consists of expelling or pushing them out of the school detecting this risk, which eventually enhances their likelihood to drop out eventually (Bowers, 2021[19]). Given their possible intrusive surveillance nature, the adoption of smart technologies relies on some level of trust in their positive, human-empowering ethical use. Should their voluntary use have adverse effects on teachers, school principals and decision-makers, they may appear as less acceptable and be resisted. Ethics about their use may require either full confidentiality about their results or a discussion and clear disclosure of how they can affect the staff using them. (This may also depend on their expected accuracy and effectiveness). As in the case of the information provided by longitudinal information systems, two different philosophies are possible. Some argue that their information should be used to reward and sanction stakeholders as an accountability mechanism, which is also a way to make them pay attention to the provided information. Others argue that the information should not be used to reward or sanction stakeholders as this may lead to their opposing the use of the information or just incentivise them to try to “game the system” – and thus lead to unethical behaviour. The jury is still out on what the best strategy is.

Building on the fact that smart technologies are socio-technical systems, their adoption and use will typically require some level of trust in teachers, schools, governments, and other stakeholders – and some attention paid to their possible adverse effects. Building trust should build both on regulation and the establishment of ethical practices, including different governance mechanisms:

• Regulation on data protection and privacy, as well as practical guidance on how to implement those provisions with different types of data and different levels of resources and competence;
• Regulation or good practice on data governance, and notably a strategy that makes data collected through smart technologies available to researchers and possibly to competitors to the vendors whose technology solution collects them;
• Regulation or guidelines about the transparency, openness and replicability of algorithms, as well as funding and support for the verification of the design and the final results of algorithms by independent parties;
• A risk-management approach to data protection and algorithm supervision that strikes an appropriate balance between risk-taking and data/usage sensitivity, acknowledging that regulations (or ethical guidance) that are too risk-averse will hinder the development of smart education technologies within a country or region and prevent reaping their possible benefits. The regulations above should be updated as smart technologies continue their rapid progress, and a close collaboration between researchers on data protection and on smart technologies may help co-design good solutions.

Infrastructure and public good

Smart technologies do usually require a strong Internet, computer and data infrastructure. Artificial intelligence systems, adaptive learning systems providing real-time information to teachers and learners, game-based assessments, blockchain, social robots, all require sufficient computer hardware in schools and universities (including students’ hardware under “bring your own device” policies), but also increasingly at home, as well as a bandwidth and networking capacity capable of stable and acceptable data transfer speeds. This also implies some level of investment in IT staff to deal with the maintenance of the hardware in (or across) educational establishments. Hardware is a necessary basis, but there is more to what a digital learning infrastructure should be.

Policy makers need to think about infrastructure also in terms of digital resources, both content and tools, that should be publicly provided to citizens, students, and educational institutions – either directly or indirectly through subsidies or funding to institutions. Budget constraints can limit these aspirations, but it remains important to define what should be the core of digital resources for all, and what should be accessible privately. A new question is the extent to which smart technologies should be part of the core digital education infrastructure. Is it enough to make some educational resources available to the public or should they also have the personalised learning features that some intelligent tutoring systems propose?

A final aspect of a digital infrastructure lies in people’s “digital skills”, that is, the ability to use digital resources as part of one’s professional practice. In the case of teachers and professors, digital skills are less about mastering the technology than about integrating technology tools, resources and outputs in their pedagogy. Unless fully automated, technology solutions are indeed mere tools for human teaching, learning or managing education systems. Professional learning opportunities for staff, both through training and organised continuous professional learning opportunities, are thus a final aspect that should be an integral part of a strong digital infrastructure.

While university researchers and researchers in public technology agencies may have the skills to develop some of those smart technologies, governments, schools and universities will typically rely on private education technology companies to provide and maintain them (hopefully through public-private partnerships, as mentioned above).

Beyond this investment in its digital infrastructure and connectivity for all, governments still have two important responsibilities in dealing with the private sector:

• Ensuring through its procurement policies and other incentives that publicly funded or purchased smart technologies are available to schools with affordable costs.
• Ensuring that some key techniques or discoveries of those smart education technologies become or remain a public, international good and allow more actors internationally to develop new interoperable solutions that will help improve education for all across the globe.
• Ensuring that staff have the learning opportunities to properly use the smart technologies and digital resources at their disposal.

Research and development

This book shows how smart technologies can be beneficial to education in various ways. At the same time, a common thread through the different chapters of the book is the insufficient amount of strong evidence about the effectiveness of the various uses of smart technologies. As noted above, there is a relative lack of evidence and transparency about the actual algorithmic accuracy of the technology solutions but more evidence on their effective use in real-life educational settings should also be produced. Social robots are one of the few areas where meta-analyses and some strong evidence results could be established about some specific use of technology to improve learning outcomes (Belpaeme and Fumihide, 2021[10]).

While education research and development is needed, new approaches may be warranted.

Policy makers and research funders should remain aware that technology is not to be tested in itself. As a mere tool that is part of a socio-technical system, the appropriate research question is rarely whether some specific technological application is “effective” but more about identifying and testing whether some specific uses yield positive outcomes. Usually, when it comes to instruction, the research question will be about pedagogy and how technology supports this pedagogy rather than about the technology itself. When it comes to administrative processes, the question may be about the interventions supported or induced by the smart technology rather than the technology itself.

Some specific technological applications of smart technology have been (and can be) researched through robust experimental research designs. However, as the development of smart technologies (and computing power) advances quickly, the evidence about their effectiveness may be quickly outdated and irrelevant as the technology gets upgraded.

As other digital technologies, smart technologies allow for new types of speedy research when they are done online: A/B research. Typically, this consists of trying two different designs (A and B) of a given technology with two different groups to identify quickly which one works best (when there are large numbers of use/exposure online). This approach can be extended to classrooms to evaluate the effectiveness in using different designs of some given smart technology or of different smart technologies pursuing similar objectives in a classroom or at the system level, factoring in the complex human behaviours that are attached to their use.

Should one wait for strong evidence before using technology in education? In practice, this may be too restrictive an approach. Requirements for strong evidence before the introduction of any innovation are often a way to protect the status quo of practices whose effectiveness is rarely proven. Disciplined innovation can happen in other ways too, notably through a proportionate use of effectiveness evidence. In the case of some smart technologies, usual evaluation processes may not be the most appropriate and sometimes it may be difficult to carry them out. It would, for example, be difficult to evaluate the effectiveness of early warning systems through a randomised control trial – one reason being that systems may have to be different from one site to the other to accommodate local circumstances. The speed of development of the technology will also typically mean that experimental results will either validate or invalidate technology solutions that are already outdated by improved versions of the solution (or new technologies altogether). Perhaps another approach would be to evaluate families of technology solutions used in certain ways.

The evidence standards required for different types of solutions should probably vary depending on the possible stakes and harm coming from the solution. Smart technologies supported by a good “theory of action” or good “underlying theory” may be acceptable for use even without a strong evidence base for low-stakes applications. When there is little underlying theory, no clear theory of action, no related basis of evidence, policy makers and stakeholders should be more reluctant to allow their use in public settings. Finally, in the case of high-stakes situations, the highest evidence standards should be in place, although algorithmic effectiveness should be compared to evidence of human effectiveness when performing similar tasks.

To further address this question, governments could:

• Invest in educational research about the use of technology in education in real life settings with a focus on pedagogy or administrative processes rather than the technology itself;
• Develop domestic and possibly international repositories of evidence on different types of uses of technology in education and different types of families of education technology;
• Develop common standards about good educational research and development in educational technology, acknowledging some of the specificities of the field.

Effectiveness

Attending school or university does not always translate into as much academic learning as one would hope for. The OECD Programme for International Student Assessment (PISA) has shown that attending school may actually lead to very different levels of learning outcomes across countries. While there is no similar evidence at the higher education level as yet, this is arguably the same at that level too. One of the key promises of smart technologies is to enhance the effectiveness of teaching and learning for better student learning.

In the classroom, applications that directly support student learning show early promise. Personalised learning aims to provide all students with the appropriate curriculum or task, and scaffold them within a task, based on a diagnosis of their knowledge and knowledge gaps. This is not only done at the academic level, focusing on the “what”, but increasingly takes into account how students learn and factors such as self-regulation, motivation or effort (Molenaar, 2021[8]). Engagement is key for learning, and solutions to keep students engaged within digital or physical learning environments are being developed to identify their affective states during learning and nudge them towards re-engagement when they seem to disengage (D’Mello, 2021[9]). Social robots perform similar tasks in different ways: they can use adaptive learning to tutor students with natural language, but they can also teach, or motivate them to learn by playing the role of a peer student. They support teachers by enabling the implementation of different types of teaching and learning strategies (Belpaeme and Fumihide, 2021[10]). Finally, smart technologies give students with impairments and special needs access to curriculum materials and allow those students to participate in learning activities to an extent that was not possible before, here again increasing the effectiveness of education (Good, 2021[11]).

Those solutions can be used and remain helpful outside of the classroom too, either for homework, as automated private tutoring or practice solutions, and for lifelong learning. In fact, the largest market for educational technology companies is the consumer market targeting students and parents directly, either for recreational learning activities or for tutoring or test preparation.

A second promise of learning effectiveness comes from classroom analytics that support teachers in providing more effective teaching. This is still a work in progress but many applications already show how a variety of solutions could support teachers in better using their time in class, for example, by suggesting when it is a good time to shift to the next teaching or learning activity, who would require their attention the most, how they could engage the whole class in collaborative learning activities. While classroom orchestration solutions can help teachers in real time, they also provide feedback on their own practice, for example, how much they talk, to whom, or how they divide their time between different types of activities (Dillenbourg, 2021[12]). Both real-time and post-hoc feedback are akin to personal professional learning opportunities for the teacher in question, and have the significant advantage of being about the specific teacher who was (digitally) observed rather than about theoretical or general teaching practice. In that sense, smart technology has real potential to improve the teaching practice of all individual teachers, and subsequently the learning outcomes of their students.

At the organisational and system levels, smart technologies also hold promise in making education more effective. While this remains relatively rare (Ifenthaler, 2021[13]), smart technologies can be integrated in most dimensions of school activities, providing administrators, teachers and learners with feedback to manage school resources as well as improve the effectiveness of teaching and learning (see Box 1.1). The rise of a new generation of assessments powered by AI also opens new avenues for recognising and evaluating competences that were hard to assess through paper and pencil tests. This could accompany most education systems in their shift towards emphasising skills (in addition to the traditional emphasis on knowledge). Game-based assessments and simulations allow assessments to be designed to be more realistic but also to assess skills such as complex problem solving, creativity or collaboration in new ways (Buckley et al., 2021[14]).

Finally, the emergence of longitudinal education data systems that follow students through the course of their studies also allows for more effective policy and organisational interventions and a better design of educational offerings. For example, in the United States, the analysis of community college graduation rates, the success of their student placement strategies in “remedial courses” and students’ study patterns as part-time or full-time enrolment led to revisiting what the educational experience of community college students actually is and “redesigning” community colleges (Bailey, Smith-Jaggars and Jenkins, 2015[15]). As in other sectors (OECD, 2019[16]), the use of data supports policy design and interventions.

Equity

Smart technologies can help education systems provide more equitable learning opportunities. In this respect, smart technologies are more ambivalent. One the one hand, they clearly do or could help reduce inequity both by increasing access to learning opportunities for all and improving learning effectiveness for those who need it the most. On the other hand, without a widespread and equitable availability of smart technologies, inequity could also rise. They may also leave achievement gaps unchanged or even widened, depending on their differential impact on different learners.

Let us start with the difficulties. There are at least two reasons why technology may have a negative effect on equity. The first, obvious reason lies in the difference in access to devices and connectivity by students from different groups, notably students from lower socio-economic backgrounds. These students may not have the devices, the connectivity or the resources that allow accessing and using smart technologies either at the school they attend or at home. A second reason is that, if technology (e.g. personalised learning) works the same for everyone, those who start with stronger prior knowledge can maintain their advantage or even make faster progress than those with less prior knowledge. In spite of supporting students with less prior knowledge, it is thus possible that technology could help more advanced students more. This would widen rather than reduce the achievement gap.

There are also many reasons to believe that smart technologies can advance the equity agenda.

First, learning technology can expand access to learning opportunities. Educational platforms proposing open educational resources (Orr, Rimini and van Damme, 2015[17]) or massive open online course (MOOC) platforms are good examples. They allow learners to access learning materials with a quality that may be superior to what they can access locally. While many studies have shown that this increased access has not decreased inequity at scale given the low take-up and the fact that most users are already well educated, a recent systematic review of their effect on equity provides a more optimistic perspective, notably for non-English MOOCs or open educational resources (Lambert, 2020[18]).

As importantly, smart technologies can reduce inequity by facilitating the inclusion of students with special needs and by adapting learning to different learning styles. Technology has, for example, made it much easier to support the diagnosis of learning difficulties such as dysgraphia, and remedial digital responses have also been developed. A variety of smart technologies applied to learning solutions also make it easier for blind or visually impaired students as well as deaf or hard-of-hearing students to access learning materials and easily perform the educational tasks required from other students. Artificial intelligence enabling speech to text (and vice versa) or automatic subtitles are the most obvious examples. Learning technologies also tackle more difficult issues and support the socio-emotional (and thus the subsequent academic) learning of autistic children. They increasingly propose ways to help children with attention deficit hyperactivity disorder (ADHD) to self-regulate and better benefit from their schooling. One caveat here is that inclusion is not just about the individual “fitting in” but also for society to be more inclusive and open to differences (Good, 2021[11]). Technology encourages that by enabling students with special needs to study in a traditional (and inclusive) learning environment, which also changes peoples’ view on disability and special needs.

Second, solutions such as early warning systems are entirely focused on reducing inequity by helping students at risk of dropping out from high school (or university) to graduate – students who drop out typically come from disadvantaged and minority backgrounds. Early warning systems also allow designing appropriate interventions by identifying the factors or indicators most likely to predict dropout (Bowers, 2021[19]). Some use of learning analytics within institutions, for example, to monitor student engagement or redesign study programmes, could also have the same effects, should the educational institution pay particular attention to inequity (Ifenthaler, 2021[13]).

Third, the use of learning analytics as exemplified by personalisation at the individual level, be it using intelligent tutoring systems or learning analytics to keep students engaged in learning, all hold promise in reducing inequity, notably by supporting students with less prior knowledge to learn at the right pace. Box 1.3 gives an example of an online solution that reduced the learning gap between the strongest and weakest students in mathematics at the beginning of the intervention. There is, however, little evidence that adaptive learning generally reduces achievement gaps between students. Classroom analytics can also give feedback to teachers on how they could improve their teaching; specifically how and when to pay more attention to different group of students in their class, based on their academic level, gender, ethnicity, etc. Adaptive learning technology can also help students practice and make progress at home, outside of the classroom, supported by intelligent tutoring systems. This may be particularly important for students coming from households where parents can support their students less effectively with their school work, be it directly or indirectly.

Efficiency

In most other sectors than education, smart technologies are used as a tool to enhance the cost-efficiency of operations, notably by automating a number of tasks and processes, making services faster and often cheaper (OECD, 2019[16]). While education might be behind most other sectors in this respect, digitalisation is also making many educational processes more efficient as interactions between stakeholders and educational institutions become increasingly automated. As noted above, also in teaching and learning, some degree of automation is gaining ground. To what extent will digitalisation allow for enhanced cost-efficiency and productivity in education?

Any discussion of cost-efficiency should keep in mind that technology incurs investment and maintenance costs, which have to be compared with the costs of current arrangements. Digital technology has not always delivered on its cost-efficiency promises in the past because one has sometimes forgotten that, beyond the initial investment, it needs to be continuously upgraded, maintained, etc.

Nevertheless, as in other sectors, there are good reasons to believe that smart technologies could increase cost-efficiency in education.

One example lies in the application (and admission) process to educational institutions. They are increasingly undertaken through digital platforms, especially in higher education, where a “matching” (or selection) process is often necessary. In cases of open admission institutions, when no selection is required, implementing seamless automated processes is even easier. The implementation of the National Education Information System (NEIS) in Korea, an e-government system allowing, inter alie, for the digital transfer of students’ academic records from one school to the other (as well as from school to university) was estimated to save USD 237 million a year in 2010 (KERIS, 2010[22]).

A second area where digitalisation could possibly lead to cost-efficiency is the provision of verifiable degrees and other credentials on blockchain. The gradual development of an infrastructure for digital credentials and the adoption of open standards will gradually lead to a different way of certifying and holding degrees, with individuals being able to manage their qualifications themselves. This is one of the strongest and most immediate arguments of cost-efficiency in the different digital solutions examined in this book.

A third area where cost-efficiency is underway is the collection of system-level statistical information. While statistical information often relied on the establishment of statistical panels (of representative samples of individuals or institutions) and often involved multiple handlings of the same data, the use of administrative data combined with the interoperability of diverse systems has made it much easier to get statistical information from operational services in almost real time (González-Sancho and Vincent-Lancrin, 2016[23]; n.d.[24]).

At the end of the day, a cost-benefit analysis comparing the benefits of smart technology, including the non-financial ones mentioned above, to that of an existing solution, will determine how cost-efficient it is for a given service (or educational goal).

Smart technologies as socio-technical systems

Whether they are already available or under development, most of the smart education technologies for teaching and learning covered in this book do not aim to replace teachers or human beings. They were in fact developed with the current education model in mind. One common thread of all analyses in the book is that most teaching and learning solutions are designed as hybrid human-AI systems and require teacher-student interactions and human oversight of the machine at different points. Molenaar (2021[8]) offers a model to better understand the continuum between fully automated and teacher-only education. Most advanced personalisation solutions require teacher intervention or alert teachers when they should intervene, for example, because students are still struggling or they need to move on to another step of their learning process. Most solutions to support classroom orchestration are also hybrid solutions that merely scaffold teachers in implementing rich learning scenarios for their students. As Dillenbourg (2021[12]) puts it, “there is a teacher in the loop” and classroom analytics are designed to support teachers in orchestrating the teaching and learning of students in the class and in providing them effectively with rich learning scenarios – not to replace them.

Contrary to how robots are often presented in other sectors, the social robots presented by Belpaeme and Tanaka (2021[10]) are not meant to replace teachers either, but to support their students for specific learning tasks, in the same spirit as personalisation tools. Not that it could not be possible one day, in a distant future. As of 2021, social robots are mainly effective in accomplishing narrowly defined tasks. They play the role of a teacher assistant, as computers do in their different way. As for telepresence robots, they enable human teachers to be present from a distance. Good (2021[11])provides an excellent case showing how smart technologies for students with special needs may actually create new social relationships between learners and the humans in charge of providing them with appropriate learning tasks – rather than suppress them.

At the system and organisational level, the use of smart technologies follows the same pattern. Early warning systems help predict dropout, but they require a human intervention for “at risk” students not to drop out (Bowers, 2021[19]). Other types of learning analytics used within educational institutions to support decision-making also provide information that needs to be acted upon; they do not make final decisions in the place of administrators and teachers (Ifenthaler, 2021[13]).

This is not to say that smart technologies never make decisions or are not designed for full automation. Personalisation systems, classroom analytics, and early warning systems all make some decisions to enact their next step or recommend one to human beings. But they typically only provide input to decisions. Game-based standardised assessments do more than provide a suggestion: they automatically score the test-takers and assess their skills – as is already the case with traditional computer-based standardised assessments. Blockchain technology does not make decisions, it just records trustfully what a variety of (usually) human actors have done, building on different social processes: accrediting institutions, awarding a qualification or credential, storing the credential on some blockchain, sharing the credential with other parties, verifying the authenticity of the credential, etc. (Smolenski, 2021[25]). But both cases also highlight the relevance of thinking of smart technologies as socio-technical systems, that is, systems in which social and technical features interact and are shaped together.

One of the challenges of game-based standardised assessments will be to create social acceptance, if not full trust, as was the case for traditional standardised assessments. Buckley et al. (2021[14]) note that building valid, reliable, and fair game-based assessments is considerably more complex and challenging than designing traditional standardised tests. While it solves a problem with a clear level of efficacy, one challenge to the widespread use of blockchain for credentialing also relates to social change and legal adaptations: social processes to certify credentials already exist and not everyone may be willing to change those social habits – or cope with the uncertainties attached to any new solution.

There are different ways to acknowledge the fact that most smart technologies are hybrid AI-human systems or that, more generally speaking, smart technologies are better designed and understood as socio-technical systems. One is to clearly communicate that while technology could play a bigger role in the future, it currently needs to be supplemented and controlled by human actions in most cases. The ways to recognise those current realities would be as follows:

• Involving teachers, students and other end users as co-designers in the research and development process would ensure the usefulness and use of smart digital solutions. It also help the people involved in understanding and shaping the social context in which smart education technologies would best be used (the classroom, home, etc.). This should be an aim even when it is challenging to involve end users, for example, students with special needs.
• Public-private partnerships between government, technology researchers within universities and companies, and the education technology industry should be a key characteristic of most research and development projects in this area. This research should go beyond the technology functionalities to analyse how it is used in context and also work on the social and legal adjustments that would be required for their widespread adoption.

Algorithm accuracy

Smart technologies have made and are making very rapid progress, and this book illustrates their potential benefits in a wide range of educational areas both in terms of teaching and learning, and administrative efficacy. Smart technologies often outperform traditional data analysis and technology thanks to more powerful algorithms. Game-based assessments allow skills to be assessed that are difficult to assess through traditional computer-based or paper-and-pencil tests. Personalisation adapts to learner characteristics in ways that pre-existing personalisation methods could not permit – and they possibly do it as well as human teachers. The new algorithms in early warning systems outperform traditional regressions in terms of predictive power and bring visibility to dropout patterns that were not traditionally acknowledged by human administrators or teachers.

Nevertheless, many of the smart technologies presented in the book are not fully mature yet. For example, while some early warning systems now approach good predictive power, Bowers (2021[19])shows that most early warning systems rely on predictors that are no better than a random guess. In the areas of student engagement, D’Mello (2021[9]) points to new approaches that are developed to better measure students’ engagement in learning using facial image analysis and other ways but also notes the inaccuracy of many of the measures used in the field of learning engagement. In the area of classroom analytics, some solutions manage to identify whether learners are working individually or in groups with a very high level of accuracy (90%) but identifying the type of teaching and learning activity remains more challenging (67% of accuracy). Those are just three examples, which are encouraging as accuracy levels can be very high, but show that this is not guaranteed for any AI-powered education application.

In the current state of technology, one policy challenge is to ensure that the developed technology solutions perform their tasks with accuracy – or to get a clear sense of that level of accuracy. In spite of the very rapid advancement of smart technologies, computers and smart education technologies still remain imperfect – though not necessarily more imperfect than humans. Some of those imperfections are expected to be rapidly solved while it may take much longer for others. The contrary would be surprising when looking at technologies considered to be at the education technology frontier. But it is even possible that for some tasks, smart technologies will never become perfectly accurate and remain confounded by false positives, etc. However, the real question is how they compare to humans. After all, human beings perform those tasks with some levels of imperfection too (when they do) as the problems that they are addressing are usually complex. Whether full accuracy should be the expected standard is an open question. Probably this should depend on the social stakes related to the task. In many cases, an “accurate enough” diagnosis or decision should be sufficient.

Given that smart technologies are not fully mature yet and still have some intrinsic limitations, it is important for users and governments to remain aware of those limitations without preventing those technologies from continuing to improve and grow thanks to their actual use. Some possible policy pointers to mitigate the limits of smart technologies while embracing their potential are as follows:

• While smart education technologies can be useful before they are fully accurate (or even without being fully accurate), they should demonstrate a certain level of accuracy in their predictions and diagnoses when they support decision-making – or just be effective enough when they have a different role. Education technology companies could be asked to demonstrate the level of accuracy or effectiveness of their technology solutions with different accuracy requirements depending on the stakes of the supported decision (when they support a decision). Those accuracy requirements should ideally be compared with the current performance of human teachers and administrators.
• When still imperfect in terms of accuracy, smart education technologies should merely inform human decision-making and reflection rather than make fully automated decisions or support a decision process that will rarely deviate from their recommendations, especially for high-stakes solutions. Those technology requirements should have a risk-mitigation rather than no-risk policy, acknowledging that smart technologies can be beneficial even when they are not fully accurate. This may just imply keeping humans in control in the final stages when social stakes are high.

Designing for use

Sometimes, education technology solutions are designed and proposed because they are possible rather than because they are useful and provide clear benefits to end users in education. Most education technology products are mere educational derivatives of solutions designed for other sectors. Even when technology applications are useful and beneficial, some teachers, learners and users may have no interest in using them. Instances of lack of use and lack of usefulness of education technology have given rise to several critiques of education technology (Cuban, 1986[26]; Reich, 2020[27]) even though the increased use of technology in classroom instruction represents one of the biggest changes in classrooms of the 2010s (Vincent-Lancrin et al., 2019[28]).

How can one overcome this problem? Several chapters in this book discuss smart technology solutions that may not be sufficiently useful to be used at scale (given stakeholders’ usual ways of working). One reason for this lack of use lies in the design of the smart technology solutions or in an insufficient understanding of how teachers can use them in their professional practice in ways that support rather than distract them. For example, classroom analytics are useful when they make visible to teachers what is either invisible or not easy to see (either in real time or after class) and when they provide information that they can act upon and interpret (Dillenbourg, 2021[12]).

One aspect of smart technologies that makes them more or less useful to their users relates to how they display the final information to end users. The interface between technology and humans is essential. For example, research shows that different types of dashboards can be more or less effective to support teachers and learners, or more appropriate in certain contexts than others (Molenaar, 2021[8]; Dillenbourg, 2021[12]). Dashboards typically display the final output of the analytics: they can take different forms (e.g. centralised, distributed, ambient) and use different display devices. While the mere appearance of social robots does not seem to matter much, social robots work better than virtual agents as users may relate to them in different ways than with a virtual image or a computer (Belpaeme and Fumihide, 2021[10]). The effectiveness of some solutions such as the ECHOES learning environment, which scaffolds autistic children’s exploration and learning of social communication skills, partly lies in a setting that fosters communication between the autistic child and the adult monitoring the software and learning environment. This important dimension was actually discovered as the tool was tested and improved rather than as a preconceived use case, showcasing the importance in designing and adapting technology solutions with end users (Good, 2021[11]).

In some cases though, the usefulness of education technology may go beyond offering a technical solution to a specific problem. This is why usefulness and algorithmic accuracy are not always related: a solution with an algorithm accurately performing its task may not be so useful, whereas algorithms imperfectly performing their task may be. Changing the stakeholders’ mindset or catalysing some broader change within an institution or an education system may be its usefulness. Generally speaking, innovation is a driver of professional learning and change (Avvisati et al., 2013[29]; Vincent-Lancrin, 2016[30]; Vincent-Lancrin et al., 2019[28]). Smart technologies play the same role.

Ifenthaler (2021[13]) shows that many universities and institutions introduce learning analytics at the institution-wide level in order to change their organisational culture or processes, and perhaps sometimes to foster new collaborations and ways of working between different stakeholders within the institution. Providing a solution to a specific problem or automating their processes may only be a secondary objective. Regardless of their effectiveness in reducing student dropout, one of the benefits that early warning systems (and the related research) has already delivered lies in a better and broader understanding of the circumstances leading students to drop out.

Bowers (2021[19]) shows that the traditional conception of students at risk of dropping out (as students with low and declining grades who do not like school) corresponds to only 38% of actual dropouts in the United States – so that traditional interventions miss out on the majority of students who actually drop out. Beyond providing real-time information, several of the functionalities of classroom analytics provide feedback to teachers on what happened in their class and one of their virtues can be to trigger professional reflection and learning, hopefully followed by behavioural change and improved teaching proficiency (Dillenbourg, 2021[12]).

Beyond design, including the final display of information to learners and other users, cost-benefit analysis is a final way to think about this “use” problem: if cheaper alternatives are available, in budget, time, cognitive load (or whatever relevant metric), new smart technology solutions may remain unattractive compared to existing human or older technology solutions. This is easily understandable when learning analytics predict, diagnose or act with little accuracy, but this can also be true for digital solutions with fully accurate or effective algorithms. Many of the digital tools that support teachers and administrators help them to solve very specific problems and sometimes the existing solutions may outperform the new ones. For example, at the system level, game-based assessments and simulations are likely to supplement rather than replace traditional standardised assessments based on batteries of questions: they are much more expensive to design, have less generalisability, and are only better suited to assess complex competences that are more difficult (or impossible) to assess through cheaper traditional alternatives (Buckley et al., 2021[14]). Smolenski (2021[25]) shows that blockchain has the potential to make the credentialing process more cost-efficient and simple for individuals. The technology is superior to some other technologies when fraud is possible, but not necessarily an adequate solution in all situations.

One last important point on usefulness relates to the affordability of smart technologies for public establishments and individuals. In the context of education systems, digital technology usually has to be very affordable to be bought – and thus be useful and have a chance to be actually used. As noted by Good (2021[11]) in the case of students with special needs, smart technologies should be designed to run on low-cost and widely available platforms (or devices). This is not always the case, but remains one of the conditions for them to be used – and to make their benefits widely accessible. Smolenski (2021[25]) also highlights the importance of open standards in the case of blockchain, partly as a way to make long term costs lower and ensure the solution is more sustainable for the end users (both institutions and individuals). This is true with many other technologies: open standards allow for greater interoperability, more sustainability over time, more competition among vendors, and often lower user costs. Technology solutions that run on widely available platforms are also more affordable and useable than specialised devices.

Several key messages emerge in terms of enhancing the usefulness and usage of smart technologies in education:

• Cost-benefit analysis should typically guide the design and adoption of smart digital solutions for different types of problems, acknowledging that benefits and costs are not just pecuniary.
• While the benefits of any solution may go beyond immediate student academic learning gains, and the costs, beyond merely financial ones, both the expected costs and benefits of smart technologies should be clearly identified and estimated either through research evidence when available and possible or a good theory of action (or theory of change).
• The display (or communication) of the information provided by learning analytics and other technology matters in making smart technologies useful to students, teachers and decision makers. More broadly speaking, the design of the interface between human and smart education is often a key aspect of the useability and impact of the digital solutions on learning or other targeted goals.
• Smart technology solutions should aim to be low cost and run on widely available platforms/devices to be as affordable as possible, possibly using open standards (and interoperability standards). Governments can support the development of those standards, preferably at the international level. Attention to affordability is essential to making smart technologies accessible to all so as not to reinforce the digital divide. Ensuring smart technology solutions can benefit all learners or institutions is key to equity and inclusiveness.

Smart technology and data governance: transparency, fairness and ethics

One key element of any socio-technical system is the broader social context in which the system operates, including its values and principles. Because they rely on large amounts of education data, including, sometimes, personal data such as biological markers, face recognition or expression, etc., or require a permanent monitoring and tracking of learners, classrooms or institutions, common concerns about the development and use of smart technologies relates to data protection and privacy, but also to ethical and political concerns. Could (or should) education establishments and systems become a new version of “Big Brother” for the sake of improved learning outcomes? Can governments and other parties be trusted to use this information for the mere sake of educational improvement – and to enforce strong data protection regimes? What could be the adverse consequences of this use, if not done properly, either in the present or in the future. Could data-rich education technologies, for example, perpetuate and reinforce biases and inequity? Embracing smart technologies implies some trust in how they are used, credible safeguards, and some level of understanding and acceptance of their processes and outputs.

Most OECD countries have strong data protection regulation that ensures that personal education data cannot be shared with (or used by) third parties beyond the educational processes for which they are collected unless certain privacy conditions are met. This is the case in the Europe Union with the General Data Protection Regulation (GDPR) and in the United States with the Family Educational Rights and Privacy Act (FERPA), which have both influenced many other data protection laws across countries. Much of the data concerns administrative micro-data (González-Sancho and Vincent-Lancrin, 2016[23];n.d.[24]). The data protection regime usually extends to vendors providing technology solutions to schools and education administrations. It is noteworthy that the enforcement and implementation of data protection regulations can vary from one country to the other (or even one place to the other within a country). Given these arguably strong safeguards, the fact that privacy and data protection issues remain central in public discussions may point to a lack of trust in how the data are used (or could be used) within the education system and beyond.

Data protection is just one aspect of data governance though. One important question relates to the relationship between governments, the data subjects (who are usually the users of (sometimes mandatory) education services), and the private sector that usually develops the smart education technologies. This relates to questions of data ownership and of competition policy in the digital world: how should administrative and other education data be shared across education technology companies and public researchers to allow for progress and for a sufficient amount of competition in the sector of smart education technologies? How can the interests of learners and other individuals be best preserved in this context? The different solutions proposed for other sectors than education could probably be adapted to education (OECD, 2019[31]; 2019[1]).

Ethical discussions should normally concern what is not regulated within a country and thus aspects for which individuals and authorities have more freedom of action. This is how we interpret the question of “ethics” in educational AI.

The regulation of algorithms is usually not as strong as regulation for data protection. A major concern about algorithms is that they could be biased and have an undesirable social impact for some population groups (based on gender, ethnicity, socio-economic status, etc.) – but also that they could be flawed or just reinforce past human biases rather than reflect current societal values. A usual requirement is to ensure they are transparent and open, and that their “decisions” can be explained and challenged when automated. In the case of the EU General Data Protection Regulation, the regulatory language about algorithms is ambiguous (articles 13-15, 22, and recital 73) and lawyers are still debating what those articles imply in terms of “right of explanation” (transparency) and of the possibility to opt out of or challenge “automated decisions” for citizens. In the European Union, only the French and Hungarian laws have an explicit law about the “right of explanation” (with French law requiring both ex-ante and ex-post explanation in an intelligible way, and Hungarian law, some level of explanation) (Malgieri, 2019[32]). In the United States, there is no regulation about algorithms and their requirements as part of FERPA (or other regulation). Most OECD countries do not have clear regulatory requirements about them as of 2021.

Because algorithms based on machine learning are trained with historical data, many observers are concerned they will reproduce past biased (human) practices, as has apparently been the case in some countries in domains other than education (finance, justice, etc.) (O’Neil, 2016[33]). Several guidelines have been developed to avoid these pitfalls, which can happen at different steps of the process: measurement (data collection or labelling), model learning (when machine learning is involved), and action (when the algorithms detect, diagnose and act, for example). Different possible measures of fairness are also possible, which makes the issue even more complex (Kizilcec and Lee, 2020[34]; Baker and Hawn, 2021[35]). Ifenthaler (2021[13]) mentions several check lists of good practice and ethics for learning analytics. Bowers (2021[19]) points to the “open algorithm” movement in the area of early warning systems and notably to two overlapping sets of criteria to ensure transparency, verification and replicability of algorithms: the AAAA (accurate, accessible, actionable and accountable) and FAIR (findable, accessible, interoperable and reproducible) principles. Molenaar (2021[8]) also points to the importance of transparency to govern learning analytics and algorithms. As was shown in 2019 with some difficulties around exams and grade assignment for university admission, when it comes to high-stakes automated decisions, transparency is also about initiating an early dialogue about the criteria, expected social outcomes, relevance and acceptability of smart algorithms with diverse stakeholders from experts through to final users and other social bodies (Box 1.3). In some cases, the algorithms can be human-coded rather than involve AI techniques.

For example, given that accurate predictors for early warning systems can rely on minimal information that does not include information about gender, race and socio-economic status (Bowers, 2021[19]), it may not be considered ethical (or even necessary) to include these kinds of indicators to diagnose dropout in early warning systems unless they improve significantly the performance of the algorithms. On the other hand, results that do not include any personal information about learners may still lead to biased or socially/politically undesirable outcomes. Ethical concerns should therefore include a verification and discussion of the effects of smart technologies on different groups and ensure they are aligned with countries’ social and political principles. As few people are, in practice, able to verify the effects and impacts of algorithms, some independent groups of stakeholders may be responsible for or even assigned this task. While anyone should be allowed to do it in the frame of an open algorithm culture (at least when algorithms lead to a decision or a quasi-decision), education researchers, non-governmental organisations, but also, possibly, independent governmental agencies, could play an enhanced role in this area.

In the case of the most advanced applications of learning analytics based on a continuous monitoring of individuals (e.g. engagement, self-regulation, classroom orchestration, game-based assessments), another question is whether stakeholders feel comfortable with some aspects of the applications even if they are legal. While the tracking and data collection necessary to power learning analytics focusing on student engagement, self-regulation or classroom orchestration have to comply with domestic data protection regulations (and algorithm regulation, if any), the question is how to make them compatible with the political values of the country where they are implemented. This may require some imagination in terms of data protection arrangements (such as deleting immediately the data once processed). As in the exam case mentioned above, this also requires a social negotiation with all stakeholders including transparency about the data collection and how they are used. This is not just a matter of regulation or even perceived ethics. Even within the same country, what can be acceptable in some communities may not be in others depending on how the smart technologies were introduced (Box 1.5).

Another pragmatic and ethical issue relates to the use of the information generated by data analytics about teachers and other staff. Typically, staff within an organisation do not benefit of the same data protection and privacy regulation as students and other users. While smart technologies and learning analytics have the potential to provide feedback and support to teachers and other education stakeholders to make better decisions and improve their professional practices, they could also be used against them and unintentionally lead to undesirable social behaviours. Classroom analytics can be used to monitor teachers’ professional behaviour, and sometimes identify shortcoming in how they orchestrate learning in their classroom (Dillenbourg, 2021[12]). Should this information be used to sanction or support them? One possible intervention for students at risk of dropping out also consists of expelling or pushing them out of the school detecting this risk, which eventually enhances their likelihood to drop out eventually (Bowers, 2021[19]). Given their possible intrusive surveillance nature, the adoption of smart technologies relies on some level of trust in their positive, human-empowering ethical use. Should their voluntary use have adverse effects on teachers, school principals and decision-makers, they may appear as less acceptable and be resisted. Ethics about their use may require either full confidentiality about their results or a discussion and clear disclosure of how they can affect the staff using them. (This may also depend on their expected accuracy and effectiveness). As in the case of the information provided by longitudinal information systems, two different philosophies are possible. Some argue that their information should be used to reward and sanction stakeholders as an accountability mechanism, which is also a way to make them pay attention to the provided information. Others argue that the information should not be used to reward or sanction stakeholders as this may lead to their opposing the use of the information or just incentivise them to try to “game the system” – and thus lead to unethical behaviour. The jury is still out on what the best strategy is.

Building on the fact that smart technologies are socio-technical systems, their adoption and use will typically require some level of trust in teachers, schools, governments, and other stakeholders – and some attention paid to their possible adverse effects. Building trust should build both on regulation and the establishment of ethical practices, including different governance mechanisms:

• Regulation on data protection and privacy, as well as practical guidance on how to implement those provisions with different types of data and different levels of resources and competence;
• Regulation or good practice on data governance, and notably a strategy that makes data collected through smart technologies available to researchers and possibly to competitors to the vendors whose technology solution collects them;
• Regulation or guidelines about the transparency, openness and replicability of algorithms, as well as funding and support for the verification of the design and the final results of algorithms by independent parties;
• A risk-management approach to data protection and algorithm supervision that strikes an appropriate balance between risk-taking and data/usage sensitivity, acknowledging that regulations (or ethical guidance) that are too risk-averse will hinder the development of smart education technologies within a country or region and prevent reaping their possible benefits. The regulations above should be updated as smart technologies continue their rapid progress, and a close collaboration between researchers on data protection and on smart technologies may help co-design good solutions.

Infrastructure and public good

Smart technologies do usually require a strong Internet, computer and data infrastructure. Artificial intelligence systems, adaptive learning systems providing real-time information to teachers and learners, game-based assessments, blockchain, social robots, all require sufficient computer hardware in schools and universities (including students’ hardware under “bring your own device” policies), but also increasingly at home, as well as a bandwidth and networking capacity capable of stable and acceptable data transfer speeds. This also implies some level of investment in IT staff to deal with the maintenance of the hardware in (or across) educational establishments. Hardware is a necessary basis, but there is more to what a digital learning infrastructure should be.

Policy makers need to think about infrastructure also in terms of digital resources, both content and tools, that should be publicly provided to citizens, students, and educational institutions – either directly or indirectly through subsidies or funding to institutions. Budget constraints can limit these aspirations, but it remains important to define what should be the core of digital resources for all, and what should be accessible privately. A new question is the extent to which smart technologies should be part of the core digital education infrastructure. Is it enough to make some educational resources available to the public or should they also have the personalised learning features that some intelligent tutoring systems propose?

A final aspect of a digital infrastructure lies in people’s “digital skills”, that is, the ability to use digital resources as part of one’s professional practice. In the case of teachers and professors, digital skills are less about mastering the technology than about integrating technology tools, resources and outputs in their pedagogy. Unless fully automated, technology solutions are indeed mere tools for human teaching, learning or managing education systems. Professional learning opportunities for staff, both through training and organised continuous professional learning opportunities, are thus a final aspect that should be an integral part of a strong digital infrastructure.

While university researchers and researchers in public technology agencies may have the skills to develop some of those smart technologies, governments, schools and universities will typically rely on private education technology companies to provide and maintain them (hopefully through public-private partnerships, as mentioned above).

Beyond this investment in its digital infrastructure and connectivity for all, governments still have two important responsibilities in dealing with the private sector:

• Ensuring through its procurement policies and other incentives that publicly funded or purchased smart technologies are available to schools with affordable costs.
• Ensuring that some key techniques or discoveries of those smart education technologies become or remain a public, international good and allow more actors internationally to develop new interoperable solutions that will help improve education for all across the globe.
• Ensuring that staff have the learning opportunities to properly use the smart technologies and digital resources at their disposal.

Research and development

This book shows how smart technologies can be beneficial to education in various ways. At the same time, a common thread through the different chapters of the book is the insufficient amount of strong evidence about the effectiveness of the various uses of smart technologies. As noted above, there is a relative lack of evidence and transparency about the actual algorithmic accuracy of the technology solutions but more evidence on their effective use in real-life educational settings should also be produced. Social robots are one of the few areas where meta-analyses and some strong evidence results could be established about some specific use of technology to improve learning outcomes (Belpaeme and Fumihide, 2021[10]).

While education research and development is needed, new approaches may be warranted.

Policy makers and research funders should remain aware that technology is not to be tested in itself. As a mere tool that is part of a socio-technical system, the appropriate research question is rarely whether some specific technological application is “effective” but more about identifying and testing whether some specific uses yield positive outcomes. Usually, when it comes to instruction, the research question will be about pedagogy and how technology supports this pedagogy rather than about the technology itself. When it comes to administrative processes, the question may be about the interventions supported or induced by the smart technology rather than the technology itself.

Some specific technological applications of smart technology have been (and can be) researched through robust experimental research designs. However, as the development of smart technologies (and computing power) advances quickly, the evidence about their effectiveness may be quickly outdated and irrelevant as the technology gets upgraded.

As other digital technologies, smart technologies allow for new types of speedy research when they are done online: A/B research. Typically, this consists of trying two different designs (A and B) of a given technology with two different groups to identify quickly which one works best (when there are large numbers of use/exposure online). This approach can be extended to classrooms to evaluate the effectiveness in using different designs of some given smart technology or of different smart technologies pursuing similar objectives in a classroom or at the system level, factoring in the complex human behaviours that are attached to their use.

Should one wait for strong evidence before using technology in education? In practice, this may be too restrictive an approach. Requirements for strong evidence before the introduction of any innovation are often a way to protect the status quo of practices whose effectiveness is rarely proven. Disciplined innovation can happen in other ways too, notably through a proportionate use of effectiveness evidence. In the case of some smart technologies, usual evaluation processes may not be the most appropriate and sometimes it may be difficult to carry them out. It would, for example, be difficult to evaluate the effectiveness of early warning systems through a randomised control trial – one reason being that systems may have to be different from one site to the other to accommodate local circumstances. The speed of development of the technology will also typically mean that experimental results will either validate or invalidate technology solutions that are already outdated by improved versions of the solution (or new technologies altogether). Perhaps another approach would be to evaluate families of technology solutions used in certain ways.

The evidence standards required for different types of solutions should probably vary depending on the possible stakes and harm coming from the solution. Smart technologies supported by a good “theory of action” or good “underlying theory” may be acceptable for use even without a strong evidence base for low-stakes applications. When there is little underlying theory, no clear theory of action, no related basis of evidence, policy makers and stakeholders should be more reluctant to allow their use in public settings. Finally, in the case of high-stakes situations, the highest evidence standards should be in place, although algorithmic effectiveness should be compared to evidence of human effectiveness when performing similar tasks.

To further address this question, governments could:

• Invest in educational research about the use of technology in education in real life settings with a focus on pedagogy or administrative processes rather than the technology itself;
• Develop domestic and possibly international repositories of evidence on different types of uses of technology in education and different types of families of education technology;
• Develop common standards about good educational research and development in educational technology, acknowledging some of the specificities of the field.