Archive for the ‘big data’ Category

What is the ‘new normal’?

Among the many words and phrases that have been coined or gained new currency since COVID-19 first struck, I find ‘the new normal’ particularly interesting. In the educational world, its meaning is so obvious that it doesn’t need spelling out. But in case you’re unclear about what I’m referring to, the title of this webinar, run by GENTEFL, the Global Educators Network Association of Teachers of English as a Foreign Language (an affiliate of IATEFL), will give you a hint.

webinar GENTEFL

Teaching in a VLE may be overstating it a bit, but you get the picture. ‘The new normal’ is the shift away from face-to-face teaching in bricks-and-mortar institutions, towards online teaching of one kind or another. The Malaysian New Straits Times refers to it as ‘E-learning, new way forward in new norm’. The TEFL Academy says that ‘digital learning is the new normal’, and the New Indian Express prefers the term ‘tech education’.

Indian express

I’ll come back to these sources in a little while.

Whose new normal?

There is, indeed, a strong possibility that online learning and teaching may become ‘the new normal’ for many people working in education. In corporate training and in higher education, ‘tech education’ will likely become increasingly common. Many universities, especially but not only in the US, Britain and Australia, have been relying on ‘international students’ (almost half a million in the UK in 2018/ 2019), in particular Chinese, to fill their coffers. With uncertainty about how and when these universities will reopen for the next academic year, a successful transition to online is a matter of survival – a challenge that a number of universities will probably not be able to rise to. The core of ELT, private TEFL schools in Inner Circle countries, likewise dependent on visitors from other countries, has also been hard hit. It is not easy for them to transition to online, since the heart of their appeal lies in their physical location.

But elsewhere, the picture is rather different. A recent Reddit discussion began as follows: ‘In Vietnam, [English language] schools have reopened and things have returned to normal almost overnight. There’s actually a teacher shortage at the moment as so many left and interest in online learning is minimal, although most schools are still offering it as an option’. The consensus in the discussion that follows is that bricks-and-mortar schools will take a hit, especially with adult (but not kids’) groups, but that ‘teaching online will not be the new normal’.

By far the greatest number of students studying English around the world are in primary and secondary schools. It is highly unlikely that online study will be the ‘new normal’ for most of these students (although we may expect to see attempts to move towards more blended approaches). There are many reasons for this, but perhaps the most glaringly obvious is that the function of schools is not exclusively educational: child-care, allowing parents to go to work, is the first among these.

We can expect some exceptions. In New York, for example, current plans include a ‘hybrid model’ (a sexed-up term for blended learning), in which students are in schools for part of the time and continue learning remotely for the rest. The idea emerged after Governor Andrew Cuomo ‘convened a committee with the Bill and Melinda Gates Foundation to reimagine education for students when school goes back in session in the fall’. How exactly this will pan out remains to be seen, but, in much of the rest of the world, where the influence of the Gates Foundation is less strong, ‘hybrid schooling’ is likely to be seen as even more unpalatable and unworkable than it is by many in New York.

In short, the ‘new normal’ will affect some sectors of English language teaching much more than others. For some, perhaps the majority, little change can be expected once state schools reopen. Smaller classes, maybe, more blended, but not a wholesale shift to ‘tech education’.

Not so new anyway!

Scott Galloway, a New York professor of marketing and author of the best-selling ‘The Four’ (an analysis of the Big Four tech firms), began a recent blog post as follows:

After COVID-19, nothing will be the same. The previous sentence is bullsh*t. On the contrary, things will never be more the same, just accelerated.

He elaborates his point by pointing out that many universities were already in deep trouble before COVID. Big tech had already moved massively into education and healthcare, which are ‘the only two sectors, other than government, that offer the margin dollars required to sate investors’ growth expectations’ (from another recent post by Galloway). Education start-ups have long been attracting cheap capital: COVID has simply sped the process up.

Coming from a very different perspective, Audrey Watters gave a conference presentation over three years ago entitled ‘Education Technology as ‘The New Normal’’. I have been writing about the normalization of digital tools in language teaching for over six years. What is new is the speed, rather than the nature, of the change.

Galloway draws an interesting parallel with the SARS virus, which, he says, ‘was huge for e-commerce in Asia, and it helped Alibaba break out into the consumer space. COVID-19 could be to education in the United States what SARS was to e-commerce in Asia’.

‘The new normal’ as a marketing tool

Earlier in this post, I mentioned three articles that discussed the ‘new normal’ in education. The first of these, from the New Straits Times, looks like a news article, but features extensive quotes from Shereen Chee, chief operating officer of Sunago Education, a Malaysian vendor of online English classes. The article is basically an advert for Sunago: one section includes the following:

Sunago combines digitisation and the human touch to create a personalised learning experience. […] Chee said now is a great time for employers to take advantage of the scheme and equip their team with enhanced English skills, so they can hit the ground running once the Covid-19 slump is over.

The second reference about ‘digital learning is the new normal’ comes from The TEFL Academy, which sells online training courses, particularly targeting prospective teachers who want to work online. The third reference, from the New Indian Express, was written by Ananth Koppar, the founder of Kshema Technologies Pvt Ltd, India’s first venture-funded software company. Koppar is hardly a neutral reporter.

Other examples abound. For example, a similar piece called ‘The ‘New Normal’ in Education’ can be found in FE News (10 June 2020). This was written by Simon Carter, Marketing and Propositions Director of RM Education, an EdTech vendor in the UK. EdTech has a long history of promoting its wares through sponsored content and adverts masquerading as reportage.

It is, therefore, a good idea, whenever you come across the phrase, ‘the new normal’, to adopt a sceptical stance from the outset. I’ll give two more examples to illustrate my point.

A recent article (1 April 2020) in the ELTABB (English Language Teachers Association Berlin Brandenburg) journal is introduced as follows:

With online language teaching being the new normal in ELT, coaching principles can help teachers and students share responsibility for the learning process.

Putting aside, for the moment, my reservations about whether online teaching is, in fact, the new normal in ‘ELT’, I’m happy to accept that coaching principles may be helpful in online teaching. But I can’t help noticing that the article was written by a self-described edupreneur and co-founder of the International Language Coaching Association (€50 annual subscription) which runs three-day training courses (€400).

My second example is a Macmillan webinar by Thom Kiddle called ‘Professional Development for teachers in the ‘new normal’. It’s a good webinar, a very good one in my opinion, but you’ll notice a NILE poster tacked to the wall behind Thom as he speaks. NILE, a highly reputed provider of teacher education courses in the UK, has invested significantly in online teacher education in recent years and is well-positioned to deal with the ‘new normal’. It’s also worth noting that the webinar host, Macmillan, is in a commercial partnership with NILE, the purpose of which is to ‘develop and promote quality teacher education programmes worldwide’. As good as the webinar is, it is also clearly, in part, an advertisement.

Thom Kiddle

The use of the phrase ‘the new normal’ as a marketing hook is not new. Although its first recorded use dates back to the first part of the 20th century, it became more widespread at the start of the 21st. One populariser of the phrase was Roger McNamee, a venture capitalist and early investor in technology, including Facebook, who wrote a book called ‘The New Normal: Great Opportunities in a Time of Great Risk’ (2004). Since then, the phrase has been used extensively to refer to the state of the business world after the financial crisis of 2018. (For more about the history of the phrase, see here.) More often than not, users of the phrase are selling the idea (and sometimes a product) that we need to get used to a new configuration of the world, one in which technology plays a greater role.

Normalizing ‘the new normal’

Of all the most unlikely sources for a critique of ‘the new normal’, the World Economic Forum has the following to offer in a blog post entitled ‘There’s nothing new about the ‘new normal’. Here’s why’:

The language of a ‘new normal’ is being deployed almost as a way to quell any uncertainty ushered in by the coronavirus. With no cure in sight, everyone from politicians and the media to friends and family has perpetuated this rhetoric as they imagine settling into life under this ‘new normal’. This framing is inviting: it contends that things will never be the same as they were before — so welcome to a new world order. By using this language, we reimagine where we were previously relative to where we are now, appropriating our present as the standard. As we weigh our personal and political responses to this pandemic, the language we employ matters. It helps to shape and reinforce our understanding of the world and the ways in which we choose to approach it. The analytic frame embodied by the persistent discussion of the ‘new normal’ helps bring order to our current turbulence, but it should not be the lens through which we examine today’s crisis.

We can’t expect the World Economic Forum to become too critical of the ‘new normal’ of digital learning, since they have been pushing for it so hard for so long. But the quote from their blog above may usefully be read in conjunction with an article by Jun Yu and Nick Couldry, called ‘Education as a domain of natural data extraction: analysing corporate discourse about educational tracking’ (Information, Communication and Society, 2020, DOI: 10.1080/1369118X.2020.1764604). The article explores the general discursive framing by which the use of big data in education has come to seem normal. The authors looked at the public discourse of eight major vendors of educational platforms that use big data (including Macmillan, Pearson, Knewton and Blackboard). They found that ‘the most fundamental move in today’s dominant commercial discourse is to promote the idea that data and its growth are natural’. In this way, ‘software systems, not teachers, [are] central to education’. Yu and Couldry’s main interest is in the way that discourse shapes the normalization of dataveillance, but, in a more general sense, the phrase, ‘the new normal’, is contributing to the normalization of digital education. If you think that’s fine, I suggest you dip into some of the books I listed in my last blog post.

In my last post , I asked why it is so easy to believe that technology (in particular, technological innovations) will offer solutions to whatever problems exist in language learning and teaching. A simple, but inadequate, answer is that huge amounts of money have been invested in persuading us. Without wanting to detract from the significance of this, it is clearly not sufficient as an explanation. In an attempt to develop my own understanding, I have been turning more and more to the idea of ‘social imaginaries’. In many ways, this is also an attempt to draw together the various interests that I have had since starting this blog.

The Canadian philosopher, Charles Taylor, describes a ‘social imaginary’ as a ‘common understanding that makes possible common practices and a widely shared sense of legitimacy’ (Taylor, 2004: 23). As a social imaginary develops over time, it ‘begins to define the contours of [people’s] worlds and can eventually come to count as the taken-for-granted shape of things, too obvious to mention’ (Taylor, 2004: 29). It is, however, not just a set of ideas or a shared narrative: it is also a set of social practices that enact those understandings, whilst at the same time modifying or solidifying them. The understandings make the practices possible, and it is the practices that largely carry the understanding (Taylor, 2004: 25). In the process, the language we use is filled with new associations and our familiarity with these associations shapes ‘our perceptions and expectations’ (Worster, 1994, quoted in Moore, 2015: 33). A social imaginary, then, is a complex system that is not technological or economic or social or political or educational, but all of these (Urry, 2016). The image of the patterns of an amorphous mass of moving magma (Castoriadis, 1987), flowing through pre-existing channels, but also, at times, striking out along new paths, may offer a helpful metaphor.

Lava flow Hawaii

Technology, of course, plays a key role in contemporary social imaginaries and the term ‘sociotechnical imaginary’ is increasingly widely used. The understandings of the sociotechnical imaginary typically express visions of social progress and a desirable future that is made possible by advances in science and technology (Jasanoff & Kim, 2015: 4). In education, technology is presented as capable of overcoming human failings and the dark ways of the past, of facilitating a ‘pedagogical utopia of natural, authentic teaching and learning’ (Friesen, forthcoming). As such understandings become more widespread and as the educational practices (platforms, apps, etc.) which both shape and are shaped by them become equally widespread, technology has come to be seen as a ‘solution’ to the ‘problem’ of education (Friesen, forthcoming). We need to be careful, however, that having shaped the technology, it does not comes to shape us (see Cobo, 2019, for a further exploration of this idea).

As a way of beginning to try to understand what is going on in edtech in ELT, which is not so very different from what is taking place in education more generally, I have sketched a number of what I consider key components of the shared understandings and the social practices that are related to them. These are closely interlocking pieces and each of them is itself embedded in much broader understandings. They evolve over time and their history can be traced quite easily. Taken together, they do, I think, help us to understand a little more why technology in ELT seems so seductive.

1 The main purpose of English language teaching is to prepare people for the workplace

There has always been a strong connection between learning an additional living language (such as English) and preparing for the world of work. The first modern language schools, such as the Berlitz schools at the end of the 19th century with their native-speaker teachers and monolingual methods, positioned themselves as primarily vocational, in opposition to the kinds of language teaching taking place in schools and universities, which were more broadly humanistic in their objectives. Throughout the 20th century, and especially as English grew as a global language, the public sector, internationally, grew closer to the methods and objectives of the private schools. The idea that learning English might serve other purposes (e.g. cultural enrichment or personal development) has never entirely gone away, as witnessed by the Council of Europe’s list of objectives (including the promotion of mutual understanding and European co-operation, and the overcoming of prejudice and discrimination) in the Common European Framework, but it is often forgotten.

The clarion calls from industry to better align education with labour markets, present and future, grow louder all the time, often finding expression in claims that ‘education is unfit for purpose.’ It is invariably assumed that this purpose is to train students in the appropriate skills to enhance their ‘human capital’ in an increasingly competitive and global market (Lingard & Gale, 2007). Educational agendas are increasingly set by the world of business (bodies like the OECD or the World Economic Forum, corporations like Google or Microsoft, and national governments which share their priorities (see my earlier post about neo-liberalism and solutionism ).

One way in which this shift is reflected in English language teaching is in the growing emphasis that is placed on ‘21st century skills’ in teaching material. Sometimes called ‘life skills’, they are very clearly concerned with the world of work, rather than the rest of our lives. The World Economic Forum’s 2018 Future of Jobs survey lists the soft skills that are considered important in the near future and they include ‘creativity’, ‘critical thinking’, ‘emotional intelligence’ and ‘leadership’. (The fact that the World Economic Forum is made up of a group of huge international corporations (e.g. J.P. Morgan, HSBC, UBS, Johnson & Johnson) with a very dubious track record of embezzlement, fraud, money-laundering and tax evasion has not resulted in much serious, public questioning of the view of education expounded by the WEF.)

Without exception, the ELT publishers have brought these work / life skills into their courses, and the topic is an extremely popular one in ELT blogs and magazines, and at conferences. Two of the four plenaries at this year’s international IATEFL conference are concerned with these skills. Pearson has a wide range of related products, including ‘a four-level competency-based digital course that provides engaging instruction in the essential work and life skills competencies that adult learners need’. Macmillan ELT made ‘life skills’ the central plank of their marketing campaign and approach to product design, and even won a British Council ELTon (see below) Award for ‘Innovation in teacher resources) in 2015 for their ‘life skills’ marketing campaign. Cambridge University Press has developed a ‘Framework for Life Competencies’ which allows these skills to be assigned numerical values.

The point I am making here is not that these skills do not play an important role in contemporary society, nor that English language learners may not benefit from some training in them. The point, rather, is that the assumption that English language learning is mostly concerned with preparation for the workplace has become so widespread that it becomes difficult to think in another way.

2 Technological innovation is good and necessary

The main reason that soft skills are deemed to be so important is that we live in a rapidly-changing world, where the unsubstantiated claim that 85% (or whatever other figure comes to mind) of current jobs won’t exist 10 years from now is so often repeated that it is taken as fact . Whether or not this is true is perhaps less important to those who make the claim than the present and the future that they like to envisage. The claim is, at least, true-ish enough to resonate widely. Since these jobs will disappear, and new ones will emerge, because of technological innovations, education, too, will need to innovate to keep up.

English language teaching has not been slow to celebrate innovation. There were coursebooks called ‘Cutting Edge’ (1998) and ‘Innovations’ (2005), but more recently the connections between innovation and technology have become much stronger. The title of the recent ‘Language Hub’ (2019) was presumably chosen, in part, to conjure up images of digital whizzkids in fashionable co-working start-up spaces. Technological innovation is explicitly promoted in the Special Interest Groups of IATEFL and TESOL. Despite a singular lack of research that unequivocally demonstrates a positive connection between technology and language learning, the former’s objective is ‘to raise awareness among ELT professionals of the power of learning technologies to assist with language learning’. There is a popular annual conference, called InnovateELT , which has the tagline ‘Be Part of the Solution’, and the first problem that this may be a solution to is that our students need to be ‘ready to take on challenging new careers’.

Last, but by no means least, there are the annual British Council ELTon awards  with a special prize for digital innovation. Among the British Council’s own recent innovations are a range of digitally-delivered resources to develop work / life skills among teens.

Again, my intention (here) is not to criticise any of the things mentioned in the preceding paragraphs. It is merely to point to a particular structure of feeling and the way that is enacted and strengthened through material practices like books, social groups, conferences and other events.

3 Technological innovations are best driven by the private sector

The vast majority of people teaching English language around the world work in state-run primary and secondary schools. They are typically not native-speakers of English, they hold national teaching qualifications and they are frequently qualified to teach other subjects in addition to English (often another language). They may or may not self-identify as teachers of ‘ELT’ or ‘EFL’, often seeing themselves more as ‘school teachers’ or ‘language teachers’. People who self-identify as part of the world of ‘ELT or ‘TEFL’ are more likely to be native speakers and to work in the private sector (including private or semi-private language schools, universities (which, in English-speaking countries, are often indistinguishable from private sector institutions), publishing companies, and freelancers). They are more likely to hold international (TEFL) qualifications or higher degrees, and they are less likely to be involved in the teaching of other languages.

The relationship between these two groups is well illustrated by the practice of training days, where groups of a few hundred state-school teachers participate in workshops organised by publishing companies and delivered by ELT specialists. In this context, state-school teachers are essentially in a client role when they are in contact with the world of ‘ELT’ – as buyers or potential buyers of educational products, training or technology.

Technological innovation is invariably driven by the private sector. This may be in the development of technologies (platforms, apps and so on), in the promotion of technology (through training days and conference sponsorship, for example), or in training for technology (with consultancy companies like ELTjam or The Consultants-E, which offer a wide range of technologically oriented ‘solutions’).

As in education more generally, it is believed that the private sector can be more agile and more efficient than state-run bodies, which continue to decline in importance in educational policy-setting. When state-run bodies are involved in technological innovation in education, it is normal for them to work in partnership with the private sector.

4 Accountability is crucial

Efficacy is vital. It makes no sense to innovate unless the innovations improve something, but for us to know this, we need a way to measure it. In a previous post , I looked at Pearson’s ‘Asking More: the Path to Efficacy’ by CEO John Fallon (who will be stepping down later this year). Efficacy in education, says Fallon, is ‘making a measurable impact on someone’s life through learning’. ‘Measurable’ is the key word, because, as Fallon claims, ‘it is increasingly possible to determine what works and what doesn’t in education, just as in healthcare.’ We need ‘a relentless focus’ on ‘the learning outcomes we deliver’ because it is these outcomes that can be measured in ‘a systematic, evidence-based fashion’. Measurement, of course, is all the easier when education is delivered online, ‘real-time learner data’ can be captured, and the power of analytics can be deployed.

Data is evidence, and it’s as easy to agree on the importance of evidence as it is hard to decide on (1) what it is evidence of, and (2) what kind of data is most valuable. While those questions remain largely unanswered, the data-capturing imperative invades more and more domains of the educational world.

English language teaching is becoming data-obsessed. From language scales, like Pearson’s Global Scale of English to scales of teacher competences, from numerically-oriented formative assessment practices (such as those used on many LMSs) to the reporting of effect sizes in meta-analyses (such as those used by John Hattie and colleagues), datafication in ELT accelerates non-stop.

The scales and frameworks are all problematic in a number of ways (see, for example, this post on ‘The Mismeasure of Language’) but they have undeniably shaped the way that we are able to think. Of course, we need measurable outcomes! If, for the present, there are privacy and security issues, it is to be hoped that technology will find solutions to them, too.

REFERENCES

Castoriadis, C. (1987). The Imaginary Institution of Society. Cambridge: Polity Press.

Cobo, C. (2019). I Accept the Terms and Conditions. Montevideo: International Development Research Centre / Center for Research Ceibal Foundation. https://adaptivelearninginelt.files.wordpress.com/2020/01/41acf-cd84b5_7a6e74f4592c460b8f34d1f69f2d5068.pdf

Friesen, N. (forthcoming) The technological imaginary in education, or: Myth and enlightenment in ‘Personalized Learning’. In M. Stocchetti (Ed.) The Digital Age and its Discontents. University of Helsinki Press. Available at https://www.academia.edu/37960891/The_Technological_Imaginary_in_Education_or_Myth_and_Enlightenment_in_Personalized_Learning_

Jasanoff, S. & Kim, S.-H. (2015). Dreamscapes of Modernity. Chicago: University of Chicago Press.

Lingard, B. & Gale, T. (2007). The emergent structure of feeling: what does it mean for critical educational studies and research?, Critical Studies in Education, 48:1, pp. 1-23

Moore, J. W. (2015). Capitalism in the Web of Life. London: Verso.

Robbins, K. & Webster, F. (1989]. The Technical Fix. Basingstoke: Macmillan Education.

Taylor, C. (2014). Modern Social Imaginaries. Durham, NC: Duke University Press.

Urry, J. (2016). What is the Future? Cambridge: Polity Press.

 

At the start of the last decade, ELT publishers were worried, Macmillan among them. The financial crash of 2008 led to serious difficulties, not least in their key Spanish market. In 2011, Macmillan’s parent company was fined ₤11.3 million for corruption. Under new ownership, restructuring was a constant. At the same time, Macmillan ELT was getting ready to move from its Oxford headquarters to new premises in London, a move which would inevitably lead to the loss of a sizable proportion of its staff. On top of that, Macmillan, like the other ELT publishers, was aware that changes in the digital landscape (the first 3G iPhone had appeared in June 2008 and wifi access was spreading rapidly around the world) meant that they needed to shift away from the old print-based model. With her finger on the pulse, Caroline Moore, wrote an article in October 2010 entitled ‘No Future? The English Language Teaching Coursebook in the Digital Age’ . The publication (at the start of the decade) and runaway success of the online ‘Touchstone’ course, from arch-rivals, Cambridge University Press, meant that Macmillan needed to change fast if they were to avoid being left behind.

Macmillan already had a platform, Campus, but it was generally recognised as being clunky and outdated, and something new was needed. In the summer of 2012, Macmillan brought in two new executives – people who could talk the ‘creative-disruption’ talk and who believed in the power of big data to shake up English language teaching and publishing. At the time, the idea of big data was beginning to reach public consciousness and ‘Big Data: A Revolution that Will Transform how We Live, Work, and Think’ by Viktor Mayer-Schönberger and Kenneth Cukier, was a major bestseller in 2013 and 2014. ‘Big data’ was the ‘hottest trend’ in technology and peaked in Google Trends in October 2014. See the graph below.

Big_data_Google_Trend

Not long after taking up their positions, the two executives began negotiations with Knewton, an American adaptive learning company. Knewton’s technology promised to gather colossal amounts of data on students using Knewton-enabled platforms. Its founder, Jose Ferreira, bragged that Knewton had ‘more data about our students than any company has about anybody else about anything […] We literally know everything about what you know and how you learn best, everything’. This data would, it was claimed, enable publishers to multiply, by orders of magnitude, the efficacy of learning materials, allowing publishers, like Macmillan, to provide a truly personalized and optimal offering to learners using their platform.

The contract between Macmillan and Knewton was agreed in May 2013 ‘to build next-generation English Language Learning and Teaching materials’. Perhaps fearful of being left behind in what was seen to be a winner-takes-all market (Pearson already had a financial stake in Knewton), Cambridge University Press duly followed suit, signing a contract with Knewton in September of the same year, in order ‘to create personalized learning experiences in [their] industry-leading ELT digital products’. Things moved fast because, by the start of 2014 when Macmillan’s new catalogue appeared, customers were told to ‘watch out for the ‘Big Tree’’, Macmillans’ new platform, which would be powered by Knewton. ‘The power that will come from this world of adaptive learning takes my breath away’, wrote the international marketing director.

Not a lot happened next, at least outwardly. In the following year, 2015, the Macmillan catalogue again told customers to ‘look out for the Big Tree’ which would offer ‘flexible blended learning models’ which could ‘give teachers much more freedom to choose what they want to do in the class and what they want the students to do online outside of the classroom’.

Macmillan_catalogue_2015

But behind the scenes, everything was going wrong. It had become clear that a linear model of language learning, which was a necessary prerequisite of the Knewton system, simply did not lend itself to anything which would be vaguely marketable in established markets. Skills development, not least the development of so-called 21st century skills, which Macmillan was pushing at the time, would not be facilitated by collecting huge amounts of data and algorithms offering personalized pathways. Even if it could, teachers weren’t ready for it, and the projections for platform adoptions were beginning to seem very over-optimistic. Costs were spiralling. Pushed to meet unrealistic deadlines for a product that was totally ill-conceived in the first place, in-house staff were suffering, and this was made worse by what many staffers thought was a toxic work environment. By the end of 2014 (so, before the copy for the 2015 catalogue had been written), the two executives had gone.

For some time previously, skeptics had been joking that Macmillan had been barking up the wrong tree, and by the time that the 2016 catalogue came out, the ‘Big Tree’ had disappeared without trace. The problem was that so much time and money had been thrown at this particular tree that not enough had been left to develop new course materials (for adults). The whole thing had been a huge cock-up of an extraordinary kind.

Cambridge, too, lost interest in their Knewton connection, but were fortunate (or wise) not to have invested so much energy in it. Language learning was only ever a small part of Knewton’s portfolio, and the company had raised over $180 million in venture capital. Its founder, Jose Ferreira, had been a master of marketing hype, but the business model was not delivering any better than the educational side of things. Pearson pulled out. In December 2016, Ferreira stepped down and was replaced as CEO. The company shifted to ‘selling digital courseware directly to higher-ed institutions and students’ but this could not stop the decline. In September of 2019, Knewton was sold for something under $17 million dollars, with investors taking a hit of over $160 million. My heart bleeds.

It was clear, from very early on (see, for example, my posts from 2014 here and here) that Knewton’s product was little more than what Michael Feldstein called ‘snake oil’. Why and how could so many people fall for it for so long? Why and how will so many people fall for it again in the coming decade, although this time it won’t be ‘big data’ that does the seduction, but AI (which kind of boils down to the same thing)? The former Macmillan executives are still at the game, albeit in new companies and talking a slightly modified talk, and Jose Ferreira (whose new venture has already raised $3.7 million) is promising to revolutionize education with a new start-up which ‘will harness the power of technology to improve both access and quality of education’ (thanks to Audrey Watters for the tip). Investors may be desperate to find places to spread their portfolio, but why do the rest of us lap up the hype? It’s a question to which I will return.

 

 

 

 

The use of big data and analytics in education continues to grow.

A vast apparatus of measurement is being developed to underpin national education systems, institutions and the actions of the individuals who occupy them. […] The presence of digital data and software in education is being amplified through massive financial and political investment in educational technologies, as well as huge growth in data collection and analysis in policymaking practices, extension of performance measurement technologies in the management of educational institutions, and rapid expansion of digital methodologies in educational research. To a significant extent, many of the ways in which classrooms function, educational policy departments and leaders make decisions, and researchers make sense of data, simply would not happen as currently intended without the presence of software code and the digital data processing programs it enacts. (Williamson, 2017: 4)

The most common and successful use of this technology so far has been in the identification of students at risk of dropping out of their courses (Jørno & Gynther, 2018: 204). The kind of analytics used in this context may be called ‘academic analytics’ and focuses on educational processes at the institutional level or higher (Gelan et al, 2018: 3). However, ‘learning analytics’, the capture and analysis of learner and learning data in order to personalize learning ‘(1) through real-time feedback on online courses and e-textbooks that can ‘learn’ from how they are used and ‘talk back’ to the teacher, and (2) individualization and personalization of the educational experience through adaptive learning systems that enable materials to be tailored to each student’s individual needs through automated real-time analysis’ (Mayer-Schönberger & Cukier, 2014) has become ‘the main keyword of data-driven education’ (Williamson, 2017: 10). See my earlier posts on this topic here and here and here.

Learning with big dataNear the start of Mayer-Schönberger and Cukier’s enthusiastic sales pitch (Learning with Big Data: The Future of Education) for the use of big data in education, there is a discussion of Duolingo. They quote Luis von Ahn, the founder of Duolingo, as saying ‘there has been little empirical work on what is the best way to teach a foreign language’. This is so far from the truth as to be laughable. Von Ahn’s comment, along with the Duolingo product itself, is merely indicative of a lack of awareness of the enormous amount of research that has been carried out. But what could the data gleaned from the interactions of millions of users with Duolingo tell us of value? The example that is given is the following. Apparently, ‘in the case of Spanish speakers learning English, it’s common to teach pronouns early on: words like “he,” “she,” and “it”.’ But, Duolingo discovered, ‘the term “it” tends to confuse and create anxiety for Spanish speakers, since the word doesn’t easily translate into their language […] Delaying the introduction of “it” until a few weeks later dramatically improves the number of people who stick with learning English rather than drop out.’ Was von Ahn unaware of the decades of research into language transfer effects? Did von Ahn (who grew up speaking Spanish in Guatemala) need all this data to tell him that English personal pronouns can cause problems for Spanish learners of English? Was von Ahn unaware of the debates concerning the value of teaching isolated words (especially grammar words!)?

The area where little empirical research has been done is not in different ways of learning another language: it is in the use of big data and learning analytics to assist language learning. Claims about the value of these technologies in language learning are almost always speculative – they are based on comparison to other school subjects (especially, mathematics). Gelan et al (2018: 2), who note this lack of research, suggest that ‘understanding language learner behaviour could provide valuable insights into task design for instructors and materials designers, as well as help students with effective learning strategies and personalised learning pathways’ (my italics). Reinders (2018: 81) writes ‘that analysis of prior experiences with certain groups or certain courses may help to identify key moments at which students need to receive more or different support. Analysis of student engagement and performance throughout a course may help with early identification of learning problems and may prompt early intervention’ (italics added). But there is some research out there, and it’s worth having a look at. Most studies that have collected learner-tracking data concern glossary use for reading comprehension and vocabulary retention (Gelan et al, 2018: 5), but a few have attempted to go further in scope.

Volk et al (2015) looked at the behaviour of the 20,000 students per day using the platform which accompanies ‘More!’ (Gerngross et al. 2008) to do their English homework for Austrian lower secondary schools. They discovered that

  • the exercises used least frequently were those that are located further back in the course book
  • usage is highest from Monday to Wednesday, declining from Thursday, with a rise again on Sunday
  • most interaction took place between 3:00 and 5:00 pm.
  • repetition of exercises led to a strong improvement in success rate
  • students performed better on multiple choice and matching exercises than they did where they had to produce some language

The authors of this paper conclude by saying that ‘the results of this study suggest a number of new avenues for research. In general, the authors plan to extend their analysis of exercise results and applied exercises to the population of all schools using the online learning platform more-online.at. This step enables a deeper insight into student’s learning behaviour and allows making more generalizing statements.’ When I shared these research findings with the Austrian lower secondary teachers that I work with, their reaction was one of utter disbelief. People get paid to do this research? Why not just ask us?

More useful, more actionable insights may yet come from other sources. For example, Gu Yueguo, Pro-Vice-Chancellor of the Beijing Foreign Studies University has announced the intention to set up a national Big Data research center, specializing in big data-related research topics in foreign language education (Yu, 2015). Meanwhile, I’m aware of only one big research project that has published its results. The EC Erasmus+ VITAL project (Visualisation Tools and Analytics to monitor Online Language Learning & Teaching) was carried out between 2015 and 2017 and looked at the learning trails of students from universities in Belgium, Britain and the Netherlands. It was discovered (Gelan et al, 2015) that:

  • students who did online exercises when they were supposed to do them were slightly more successful than those who were late carrying out the tasks
  • successful students logged on more often, spent more time online, attempted and completed more tasks, revisited both exercises and theory pages more frequently, did the work in the order in which it was supposed to be done and did more work in the holidays
  • most students preferred to go straight into the assessed exercises and only used the theory pages when they felt they needed to; successful students referred back to the theory pages more often than unsuccessful students
  • students made little use of the voice recording functionality
  • most online activity took place the day before a class and the day of the class itself

EU funding for this VITAL project amounted to 274,840 Euros[1]. The technology for capturing the data has been around for a long time. In my opinion, nothing of value, or at least nothing new, has been learnt. Publishers like Pearson and Cambridge University Press who have large numbers of learners using their platforms have been capturing learning data for many years. They do not publish their findings and, intriguingly, do not even claim that they have learnt anything useful / actionable from the data they have collected. Sure, an exercise here or there may need to be amended. Both teachers and students may need more support in using the more open-ended functionalities of the platforms (e.g. discussion forums). But are they getting ‘unprecedented insights into what works and what doesn’t’ (Mayer-Schönberger & Cukier, 2014)? Are they any closer to building better pedagogies? On the basis of what we know so far, you wouldn’t want to bet on it.

It may be the case that all the learning / learner data that is captured could be used in some way that has nothing to do with language learning. Show me a language-learning app developer who does not dream of monetizing the ‘behavioural surplus’ (Zuboff, 2018) that they collect! But, for the data and analytics to be of any value in guiding language learning, it must lead to actionable insights. Unfortunately, as Jørno & Gynther (2018: 198) point out, there is very little clarity about what is meant by ‘actionable insights’. There is a danger that data and analytics ‘simply gravitates towards insights that confirm longstanding good practice and insights, such as “students tend to ignore optional learning activities … [and] focus on activities that are assessed” (Jørno & Gynther, 2018: 211). While this is happening, the focus on data inevitably shapes the way we look at the object of study (i.e. language learning), ‘thereby systematically excluding other perspectives’ (Mau, 2019: 15; see also Beer, 2019). The belief that tech is always the solution, that all we need is more data and better analytics, remains very powerful: it’s called techno-chauvinism (Broussard, 2018: 7-8).

References

Beer, D. 2019. The Data Gaze. London: Sage

Broussard, M. 2018. Artificial Unintelligence. Cambridge, Mass.: MIT Press

Gelan, A., Fastre, G., Verjans, M., Martin, N., Jansenswillen, G., Creemers, M., Lieben, J., Depaire, B. & Thomas, M. 2018. ‘Affordances and limitations of learning analytics for computer­assisted language learning: a case study of the VITAL project’. Computer Assisted Language Learning. pp. 1­26. http://clok.uclan.ac.uk/21289/

Gerngross, G., Puchta, H., Holzmann, C., Stranks, J., Lewis-Jones, P. & Finnie, R. 2008. More! 1 Cyber Homework. Innsbruck, Austria: Helbling

Jørno, R. L. & Gynther, K. 2018. ‘What Constitutes an “Actionable Insight” in Learning Analytics?’ Journal of Learning Analytics 5 (3): 198 – 221

Mau, S. 2019. The Metric Society. Cambridge: Polity Press

Mayer-Schönberger, V. & Cukier, K. 2014. Learning with Big Data: The Future of Education. New York: Houghton Mifflin Harcourt

Reinders, H. 2018. ‘Learning analytics for language learning and teaching’. JALT CALL Journal 14 / 1: 77 – 86 https://files.eric.ed.gov/fulltext/EJ1177327.pdf

Volk, H., Kellner, K. & Wohlhart, D. 2015. ‘Learning Analytics for English Language Teaching.’ Journal of Universal Computer Science, Vol. 21 / 1: 156-174 http://www.jucs.org/jucs_21_1/learning_analytics_for_english/jucs_21_01_0156_0174_volk.pdf

Williamson, B. 2017. Big Data in Education. London: Sage

Yu, Q. 2015. ‘Learning Analytics: The next frontier for computer assisted language learning in big data age’ SHS Web of Conferences, 17 https://www.shs-conferences.org/articles/shsconf/pdf/2015/04/shsconf_icmetm2015_02013.pdf

Zuboff, S. 2019. The Age of Surveillance Capitalism. London: Profile Books

 

[1] See https://ec.europa.eu/programmes/erasmus-plus/sites/erasmusplus2/files/ka2-2015-he_en.pdf

About two and a half years ago when I started writing this blog, there was a lot of hype around adaptive learning and the big data which might drive it. Two and a half years are a long time in technology. A look at Google Trends suggests that interest in adaptive learning has been pretty static for the last couple of years. It’s interesting to note that 3 of the 7 lettered points on this graph are Knewton-related media events (including the most recent, A, which is Knewton’s latest deal with Hachette) and 2 of them concern McGraw-Hill. It would be interesting to know whether these companies follow both parts of Simon Cowell’s dictum of ‘Create the hype, but don’t ever believe it’.

Google_trends

A look at the Hype Cycle (see here for Wikipedia’s entry on the topic and for criticism of the hype of Hype Cycles) of the IT research and advisory firm, Gartner, indicates that both big data and adaptive learning have now slid into the ‘trough of disillusionment’, which means that the market has started to mature, becoming more realistic about how useful the technologies can be for organizations.

A few years ago, the Gates Foundation, one of the leading cheerleaders and financial promoters of adaptive learning, launched its Adaptive Learning Market Acceleration Program (ALMAP) to ‘advance evidence-based understanding of how adaptive learning technologies could improve opportunities for low-income adults to learn and to complete postsecondary credentials’. It’s striking that the program’s aims referred to how such technologies could lead to learning gains, not whether they would. Now, though, with the publication of a report commissioned by the Gates Foundation to analyze the data coming out of the ALMAP Program, things are looking less rosy. The report is inconclusive. There is no firm evidence that adaptive learning systems are leading to better course grades or course completion. ‘The ultimate goal – better student outcomes at lower cost – remains elusive’, the report concludes. Rahim Rajan, a senior program office for Gates, is clear: ‘There is no magical silver bullet here.’

The same conclusion is being reached elsewhere. A report for the National Education Policy Center (in Boulder, Colorado) concludes: Personalized Instruction, in all its many forms, does not seem to be the transformational technology that is needed, however. After more than 30 years, Personalized Instruction is still producing incremental change. The outcomes of large-scale studies and meta-analyses, to the extent they tell us anything useful at all, show mixed results ranging from modest impacts to no impact. Additionally, one must remember that the modest impacts we see in these meta-analyses are coming from blended instruction, which raises the cost of education rather than reducing it (Enyedy, 2014: 15 -see reference at the foot of this post). In the same vein, a recent academic study by Meg Coffin Murray and Jorge Pérez (2015, ‘Informing and Performing: A Study Comparing Adaptive Learning to Traditional Learning’) found that ‘adaptive learning systems have negligible impact on learning outcomes’.

future-ready-learning-reimagining-the-role-of-technology-in-education-1-638In the latest educational technology plan from the U.S. Department of Education (‘Future Ready Learning: Reimagining the Role of Technology in Education’, 2016) the only mentions of the word ‘adaptive’ are in the context of testing. And the latest OECD report on ‘Students, Computers and Learning: Making the Connection’ (2015), finds, more generally, that information and communication technologies, when they are used in the classroom, have, at best, a mixed impact on student performance.

There is, however, too much money at stake for the earlier hype to disappear completely. Sponsored cheerleading for adaptive systems continues to find its way into blogs and national magazines and newspapers. EdSurge, for example, recently published a report called ‘Decoding Adaptive’ (2016), sponsored by Pearson, that continues to wave the flag. Enthusiastic anecdotes take the place of evidence, but, for all that, it’s a useful read.

In the world of ELT, there are plenty of sales people who want new products which they can call ‘adaptive’ (and gamified, too, please). But it’s striking that three years after I started following the hype, such products are rather thin on the ground. Pearson was the first of the big names in ELT to do a deal with Knewton, and invested heavily in the company. Their relationship remains close. But, to the best of my knowledge, the only truly adaptive ELT product that Pearson offers is the PTE test.

Macmillan signed a contract with Knewton in May 2013 ‘to provide personalized grammar and vocabulary lessons, exam reviews, and supplementary materials for each student’. In December of that year, they talked up their new ‘big tree online learning platform’: ‘Look out for the Big Tree logo over the coming year for more information as to how we are using our partnership with Knewton to move forward in the Language Learning division and create content that is tailored to students’ needs and reactive to their progress.’ I’ve been looking out, but it’s all gone rather quiet on the adaptive / platform front.

In September 2013, it was the turn of Cambridge to sign a deal with Knewton ‘to create personalized learning experiences in its industry-leading ELT digital products for students worldwide’. This year saw the launch of a major new CUP series, ‘Empower’. It has an online workbook with personalized extra practice, but there’s nothing (yet) that anyone would call adaptive. More recently, Cambridge has launched the online version of the 2nd edition of Touchstone. Nothing adaptive there, either.

Earlier this year, Cambridge published The Cambridge Guide to Blended Learning for Language Teaching, edited by Mike McCarthy. It contains a chapter by M.O.Z. San Pedro and R. Baker on ‘Adaptive Learning’. It’s an enthusiastic account of the potential of adaptive learning, but it doesn’t contain a single reference to language learning or ELT!

So, what’s going on? Skepticism is becoming the order of the day. The early hype of people like Knewton’s Jose Ferreira is now understood for what it was. Companies like Macmillan got their fingers badly burnt when they barked up the wrong tree with their ‘Big Tree’ platform.

Noel Enyedy captures a more contemporary understanding when he writes: Personalized Instruction is based on the metaphor of personal desktop computers—the technology of the 80s and 90s. Today’s technology is not just personal but mobile, social, and networked. The flexibility and social nature of how technology infuses other aspects of our lives is not captured by the model of Personalized Instruction, which focuses on the isolated individual’s personal path to a fixed end-point. To truly harness the power of modern technology, we need a new vision for educational technology (Enyedy, 2014: 16).

Adaptive solutions aren’t going away, but there is now a much better understanding of what sorts of problems might have adaptive solutions. Testing is certainly one. As the educational technology plan from the U.S. Department of Education (‘Future Ready Learning: Re-imagining the Role of Technology in Education’, 2016) puts it: Computer adaptive testing, which uses algorithms to adjust the difficulty of questions throughout an assessment on the basis of a student’s responses, has facilitated the ability of assessments to estimate accurately what students know and can do across the curriculum in a shorter testing session than would otherwise be necessary. In ELT, Pearson and EF have adaptive tests that have been well researched and designed.

Vocabulary apps which deploy adaptive technology continue to become more sophisticated, although empirical research is lacking. Automated writing tutors with adaptive corrective feedback are also developing fast, and I’ll be writing a post about these soon. Similarly, as speech recognition software improves, we can expect to see better and better automated adaptive pronunciation tutors. But going beyond such applications, there are bigger questions to ask, and answers to these will impact on whatever direction adaptive technologies take. Large platforms (LMSs), with or without adaptive software, are already beginning to look rather dated. Will they be replaced by integrated apps, or are apps themselves going to be replaced by bots (currently riding high in the Hype Cycle)? In language learning and teaching, the future of bots is likely to be shaped by developments in natural language processing (another topic about which I’ll be blogging soon). Nobody really has a clue where the next two and a half years will take us (if anywhere), but it’s becoming increasingly likely that adaptive learning will be only one very small part of it.

 

Enyedy, N. 2014. Personalized Instruction: New Interest, Old Rhetoric, Limited Results, and the Need for a New Direction for Computer-Mediated Learning. Boulder, CO: National Education Policy Center. Retrieved 17.07.16 from http://nepc.colorado.edu/publication/personalized-instruction

‘Sticky’ – as in ‘sticky learning’ or ‘sticky content’ (as opposed to ‘sticky fingers’ or a ‘sticky problem’) – is itself fast becoming a sticky word. If you check out ‘sticky learning’ on Google Trends, you’ll see that it suddenly spiked in September 2011, following the slightly earlier appearance of ‘sticky content’. The historical rise in this use of the word coincides with the exponential growth in the number of references to ‘big data’.

I am often asked if adaptive learning really will take off as a big thing in language learning. Will adaptivity itself be a sticky idea? When the question is asked, people mean the big data variety of adaptive learning, rather than the much more limited adaptivity of spaced repetition algorithms, which, I think, is firmly here and here to stay. I can’t answer the question with any confidence, but I recently came across a book which suggests a useful way of approaching the question.

41u+NEyWjnL._SY344_BO1,204,203,200_‘From the Ivory Tower to the Schoolhouse’ by Jack Schneider (Harvard Education Press, 2014) investigates the reasons why promising ideas from education research fail to get taken up by practitioners, and why other, less-than-promising ideas, from a research or theoretical perspective, become sticky quite quickly. As an example of the former, Schneider considers Robert Sternberg’s ‘Triarchic Theory’. As an example of the latter, he devotes a chapter to Howard Gardner’s ‘Multiple Intelligences Theory’.

Schneider argues that educational ideas need to possess four key attributes in order for teachers to sit up, take notice and adopt them.

  1. perceived significance: the idea must answer a question central to the profession – offering a big-picture understanding rather than merely one small piece of a larger puzzle
  2. philosophical compatibility: the idea must clearly jibe with closely held [teacher] beliefs like the idea that teachers are professionals, or that all children can learn
  3. occupational realism: it must be possible for the idea to be put easily into immediate use
  4. transportability: the idea needs to find its practical expression in a form that teachers can access and use at the time that they need it – it needs to have a simple core that can travel through pre-service coursework, professional development seminars, independent study and peer networks

To what extent does big data adaptive learning possess these attributes? It certainly comes up trumps with respect to perceived significance. The big question that it attempts to answer is the question of how we can make language learning personalized / differentiated / individualised. As its advocates never cease to remind us, adaptive learning holds out the promise of moving away from a one-size-fits-all approach. The extent to which it can keep this promise is another matter, of course. For it to do so, it will never be enough just to offer different pathways through a digitalised coursebook (or its equivalent). Much, much more content will be needed: at least five or six times the content of a one-size-fits-all coursebook. At the moment, there is little evidence of the necessary investment into content being made (quite the opposite, in fact), but the idea remains powerful nevertheless.

When it comes to philosophical compatibility, adaptive learning begins to run into difficulties. Despite the decades of edging towards more communicative approaches in language teaching, research (e.g. the research into English teaching in Turkey described in a previous post), suggests that teachers still see explanation and explication as key functions of their jobs. They believe that they know their students best and they know what is best for them. Big data adaptive learning challenges these beliefs head on. It is no doubt for this reason that companies like Knewton make such a point of claiming that their technology is there to help teachers. But Jose Ferreira doth protest too much, methinks. Platform-delivered adaptive learning is a direct threat to teachers’ professionalism, their salaries and their jobs.

Occupational realism is more problematic still. Very, very few language teachers around the world have any experience of truly blended learning, and it’s very difficult to envisage precisely what it is that the teacher should be doing in a classroom. Publishers moving towards larger-scale blended adaptive materials know that this is a big problem, and are actively looking at ways of packaging teacher training / teacher development (with a specific focus on blended contexts) into the learner-facing materials that they sell. But the problem won’t go away. Education ministries have a long history of throwing money at technological ‘solutions’ without thinking about obtaining the necessary buy-in from their employees. It is safe to predict that this is something that is unlikely to change. Moreover, learning how to become a blended teacher is much harder than learning, say, how to make good use of an interactive whiteboard. Since there are as many different blended adaptive approaches as there are different educational contexts, there cannot be (irony of ironies) a one-size-fits-all approach to training teachers to make good use of this software.

Finally, how transportable is big data adaptive learning? Not very, is the short answer, and for the same reasons that ‘occupational realism’ is highly problematic.

Looking at things through Jack Schneider’s lens, we might be tempted to come to the conclusion that the future for adaptive learning is a rocky path, at best. But Schneider doesn’t take political or economic considerations into account. Sternberg’s ‘Triarchic Theory’ never had the OECD or the Gates Foundation backing it up. It never had millions and millions of dollars of investment behind it. As we know from political elections (and the big data adaptive learning issue is a profoundly political one), big bucks can buy opinions.

It may also prove to be the case that the opinions of teachers don’t actually matter much. If the big adaptive bucks can win the educational debate at the highest policy-making levels, teachers will be the first victims of the ‘creative disruption’ that adaptivity promises. If you don’t believe me, just look at what is going on in the U.S.

There are causes for concern, but I don’t want to sound too alarmist. Nobody really has a clue whether big data adaptivity will actually work in language learning terms. It remains more of a theory than a research-endorsed practice. And to end on a positive note, regardless of how sticky it proves to be, it might just provide the shot-in-the-arm realisation that language teachers, at their best, are a lot more than competent explainers of grammar or deliverers of gap-fills.

The cheer-leading for big data in education continues unabated. Almost everything you read online on the subject is an advertisement, usually disguised as a piece of news or a blog post, but which can invariably be traced back to an organisation with a vested interest in digital disruption.  A typical example is this advergraphic which comes under a banner that reads ‘Big Data Improves Education’. The site, Datafloq, is selling itself as ‘the one-stop-shop around Big Data.’ Their ‘vision’ is ‘Connecting Data and People and [they] aim to achieve that by spurring the understanding, acceptance and application of Big Data in order to drive innovation and economic growth.’

Critical voices are rare, but growing. There’s a very useful bibliography of recent critiques here. And in the world of English language teaching, I was pleased to see that there’s a version of Gavin Dudeney’s talk, ‘Of Big Data & Little Data’, now up on YouTube. The slides which accompany his talk can be accessed here.

His main interest is in reclaiming the discourse of edtech in ELT, in moving away from the current obsession with numbers, and in returning the focus to what he calls ‘old edtech’ – the everyday technological practices of the vast majority of ELT practitioners.2014-12-01_2233

It’s a stimulating and deadpan-entertaining talk and well worth 40 minutes of your time. Just fast-forward the bit when he talks about me.

If you’re interested in hearing more critical voices, you may also like to listen to a series of podcasts, put together by the IATEFL Learning Technologies and Global Issues Special Interest Groups. In the first of these, I interview Neil Selwyn and, in the second, Lindsay Clandfield interviews Audrey Watters of Hack Education.

 

2014-09-30_2216Jose Ferreira, the fast-talking sales rep-in-chief of Knewton, likes to dazzle with numbers. In a 2012 talk hosted by the US Department of Education, Ferreira rattles off the stats: So Knewton students today, we have about 125,000, 180,000 right now, by December it’ll be 650,000, early next year it’ll be in the millions, and next year it’ll be close to 10 million. And that’s just through our Pearson partnership. For each of these students, Knewton gathers millions of data points every day. That, brags Ferreira, is five orders of magnitude more data about you than Google has. … We literally have more data about our students than any company has about anybody else about anything, and it’s not even close. With just a touch of breathless exaggeration, Ferreira goes on: We literally know everything about what you know and how you learn best, everything.

The data is mined to find correlations between learning outcomes and learning behaviours, and, once correlations have been established, learning programmes can be tailored to individual students. Ferreira explains: We take the combined data problem all hundred million to figure out exactly how to teach every concept to each kid. So the 100 million first shows up to learn the rules of exponents, great let’s go find a group of people who are psychometrically equivalent to that kid. They learn the same ways, they have the same learning style, they know the same stuff, because Knewton can figure out things like you learn math best in the morning between 8:40 and 9:13 am. You learn science best in 42 minute bite sizes the 44 minute mark you click right, you start missing questions you would normally get right.

The basic premise here is that the more data you have, the more accurately you can predict what will work best for any individual learner. But how accurate is it? In the absence of any decent, independent research (or, for that matter, any verifiable claims from Knewton), how should we respond to Ferreira’s contribution to the White House Education Datapalooza?

A 51Oy5J3o0yL._AA258_PIkin4,BottomRight,-46,22_AA280_SH20_OU35_new book by Stephen Finlay, Predictive Analytics, Data Mining and Big Data (Palgrave Macmillan, 2014) suggests that predictive analytics are typically about 20 – 30% more accurate than humans attempting to make the same judgements. That’s pretty impressive and perhaps Knewton does better than that, but the key thing to remember is that, however much data Knewton is playing with, and however good their algorithms are, we are still talking about predictions and not certainties. If an adaptive system could predict with 90% accuracy (and the actual figure is typically much lower than that) what learning content and what learning approach would be effective for an individual learner, it would still mean that it was wrong 10% of the time. When this is scaled up to the numbers of students that use Knewton software, it means that millions of students are getting faulty recommendations. Beyond a certain point, further expansion of the data that is mined is unlikely to make any difference to the accuracy of predictions.

A further problem identified by Stephen Finlay is the tendency of people in predictive analytics to confuse correlation and causation. Certain students may have learnt maths best between 8.40 and 9.13, but it does not follow that they learnt it best because they studied at that time. If strong correlations do not involve causality, then actionable insights (such as individualised course design) can be no more than an informed gamble.

Knewton’s claim that they know how every student learns best is marketing hyperbole and should set alarm bells ringing. When it comes to language learning, we simply do not know how students learn (we do not have any generally accepted theory of second language acquisition), let alone how they learn best. More data won’t help our theories of learning! Ferreira’s claim that, with Knewton, every kid gets a perfectly optimized textbook, except it’s also video and other rich media dynamically generated in real time is equally preposterous, not least since the content of the textbook will be at least as significant as the way in which it is ‘optimized’. And, as we all know, textbooks have their faults.

Cui bono? Perhaps huge data and predictive analytics will benefit students; perhaps not. We will need to wait and find out. But Stephen Finlay reminds us that in gold rushes (and internet booms and the exciting world of Big Data) the people who sell the tools make a lot of money. Far more strike it rich selling picks and shovels to prospectors than do the prospectors. Likewise, there is a lot of money to be made selling Big Data solutions. Whether the buyer actually gets any benefit from them is not the primary concern of the sales people. (p.16/17) Which is, perhaps, one of the reasons that some sales people talk so fast.

Talk the big data talk

Posted: March 7, 2014 in big data
Tags: ,

Pearson’s Efficacy document has a chapter called ‘A New Era of Learning Efficacy on a Planet of Smarter Systems’ by Jon Iwata. It’s basically a paean of praise to the potential of big data.  I threw the text into a word cloud program to see what would come up. And what we get is a handy little guide for anyone who wants to bluff their way in a discussion about adaptive learning. Alternatively, you could use it for buzz word bingo.

Efficacy

Some of the words you won’t be needing at all are ‘teach’, ‘teachers’, ‘classrooms’ or ‘lessons’. Sorry about the blur in the image: it’s my laptop having an emotional response.