Researching research: part 2

(This post won’t make a lot of sense unless you read the previous one – Researching research: part 1!)

I suggested in the previous post that the research of Jayaprakash et al had confirmed something that we already knew concerning the reasons why some students drop out of college. However, predictive analytics are only part of the story. As the authors of this paper point out, they ‘do not influence course completion and retention rates without being combined with effective intervention strategies aimed at helping at-risk students succeed’. The point of predictive analytics is to facilitate the deployment of effective and appropriate interventions strategies, and to do this sooner than would be possible without the use of the analytics. So, it is to these intervention strategies that I now turn.

Interventions to help at-risk students included the following:

• Sending students messages to inform them that they are at risk of not completing the course (‘awareness messaging’)
• Making students more aware of the available academic support services (which could, for example, direct them to a variety of campus-based or online resources)
• Promoting peer-to-peer engagement (e.g. with an online ‘student lounge’ discussion forum)
• Providing access to self-assessment tools

The design of these interventions was based on the work that had been done at Purdue, which was, in turn, inspired by the work of Vince Tinto, one of the world’s leading experts on student retention issues.

The work done at Purdue had shown that simple notifications to students that they were at risk could have a significant, and positive, effect on student behaviour. Jayaprakash and the research team took the students who had been identified as at-risk by the analytics and divided them into three groups: the first were issued with ‘awareness messages’, the second were offered a combination of the other three interventions in the bullet point list above, and the third, a control group, had no interventions at all. The results showed that the students who were in treatment groups (of either kind of intervention) showed a statistically significant improvement compared to those who received no treatment at all. However, there seemed to be no difference in the effectiveness of the different kinds of intervention.

So far, so good, but, once again, I was left thinking that I hadn’t really learned very much from all this. But then, in the last five pages, the article suddenly got very interesting. Remember that the primary purpose of this whole research project was to find ways of helping not just at-risk students, but specifically socioeconomically disadvantaged at-risk students (such as those receiving Pell Grants). Accordingly, the researchers then focussed on this group. What did they find?

Once again, interventions proved more effective at raising student scores than no intervention at all. However, the averages of final scores are inevitably affected by drop-out rates (since students who drop out do not have final scores which can be included in the averages). At Purdue, the effect of interventions on drop-out rates had not been found to be significant. Remember that Purdue has a relatively well-off student demographic. However, in this research, which focussed on colleges with a much higher proportion of students on Pell Grants, the picture was very different. Of the Pell Grant students who were identified as at-risk and who were given some kind of treatment, 25.6% withdrew from the course. Of the Pell Grant students who were identified as at-risk but who were not ‘treated’ in any way (i.e. those in the control group), only 14.1% withdrew from the course. I recommend that you read those numbers again!

The research programme had resulted in substantially higher drop-out rates for socioeconomically disadvantaged students – the precise opposite of what it had set out to achieve. Jayaprakash et al devote one page of their article to the ethical issues this raises. They suggest that early intervention, resulting in withdrawal, might actually be to the benefit of some students who were going to fail whatever happened. It is better to get a ‘W’ (withdrawal) grade on your transcript than an ‘F’ (fail), and you may avoid wasting your money at the same time. This may be true, but it would be equally true that not allowing at-risk students (who, of course, are disproportionately from socioeconomically disadvantaged backgrounds) into college at all might also be to their ‘benefit’. The question, though, is: who has the right to make these decisions on behalf of other people?

The authors also acknowledge another ethical problem. The predictive analytics which will prompt the interventions are not 100% accurate. 85% accuracy could be considered a pretty good figure. This means that some students who are not at-risk are labelled as at-risk, and other who are at-risk are not identified. Of these two possibilities, I find the first far more worrying. We are talking about the very real possibility of individual students being pushed into making potentially life-changing decisions on the basis of dodgy analytics. How ethical is that? The authors’ conclusion is that the situation forces them ‘to develop the most accurate predictive models possible, as well as to take steps to reduce the likelihood that any intervention would result in the necessary withdrawal of a student’.

I find this extraordinary. It is premised on the assumption that predictive models can be made much, much more accurate. They seem to be confusing prediction and predeterminism. A predictive model is, by definition, only predictive. There will always be error. How many errors are ethically justifiable? And, the desire to reduce the likelihood of unnecessary withdrawals is a long way from the need to completely eliminate the likelihood of unnecessary withdrawals, which seems to me to be the ethical position. More than anything else in the article, this sentence illustrates that the a priori assumption is that predictive analytics can be a force for good, and that the only real problem is getting the science right. If a number of young lives are screwed up along the way, we can at least say that science is getting better.

In the authors’ final conclusion, they describe the results of their research as ‘promising’. They do not elaborate on who it is promising for. They say that relatively simple intervention strategies can positively impact student learning outcomes, but they could equally well have said that relatively simple intervention strategies can negatively impact learning outcomes. They could have said that predictive analytics and intervention programmes are fine for the well-off, but more problematic for the poor. Remembering once more that the point of the study was to look at the situation of socioeconomically disadvantaged at-risk students, it is striking that there is no mention of this group in the researchers’ eight concluding points. The vast bulk of the paper is devoted to technical descriptions of the design and training of the software; the majority of the conclusions are about the validity of that design and training. The ostensibly intended beneficiaries have got lost somewhere along the way.

How and why is it that a piece of research such as this can so positively slant its results? In the third and final part of this mini-series, I will turn my attention to answering that question.

Saving the world (adaptive marketing)

‘Adaptive’ is a buzzword in the marketing of educational products. Chris Dragon, President of Pearson Digital Learning, complained on the Pearson Research blog. that there are so many EdTech providers claiming to be ‘adaptive’ that you have to wonder if they are not using the term too loosely. He talks about semantic satiation, the process whereby ‘temporary loss of meaning [is] experienced when one is exposed to the uninterrupted repetition of a word or phrase’. He then goes on to claim that Pearson’s SuccessMaker (‘educational software that differentiates and personalizes K-8 reading and math instruction’) is the real adaptive McCoy.

‘Adaptive’ is also a buzzword in marketing itself. Google the phrase ‘adaptive marketing’ and you’ll quickly come up with things like Adaptive Marketing Set to Become the Next Big Thing or Adaptive marketing changes the name of the game. Adaptive marketing is what you might expect: the use of big data to track customers and enable ‘marketers to truly tailor their activities in rapid and unparalleled ways to meet their customers’ interests and needs’ (Advertising Age, February 2012). It strikes me that this sets up an extraordinary potential loop: students using adaptive learning software that generates a huge amount of data which could then be used by adaptive marketers to sell other products.

I decided it might be interesting to look at the way one adaptive software company markets itself. Knewton, for example, which claims its products are more adaptive than anybody else’s.

Knewton clearly spend a lot of time and money on their marketing efforts. There is their blog and a magazine called ‘The Knerd’. There are very regular interviews by senior executives with newspapers, magazines and other blogs. There are very frequent conference presentations. All of these are easily accessible, so it is quite easy to trace Knewton’s marketing message. And even easier when they are so open about it. David Liu, Chief Operating Officer has given an interview  in which he outlines his company’s marketing strategy. Knewton, he says, focuses on driving organic interests and traffic. To that end, we have a digital marketing group that’s highly skilled and focused on creating content marketing so users, influencers and partners alike can understand our product, the value we bring and how to work with us. We also use a lot of advanced digital and online lead generation type of techniques to target potential partners and users to be able to get the right people in those discussions.

The message consists of four main strands, which I will call EdTech, EduCation, EduBusiness and EdUtopia. Depending on the audience, the marketing message will be adapted, with one or other of these strands given more prominence.

1 EdTech

Hardly surprisingly, Knewton focuses on what they call their ‘heavy duty infrastructure for an adaptive world’. They are very proud of their adaptive credentials, their ‘rigorous data science’. The basic message is that ‘only Knewton provides true personalization for any student, anywhere’. They are not shy of using technical jargon and providing technical details to prove their point.

2 EduCation

The key message here is effectiveness (Knewton also uses the term ‘efficacy’). Statistics about growth in pass rates and reduction in withdrawal rates at institutions are cited. At the same time, teachers are directly appealed to with statements like ‘as a teacher, you get tools you never had before’ and ‘teachers will be able to add their own content, upload it, tag it and seamlessly use it’. Accompanying this fairly direct approach is a focus on buzz words and phrases which can be expected to resonate with teachers. Recent blog posts include in their headlines: ‘supporting creativity’, ‘student-centred learning’, ‘peer mentoring’, ‘formative evaluation’, ‘continuous assessment’, ‘learning styles’, ‘scaffolding instruction’, ‘real-world examples’, ‘enrichment’ or ‘lifelong learning’.

There is an apparent openness in Knewton’s readiness to communicate with the rest of the world. The blog invites readers to start discussions and post comments. Almost no one does. But one blog post by Jose Ferreira called ‘Rebooting Learning Styles’  provoked a flurry of highly critical and well-informed responses. These remain unanswered. A similar thing happened when David Liu did a guest post at eltjam. A flurry of criticism, but no response. My interpretation of this is that Knewton are a little scared of engaging in debate and of having their marketing message hijacked.

3 EduBusiness

Here’s a sample of ways that Knewton speak to potential customers and investors:

an enormous new market of online courses that bring high margin revenue and rapid growth for institutions that start offering them early and declining numbers for those who do not.

Because Knewton is trying to disrupt the traditional industry, we have nothing to lose—we’re not cannibalising ourselves—by partnering.

Unlike other groups dabbling in adaptive learning, Knewton doesn’t force you to buy pre-fabricated products using our own content. Our platform makes it possible for anyone — publishers, instructors, app developers, and others — to build her own adaptive applications using any content she likes.

The data platform industries tend to have a winner-take-all dynamic. You take that and multiply it by a very, very high-stakes product and you get an even more winner-take-all dynamic.

4 EdUtopia

I personally find this fourth strand the most interesting. Knewton are not unique in adopting this line, but it is a sign of their ambition that they choose to do so. All of the quotes that follow are from Jose Ferreira:

We can’t improve education by curing poverty. We have to cure poverty by improving education.

Edtech is our best hope to narrow — at scale — the Achievement Gap between rich and poor. Yet, for a time, it will increase that gap. Society must push past that unfortunate moment and use tech-assisted outcome improvements as the rationale to drive spending in poor schools.

I started Knewton to do my bit to fix the world’s education system. Education is among the most important problems we face, because it’s the ultimate “gateway” problem. That is, it drives virtually every global problem that we face as a species. But there’s a flip-side: if we can fix education, then we’ll dramatically improve the other problems, too. So in fact, I started Knewton not just to help fix education but to try to fix just about everything.

What if the girl who invents the cure for ovarian cancer is growing up in a Cambodian fishing village and otherwise wouldn’t have a chance? As distribution of technology continues to improve, adaptive learning will give her and similar students countless opportunities that they otherwise wouldn’t have.

But our ultimate vision – and what really motivated me to start the company – is to solve the access problem for the human race once and for all. Only 22% of the world finishes high school; only 55% finish sixth grade. This is a preventable tragedy. Adaptive learning can give students around the world access to high-quality education they wouldn’t otherwise have.

Voxy – ‘the future of language learning’?

Voxy is another language learning platform that likes to tout itself as ‘the future of language learning’. It has over 2.5 million users and claims to be the No. 1 education iTunes app in 23 countries. Pearson is a major investor and has a seat on the Voxy board. Unsurprisingly, it boasts ‘a new sophisticated and patented adaptive learning technology, […] a dynamic feedback loop which results in lessons and courses that calibrate to the learner. These improvements are fundamental to what makes Voxy unique as lessons become even more personalized.’

Voxy uses an integrated web / mobile / SMS platform to deliver its learning programme, which is based around authentic, up-to-date texts. I spent a morning as an advanced learner of English exploring what it had to offer. In what I did, everything was in English, but I imagine this is not the case for lower-level learners. Voxy was originally launched for speakers of Spanish and Portuguese.

As far as I could tell, there is very little that is (what I would call) adaptive. There is, no doubt, adaptive software at work in the vocabulary revision exercises, but it’s hard to see this operating. Before starting, users are asked about their level and what they want to ‘accomplish with English’. The six possible answers are ‘advance my career’, ‘enjoy English media’, ‘pass my English tests’, ‘travel abroad’. ‘day-to-day tasks’ and ‘social and lifestyle’. I was next asked about my interests, and the possible answers here were sports, celebrities and entertainment, business, technology, health and politics. Having answered these questions, my personalized course was ready.

I was offered a deal of $20 a month, with a free trial. This gave me access to the main course, a faily rudimentary grammar guide, a list of words I had ‘studied’, a proficiency test (reading, listening and TOEFL-style M/C grammar) and 13 hours with a ‘live’ tutor.

I decided that I couldn’t pretend to be a real learner and hook up with a tutor. Users can choose a tutor from a menu where the tutors are photographed (obligatory smile). They are young graduates and some, but not all, are described as having ‘Certification: Teaching English’, whatever this means. There are also tutor statements, one of which reads ‘I love that both teaching and studying foreign languages are abound with opportunities to experience international differences and similarities on a personal level’ (sic).

I concentrated on the main course which offered 18 lessons related to each of my declared interests. These were based on authentic texts from sources like Financial Times and New York Daily News. These were generally interesting and up-to-date. In some cases, the article was only 24 hours old.

The usual procedure was to (1) read the text, (2) tap on highlighted words, which would bring up dictionary definitions and a recording of the word, (3) listen to a recording of the text (read very slowly – far too slowly for anyone with an advanced level), (4) answer 2 -4 multiple choice questions, (5) be shown short gapped extracts from the text alongside 4 or 5 boxes, which, when you click on them gave a recording of different words, one of which was the correct answer to the highlighted gap in the text, and (6) do a word – definition matching task (the words from stage 5).

According to Wikipedia, Voxy is based on the principles of task-based language teaching. Jane Willis might beg to differ. What I saw was closer to those pre-1970s textbooks where texts were followed by glossaries. Voxy is technologically advanced, but methodologically, it is positively antediluvian.

A further problem concerns task design. Perhaps because the tasks that accompany the texts have to be produced very quickly (if the texts are really to be hot off the press), there were errors that no experienced materials writer would make, and no experienced ELT editor would fail to spot. The sorts of problems that I identifed included the following:

• No clear rationale in the selection of vocabulary items; no apparent awareness of word difficulty or frequency.
• No clear rationale in the selection of multiple choice items.
• Many M/C vocabulary questions can be answered without understanding the word (simply by using the memory).
• Vocabulary definition matching tasks often contain language in the definitions which is more complex than the target item.
• The vocabulary definition matching tasks can mostly be done simply by eliminating the distractors (which have been plucked out of thin air, and have not previously appeared).
• The definitions in these matching tasks often do not use the same grammar as the target item (e.g. an infinitive in the definition has to be matched to a participle target word).
• Errors (e.g. ‘The brain reacts more strongly to rejection in real life that online rejection’ (sic) in one M/C item).

I could go on. The material has clearly not been written by experienced writers, it has not been properly edited or trialled. The texts may be interesting, but that’s the only positive that I can offer for the main part of the course that I looked at.

My greatest disappointment concerns the poor use that the technology has been put to. Contrary to Voxy’s claims, this is not a new way to learn a language, it’s not particularly fun and it’s hard to believe that it could be effective. Perhaps my, admittedly limited, experience with Voxy’s product was unrepresentative. Using authentic materials is a good idea, but this needs to be combined with decent social networking possibilities, a much more sophisticated use of adaptive technology, proper investment in item-writers and editors, and more. The future of language learning? Probably not.

Personalization for all

Personalization is one of the key leitmotifs in current educational discourse. The message is clear: personalization is good, one-size-fits-all is bad. ‘How to personalize learning and how to differentiate instruction for diverse classrooms are two of the great educational challenges of the 21^st century,’ write Trilling and Fadel, leading lights in the Partnership for 21st Century Skills (P21)[1]. Barack Obama has repeatedly sung the praises of, and the need for, personalized learning and his policies are fleshed out by his Secretary of State, Arne Duncan, in speeches and on the White House blog: ‘President Obama described the promise of personalized learning when he launched the ConnectED initiative last June. Technology is a powerful tool that helps create robust personalized learning environments.’ In the UK, personalized learning has been government mantra for over 10 years. The EU, UNESCO, OECD, the Gates Foundation – everyone, it seems, is singing the same tune.

Personalization, we might all agree, is a good thing. How could it be otherwise? No one these days is going to promote depersonalization or impersonalization in education. What exactly it means, however, is less clear. According to a UNESCO Policy Brief[2], the term was first used in the context of education in the 1970s by Victor Garcìa Hoz, a senior Spanish educationalist and member of Opus Dei at the University of Madrid. This UNESCO document then points out that ‘unfortunately, up to this date there is no single definition of this concept’.

In ELT, the term has been used in a very wide variety of ways. These range from the far-reaching ideas of people like Gertrude Moskowitz, who advocated a fundamentally learner-centred form of instruction, to the much more banal practice of getting students to produce a few personalized examples of an item of grammar they have just studied. See Scott Thornbury’s A-Z blog for an interesting discussion of personalization in ELT.

As with education in general, and ELT in particular, ‘personalization’ is also bandied around the adaptive learning table. Duolingo advertises itself as the opposite of one-size-fits-all, and as an online equivalent of the ‘personalized education you can get from a small classroom teacher or private tutor’. Babbel offers a ‘personalized review manager’ and Rosetta Stone’s Classroom online solution allows educational institutions ‘to shift their language program away from a ‘one-size-fits-all-curriculum’ to a more individualized approach’. As far as I can tell, the personalization in these examples is extremely restricted. The language syllabus is fixed and although users can take different routes up the ‘skills tree’ or ‘knowledge graph’, they are totally confined by the pre-determination of those trees and graphs. This is no more personalized learning than asking students to make five true sentences using the present perfect. Arguably, it is even less!

This is not, in any case, the kind of personalization that Obama, the Gates Foundation, Knewton, et al have in mind when they conflate adaptive learning with personalization. Their definition is much broader and summarised in the US National Education Technology Plan of 2010: ‘Personalized learning means instruction is paced to learning needs, tailored to learning preferences, and tailored to the specific interests of different learners. In an environment that is fully personalized, the learning objectives and content as well as the method and pace may all vary (so personalization encompasses differentiation and individualization).’ What drives this is the big data generated by the students’ interactions with the technology (see ‘Part 4: big data and analytics’ of ‘The Guide’ on this blog).

What remains unclear is exactly how this might work in English language learning. Adaptive software can only personalize to the extent that the content of an English language learning programme allows it to do so. It may be true that each student using adaptive software ‘gets a more personalised experience no matter whose content the student is consuming’, as Knewton’s David Liu puts it. But the potential for any really meaningful personalization depends crucially on the nature and extent of this content, along with the possibility of variable learning outcomes. For this reason, we are not likely to see any truly personalized large-scale adaptive learning programs for English any time soon.

Nevertheless, technology is now central to personalized language learning. A good learning platform, which allows learners to connect to ‘social networking systems, podcasts, wikis, blogs, encyclopedias, online dictionaries, webinars, online English courses, various apps’, etc (see Alexandra Chistyakova’s eltdiary), means that personalization could be more easily achieved.

For the time being, at least, adaptive learning systems would seem to work best for ‘those things that can be easily digitized and tested like math problems and reading passages’ writes Barbara Bray . Or low level vocabulary and grammar McNuggets, we might add. Ideal for, say, ‘English Grammar in Use’. But meaningfully personalized language learning?

‘Personalized learning’ sounds very progressive, a utopian educational horizon, and it sounds like it ought to be the future of ELT (as Cleve Miller argues). It also sounds like a pretty good slogan on which to hitch the adaptive bandwagon. But somehow, just somehow, I suspect that when it comes to adaptive learning we’re more likely to see more testing, more data collection and more depersonalization.

[1] Trilling, B. & Fadel, C. 2009 21^st Century Skills (San Francisco: Wiley) p.33

[2] Personalized learning: a new ICT­enabled education approach, UNESCO Institute for Information Technologies in Education, Policy Brief March 2012 iite.unesco.org/pics/publications/en/files/3214716.pdf

 

Creative destruction and the winner-take-all market

Let’s take a look at the business of adaptive learning from a publisher’s perspective. Not an ELT publisher, but someone a few rungs higher on the ladder with strategic responsibilities. You might not know a great deal about ELT. It is, after all, only one of a number of divisions you are responsible for, and not an especially profitable one at that. You will, however, know a lot about creative destruction, the process by which one industry is replaced by another. The decline and demise of printed magazines, newspapers and books, of book reviewers and traditional booksellers, and their replacement by digital products will be such a part of your professional knowledge that they hardly need to be mentioned. Graphs such as the one below from PricewaterhouseCoopers (PwC) will be terribly familiar. You will also be aware that the gales of creative destruction in publishing are blowing harder than ever before.

In fact, you probably owe your job to your ability to talk convincingly about creative destruction and how to profit from it. Whatever your particular strategy for the future might be, you will have noted the actions of others. You will have evaluated advice, such as the following, from Publishing Perspectives

• Do not delay in taking action when there are clear signals of a decline in market value.
• Trade your low-profit print assets (even though some may have limited digital value) for core product that has a higher likelihood of success and can be exploited digitally.
• Look for an orderly transition from print to digital product which enables a company to reinvent itself.

You will be looking to invest in technology, and prioritizing the acquisition of technological expertise (through partnerships or the purchase of start-ups) over the development of traditional ELT products. Your company will be restructured, and possibly renamed, to facilitate the necessary changes.

You will also know that big data and analytics have already transformed other industries. And you will know that educational publishing is moving towards a winner-take-all business market, where ‘the best performers are able to capture a very large share of the rewards, and the remaining competitors are left with very little’ (Investopedia). Erik Brynjolfsson and Andrew McAfee’s new book, The Second Machine Age (New York: Norton, 2014), argues that ‘each time a market becomes more digital, winner-take-all economics become a little more compelling …Digitization creates winner-take-all markets because [there are] enormous economies of scale, giving the market leader a huge cost advantage and room to beat the price of any competitor while still making a good profit’ (pp.153-155).

It is in this light that we need to understand the way that companies like Pearson and Macmillan are banking everything on a digital future. Laurie Harrison’s excellent blog post at eltjam  summarises the Pearson position: ‘the world’s biggest education publisher is spending £150m on a total restructure which involves an immediate move to digital learning, a focus on emerging markets, and a transformation from publisher to education services provider. If the English language learning market is worth $4billion a year, then Pearson still only have a very small chunk of it. And if you’re a company as successful and ambitious as Pearson, that just isn’t good enough – so a change of direction is needed. In order to deliver this change, the company have recently announced their new senior management team.’

Adaptive learning fits the new business paradigm perfectly. If the hype is to be believed, adaptive learning will be a game-changer. ‘The shifting of education from analog to digital is a one-time event in the history of the human race. At scale, it will have as big an effect on the world as indoor plumbing or electricity,’ writes Jose Ferreira of Knewton. ‘True disruption,’ he says elsewhere, ‘happens when entrepreneurs aim big and go after a giant problem, a problem that, if solved, would usher in an era of large-scale transformation across industries and nations. … Education is the last of the information industries to move online,’ he goes on. ‘When it breaks, it breaks fast. And that’s going to happen in the next five years. All the education content will go online in the next 10 years. And textbooks will go away. … Ultimately, all learning materials will be digital and they will all be adaptive.’

Ferreira clearly knows all about creative disruption. He also knows about winner-take-all markets. ‘The question is who is going to power [the] platform,’ he writes. ‘It’s probably going to be one or two companies’. He states his ambition for Knewton very clearly: ‘Knewton’s goal is to be like Amazon Web Services for education’. ‘It’s pretty clear to us,’ he writes, ‘that there’s going to be one dominant data platform for education, the way there’s one dominant data platform for search, social media, etailing. But in education, it’s going to be even more winner-take-all; there will be a number of companies that make up the platform, like Wintel. People might make a perverse choice to use Bing for search because they don’t like Google. But no one’s going to make the choice to subject their kid to the second-best adaptive learning platform, if that means there’s a 23% structural disadvantage. The data platform industries tend to have a winner-take-all dynamic. You take that and multiply it by a very, very high-stakes product and you get an even more winner-take-all dynamic.’

What is at stake in this winner-take-all market? Over to Jose Ferreira one more time: ‘The industry is massive. It’s so massive that virtually nobody I’ve met truly grasps how big it is. It’s beyond their frame of reference. The total amount of money (both public and private) spent annually exceeds all spending, both online and offline, of every other information industry combined: that is, all media, entertainment, games, news, software, Internet and mobile media, e-tailing, etc.’

But, still, a few questions continue to nag away at me. If all of this is so certain, why does Jose Ferreira feel the need to talk about it so much? If all of this is so certain, why don’t all the ELT publishers jump on the bandwagon? What sort of track record does economic forecasting have, anyway?

Adaptive reading?

There is a lot that technology can do to help English language learners develop their reading skills. The internet makes it possible for learners to read an almost limitless number of texts that will interest them, and these texts can evaluated for readability and, therefore, suitability for level (see here for a useful article). RSS opens up exciting possibilities for narrow reading and the positive impact of multimedia-enhanced texts was researched many years ago. There are good online bilingual dictionaries and other translation tools. There are apps that go with graded readers (see this review in the Guardian) and there are apps that can force you to read at a certain speed. And there is more. All of this could very effectively be managed on a good learning platform.

Could adaptive software add another valuable element to reading skills development?

Adaptive reading programs are spreading in the US in primary education, and, with some modifications, could be used in ELT courses for younger learners and for those who do not have the Roman alphabet. One of the most well-known has been developed by Lexia Learning®, a company that won a $500,000 grant from the Gates Foundation last year. Lexia Learning® was bought by Rosetta Stone® for $22.5 million in June 2013.

One of their products, Lexia Reading Core5, ‘provides explicit, systematic, personalized learning in the six areas of reading instruction, and delivers norm-referenced performance data and analysis without interrupting the flow of instruction to administer a test. Designed specifically to meet the Common Core and the most rigorous state standards, this research-proven, technology-based approach accelerates reading skills development, predicts students’ year-end performance and provides teachers data-driven action plans to help differentiate instruction’.

The predictable claim that it is ‘research-proven’ has not convinced everyone. Richard Allington, a professor of literacy studies at the University of Tennessee and a past president of both the International Reading Association and the National Reading Association, has said that all the companies that have developed this kind of software ‘come up with evidence – albeit potential evidence — that kids could improve their abilities to read by using their product. It’s all marketing. They’re selling a product. Lexia is one of these programs. But there virtually are no commercial programs that have any solid, reliable evidence that they improve reading achievement.’[1] He has argued that the $12 million that has been spent on the Lexia programs would have been better spent on a national program, developed at Ohio State University, that matches specially trained reading instructors with students known to have trouble learning to read.

But what about ELT? For an adaptive program like Lexia’s to work, reading skills need to be broken down in a similar way to the diagram shown above. Let’s get some folk linguistics out of the way first. The sub-skills of reading are not skimming, scanning, inferring meaning from context, etc. These are strategies that readers adopt voluntarily in order to understand a text better. If a reader uses these strategies in their own language, they are likely to transfer these strategies to their English reading. It seems that ELT instruction in strategy use has only limited impact, although this kind of training may be relevant to preparation for exams. This insight is taking a long time to filter down to course and coursebook design, but there really isn’t much debate[2]. Any adaptive ELT reading program that confuses reading strategies with reading sub-skills is going to have big problems.

What, then, are the sub-skills of reading? In what ways could reading be broken down into a skill tree so that it is amenable to adaptive learning? Researchers have provided different answers. Munby (1978), for example, listed 19 reading microskills, Heaton (1988) listed 14. However, a bigger problem is that other researchers (e.g. Lunzer 1979, Rost 1993) have failed to find evidence that distinct sub-skills actually exist. While it is easier to identify sub-skills for very low level readers (especially for those whose own language is very different from English), it is simply not possible to do so for higher levels.

Reading in another language is a complex process which involves both top-down and bottom-up strategies, is intimately linked to vocabulary knowledge and requires the activation of background, cultural knowledge. Reading ability, in the eyes of some researchers, is unitary or holistic. Others prefer to separate things into two components: word recognition and comprehension[3]. Either way, a consensus is beginning to emerge that teachers and learners might do better to focus on vocabulary extension (and this would include extensive reading) than to attempt to develop reading programs that assume the multidivisible nature of reading.

All of which means that adaptive learning software and reading skills in ELT are unlikely bedfellows. To be sure, an increased use of technology (as described in the first paragraph of this post) in reading work will generate a lot of data about learner behaviours. Analysis of this data may lead to actionable insights, and it may not! It will be interesting to find out.

 

[1] http://www.khi.org/news/2013/jun/17/budget-proviso-reading-program-raises-questions/

[2] See, for example, Walter, C. & M. Swan. 2008. ‘Teaching reading skills: mostly a waste of time?’ in Beaven, B. (ed.) IATEFL 2008 Exeter Conference Selections. (Canterbury: IATEFL). Or go back further to Alderson, J. C. 1984 ‘Reading in a foreign language: a reading problem or a language problem?’ in J.C. Alderson & A. H. Urquhart (eds.) Reading in a Foreign Language (London: Longman)

[3] For a useful summary of these issues, see ‘Reading abilities and strategies: a short introduction’ by Feng Liu (International Education Studies 3 / 3 August 2010) www.ccsenet.org/journal/index.php/ies/article/viewFile/6790/5321

Privacy

I mentioned the issue of privacy very briefly in Part 9 of the ‘Guide’, and it seems appropriate to take a more detailed look.

Adaptive learning needs big data. Without the big data, there is nothing for the algorithms to work on, and the bigger the data set, the better the software can work. Adaptive language learning will be delivered via a platform, and the data that is generated by the language learner’s interaction with the English language program on the platform is likely to be only one, very small, part of the data that the system will store and analyse. Full adaptivity requires a psychometric profile for each student.

It would make sense, then, to aggregate as much data as possible in one place. Besides the practical value in massively combining different data sources (in order to enhance the usefulness of the personalized learning pathways), such a move would possibly save educational authorities substantial amounts of money and allow educational technology companies to mine the rich goldmine of student data, along with the standardised platform specifications, to design their products.

And so it has come to pass. The Gates Foundation (yes, them again) provided most of the $100 million funding. A division of Murdoch’s News Corp built the infrastructure. Once everything was ready, a non-profit organization called inBloom was set up to run the thing. The inBloom platform is open source and the database was initially free, although this will change. Preliminary agreements were made with 7 US districts and involved millions of children. The data includes ‘students’ names, birthdates, addresses, social security numbers, grades, test scores, disability status, attendance, and other confidential information’ (Ravitch, D. ‘Reign of Error’ NY: Knopf, 2013, p. 235-236). Under federal law, this information can be ‘shared’ with private companies selling educational technology and services.

The edtech world rejoiced. ‘This is going to be a huge win for us’, said one educational software provider; ‘it’s a godsend for us,’ said another. Others are not so happy. If the technology actually works, if it can radically transform education and ‘produce game-changing outcomes’ (as its proponents claim so often), the price to be paid might just conceivably be worth paying. But the price is high and the research is not there yet. The price is privacy.

The problem is simple. InBloom itself acknowledges that it ‘cannot guarantee the security of the information stored… or that the information will not be intercepted when it is being transmitted.’ Experience has already shown us that organisations as diverse as the CIA or the British health service cannot protect their data. Hackers like a good challenge. So do businesses.

The anti-privatization (and, by extension, the anti-adaptivity) lobby in the US has found an issue which is resonating with electors (and parents). These dissenting voices are led by Class Size Matters, and their voice is being heard. Of the original partners of inBloom, only one is now left. The others have all pulled out, mostly because of concerns about privacy, although the remaining partner, New York, involves personal data on 2.7 million students, which can be shared without any parental notification or consent.

This might seem like a victory for the anti-privatization / anti-adaptivity lobby, but it is likely to be only temporary. There are plenty of other companies that have their eyes on the data-mining opportunities that will be coming their way, and Obama’s ‘Race to the Top’ program means that the inBloom controversy will be only a temporary setback. ‘The reality is that it’s going to be done. It’s not going to be a little part. It’s going to be a big part. And it’s going to be put in place partly because it’s going to be less expensive than doing professional development,’ says Eva Baker of the Center for the Study of Evaluation at UCLA.

It is in this light that the debate about adaptive learning becomes hugely significant. Class Size Matters, the odd academic like Neil Selwyn or the occasional blogger like myself will not be able to reverse a trend with seemingly unstoppable momentum. But we are, collectively, in a position to influence the way these changes will take place.

If you want to find out more, check out the inBloom and Class Size Matters links. And you might like to read more from the news reports which I have used for information in this post. Of these, the second was originally published by Scientific American (owned by Macmillan, one of the leading players in ELT adaptive learning). The third and fourth are from Education Week, which is funded in part by the Gates Foundation.

http://www.reuters.com/article/2013/03/03/us-education-database-idUSBRE92204W20130303

http://www.salon.com/2013/08/01/big_data_puts_teachers_out_of_work_partner/

http://www.edweek.org/ew/articles/2014/01/08/15inbloom_ep.h33.html

http://blogs.edweek.org/edweek/marketplacek12/2013/12/new_york_battle_over_inBloom_data_privacy_heading_to_court.html

busuu

busuu is an online language learning service. I did not refer to it in the ‘guide’ because it does not seem to use any adaptive learning software yet, but this is set to change. According to founder Bernhard Niesner, the company is already working on incorporation of adaptive software.

A few statistics will show the significance of busuu. The site currently has over 40 million users (El Pais, 8 February 2014) and is growing by 40,000 a day. The basic service is free, but the premium service costs Euro 69.99 a year. The company will not give detailed user statistics, but say that ‘hundreds of thousands’ are paying for the premium service, that turnover was a 7-figure number last year and will rise to 8 figures this year.

It is easy to understand why traditional publishers might be worried about competition like busuu and why they are turning away from print-based courses.

Busuu offers 12 languages, but, as a translation-based service, any one of these languages can only be studied if you speak one of the other languages on offer. The levels of the different courses are tagged to the CEFR.

In some ways, busuu is not so different from competitors like duolingo. Students are presented with bilingual vocabulary sets, accompanied by pictures, which are tested in a variety of ways. As with duolingo, some of this is a little strange. For German at level A1, I did a vocabulary set on ‘pets’ which presented the German words for a ferret, a tortoise and a guinea-pig, among others. There are dialogues, which are both written and recorded, that are sometimes surreal.

Child: Mum, look over there, there’s a dog without a collar, can we take it?

Mother: No, darling, our house is too small to have a dog.

Child: Mum your bedroom is very big, it can sleep with dad and you.

Mother: Come on, I’ll buy you a toy dog.

The dialogues are followed up by multiple choice questions which test your memory of the dialogue. There are also writing exercises where you are given a picture from National Geographic and asked to write about it. It’s not always clear what one is supposed to write. What would you say about a photo that showed a large number of parachutes in the sky, beyond ‘I can see a lot of parachutes’?

There are also many gamification elements. There is a learning carrot where you can set your own learning targets and users can earn ‘busuuberries’ which can then be traded in for animations in a ‘language garden’.

But in one significant respect, busuu differs from its competitors. It combines the usual vocabulary, grammar and dialogue work with social networking. Users can interact with text or video, and feedback on written work comes from other users. My own experience with this was mixed, but the potential is clear. Feedback on other learners’ work is encouraged by the awarding of ‘busuuberries’.

We will have to wait and see what busuu does with adaptive software and what it will do with the big data it is generating. For the moment, its interest lies in illustrating what could be done with a learning platform and adaptive software. The big ELT publishers know they have a new kind of competition and, with a lot more money to invest than busuu, we have to assume that what they will launch a few years from now will do everything that busuu does, and more. Meanwhile, busuu are working on site redesign and adaptivity. They would do well, too, to sort out their syllabus!

Part 9: neo-liberalism and solutionism

The drive towards adaptive learning is being fuelled less by individual learners or teachers than it is by commercial interests, large educational institutions and even larger agencies, including national governments. How one feels about adaptive learning is likely to be shaped by one’s beliefs about how education should be managed.

Huge amounts of money are at stake. Education is ‘a global marketplace that is estimated conservatively to be worth in excess of $5 trillion per annum’ (Selwyn, Distrusting Educational Technology 2013, p.2). With an eye on this pot, in one year, 2012, ‘venture capital funds, private equity investors and transnational corporations like Pearson poured over $1.1 billion into education technology companies’[1] Knewton, just one of a number of adaptive learning companies, managed to raise $54 million before it signed multi-million dollar contracts with ELT publishers like Macmillan and Cambridge University Press. In ELT, some publishing companies are preferring to sit back and wait to see what happens. Most, however, have their sights firmly set on the earnings potential and are fully aware that late-starters may never be able to catch up with the pace-setters.

The nexus of vested interests that is driving the move towards adaptive learning is both tight and complicated. Fuller accounts of this can be found in Stephen Ball’s ‘Education Inc.’ (2012) and Joel Spring’s ‘Education Networks’ (2012) but for this post I hope that a few examples will suffice.

Leading the way is the Bill and Melinda Gates Foundation, the world’s largest private foundation with endowments of almost $40 billion. One of its activities is the ‘Adaptive Learning Market Acceleration Program’ which seeks to promote adaptive learning and claims that the adaptive learning loop can defeat the iron triangle of costs, quality and access (referred to in The Selling Points of Adaptive Learning, above). It is worth noting that this foundation has also funded Teach Plus, an organisation that has been lobbying US ‘state legislatures to eliminate protection of senior teachers during layoffs’ (Spring, 2012, p.51). It also supports the Foundation for Excellence in Education, ‘a major advocacy group for expanding online instruction by changing state laws’ (ibid., p.51). The chairman of this foundation is Jeb Bush, brother of ex-president Bush, who took the message of his foundation’s ‘Digital Learning Now!’ program on the road in 2011. The message, reports Spring (ibid. p.63) was simple: ‘the economic crises provided an opportunity to reduce school budgets by replacing teachers with online courses.’ The Foundation for Excellence in Education is also supported by the Walton Foundation (the Walmart family) and iQity, a company whose website makes clear its reasons for supporting Jeb Bush’s lobbying. ‘The iQity e-Learning Platform is the most complete solution available for the electronic search and delivery of curriculum, courses, and other learning objects. Delivering over one million courses each year, the iQity Platform is a proven success for students, teachers, school administrators, and district offices; as well as state, regional, and national education officials across the country.[2]’

Another supporter of the Foundation for Excellence in Education is the Pearson Foundation, the philanthropic arm of Pearson. The Pearson Foundation, in its turn, is supported by the Gates Foundation. In 2011, the Pearson Foundation received funding from the Gates Foundation to create 24 online courses, four of which would be distributed free and the others sold by Pearson the publishers (Spring, 2012, p.66).

The campaign to promote online adaptive learning is massively funded and extremely well-articulated. It receives support from transnational agencies such as the World Bank, WTO and OECD, and its arguments are firmly rooted in the discourse ‘of international management consultancies and education businesses’ (Ball, 2012, p.11-12). It is in this context that observers like Neil Selwyn connect the growing use of digital technologies in education to the corporatisation and globalisation of education and neo-liberal ideology.

Adaptive learning also holds rich promise for those who can profit from the huge amount of data it will generate. Jose Fereira, CEO of Knewton, acknowledges that adaptive learning has ‘the capacity to produce a tremendous amount of data, more than maybe any other industry’[3]. He continues ‘Big data is going to impact education in a big way. It is inevitable. It has already begun. If you’re part of an education organization, you need to have a vision for how you will take advantage of big data. Wait too long and you’ll wake up to find that your competitors (and the instructors that use them) have left you behind with new capabilities and insights that seem almost magical.’ Rather paradoxically, he then concludes that ‘we must all commit to the principle that the data ultimately belong to the students and the schools’. It is not easy to understand how such data can be both the property of individuals and, at the same time, be used by educational organizations to gain competitive advantage.

The existence and exploitation of this data may also raise concerns about privacy. In the same way that many people do not fully understand the extent or purpose of ‘dataveillance’ by cookies when they are browsing the internet, students cannot be expected to fully grasp the extent or potential commercial use of the data that they generate when engaged in adaptive learning programs.

Selwyn (Distrusting Educational Technology 2013, p.59-60) highlights a further problem connected with the arrival of big data. ‘Dataveillance’, he writes, also ‘functions to decrease the influence of ‘human’ experience and judgement, with it no longer seeming to matter what a teacher may personally know about a student in the face of his or her ‘dashboard’ profile and aggregated tally of positive and negative ‘events’. As such, there would seem to be little room for ‘professional’ expertise or interpersonal emotion when faced with such data. In these terms, institutional technologies could be said to be both dehumanizing and deprofessionalizing the relationships between people in an education context – be they students, teachers, administrators or managers.’

Adaptive learning in online and blended programs may well offer a number of advantages, but these will need to be weighed against the replacement or deskilling of teachers, and the growing control of big business over educational processes and content. Does adaptive learning increase the risk of transforming language teaching into a digital diploma mill (Noble, Digital Diploma Mills: The automation of higher education 2002)?

Solutionism

Evgeney Morozov’s 2013 best-seller, ‘To Save Everything, Click Here’, takes issue with our current preoccupation with finding technological solutions to complex and contentious problems. If adaptive learning is being presented as a solution, what is the problem that it is the solution of? In Morosov’s analysis, it is not an educational problem. ‘Digital technologies might be a perfect solution to some problems,’ he writes, ‘but those problems don’t include education – not if by education we mean the development of the skills to think critically about any given issue’ (Morosov, 2013, p.8). Only if we conceive of education as the transmission of bits of information (and in the case of language education as the transmission of bits of linguistic information), could adaptive learning be seen as some sort of solution to an educational problem. The push towards adaptive learning in ELT can be seen, in Morosov’s terms, as reaching ‘for the answer before the questions have been fully asked’ (ibid., p.6).

The world of education has been particularly susceptible to the dreams of a ‘technical fix’. Its history, writes Neil Selwyn, ‘has been characterised by attempts to use the ‘power’ of technology in order to solve problems that are non-technological in nature. […] This faith in the technical fix is pervasive and relentless – especially in the minds of the key interests and opinion formers of this digital age. As the co-founder of the influential Wired magazine reasoned more recently, ‘tools and technology drive us. Even if a problem has been caused by technology, the answer will always be more technology’ (Selwyn, Education in a Digital World 2013, p.36).

Morosov cautions against solutionism in all fields of human activity, pointing out that, by the time a problem is ‘solved’, it becomes something else entirely. Anyone involved in language teaching would be well-advised to identify and prioritise the problems that matter to them before jumping to the conclusion that adaptive learning is the ‘solution’. Like other technologies, it might, just possibly, ‘reproduce, perpetuate, strengthen and deepen existing patterns of social relations and structures – albeit in different forms and guises. In this respect, then, it is perhaps best to approach educational technology as a ‘problem changer’ rather than a ‘problem solver’ (Selwyn, Education in a Digital World 2013, p.21).

---

[1] Philip McRae Rebirth of the Teaching Machine through the Seduction of Data Analytics: This time it’s personal April 14, 2013 http://philmcrae.com/2/post/2013/04/rebirth-of-the-teaching-maching-through-the-seduction-of-data-analytics-this-time-its-personal1.html (last accessed 13 January 2014)

[2] http://www.iq-ity.com/ (last accessed 13 January, 2014)

[3] http://www.knewton.com/blog/ceo-jose-ferreira/big-data-in-education/ (last accessed 13 January, 2104)

Part 7: The next 4 to 5 years of adaptive learning in ELT – 10 predictions

Given what we know, it is possible to make some predictions about what the next generation of adult ELT materials will be like when they emerge a few years from now. Making predictions is always a hazardous game, but there are a number of reasonable certainties that can be identified, based on the statements and claims of the major publishers and software providers.

1 Major publishers will move gradually away from traditional coursebooks (whether in print or ebook format) towards the delivery of learning content on learning platforms. At its most limited, this will be in the form of workbook-style material with an adaptive element. At its most developed, this will be in the form of courses that can be delivered entirely without traditional coursebooks. These will allow teachers or institutions to decide the extent to which they wish to blend online and face-to-face instruction.

2 The adaptive elements of these courses will focus primarily or exclusively on discrete item grammar, vocabulary, functional language and phonology, since these lend themselves most readily to the software. These courses will be targeted mainly at lower level (B1 and below) learners.

3 The methodological approach of these courses will be significantly influenced by the expectations of the markets where they are predicted to be most popular and most profitable: South and Central America, the Arabian Gulf and Asia.

4 These courses will permit multiple modifications to suit local requirements. They will also allow additional content to be uploaded.

5 Assessment will play an important role in the design of all these courses. Things like discrete item grammar, vocabulary, functional language and phonology, which lend themselves most readily to assessment, will be prioritized over language skills, which are harder to assess.

6 The discrete items of language that are presented will be tagged to level descriptors, using scales like the Common European Framework or English Profile.

7 Language skills work will be included, but only in the more sophisticated (and better-funded) projects will these components be closely tied to the adaptive software.

8 Because of technological differences between different parts of the world, adaptive courses will co-exist with closely related, more traditional print (or ebook) courses.

9 Training for teachers (especially concerning blended learning) will become an increasingly important part of the package sold by the major publishers.

10 These courses will be more than ever driven by the publishers’ perceptions of what the market wants. There will be a concomitant decrease in the extent to which individual authors, or author teams, influence the material.