After a break of two years, I recently returned to Duolingo in an attempt to build my German vocabulary. The attempt lasted a week. A few small things had changed, but the essentials had not, and my amusement at translating sentences like The duck eats oranges, A red dog wears white clothes or The fly is important soon turned to boredom and irritation. There are better, free ways of building vocabulary in another language.
Whilst little is new in the learning experience of Duolingo, there are significant developments at the company. The first of these is a new funding round in which they raised a further $20 million, bringing total investment to close to $40 million. Duolingo now has more than 25 million users, half of whom are described as ‘active’, and, according to Louis von Ahn, the company’s founder, their ambition is to dominate the language learning market. Approaching their third anniversary, though, Duolingo will need, before long, to turn a profit or, at least, to break even. The original plan, to use the language data generated by users of the site to power a paying translation service, is beginning to bear fruit, with contracts with CNN and BuzzFeed. But Duolingo is going to need other income streams. This may well be part of the reason behind their decision to develop and launch their own test.
Duolingo’s marketing people, however, are trying to get another message across: Every year, over 30 million job seekers and students around the world are forced to take a test to prove that they know English in order to apply for a job or school. For some, these tests can cost their family an entire month’s salary. And not only that, taking them typically requires traveling to distant examination facilities and waiting weeks for the results. We believe there should be a better way. This is why today I’m proud to announce the beta release of the Duolingo Test Center, which was created to give everyone equal access to jobs and educational opportunities. Now anyone can conveniently certify their English skills from home, on their mobile device, and for only $20. That’s 1/10th the cost of existing tests. Talking the creative disruption talk, Duolingo wants to break into the “archaic” industry of language proficiency tests. Basically, then, they want to make the world a better place. I seem to have heard this kind of thing before.
The tests will cost $20. Gina Gotthilf , Duolingo’s head of marketing, explains the pricing strategy: We came up with the smallest value that works for us and that a lot of people can pay. Duolingo’s main markets are now the BRICS countries. In China, for example, 1.5 million people signed up with Duolingo in just one week in April of this year, according to @TECHINASIA . Besides China, Duolingo has expanded into India, Japan, Korea, Taiwan, Hong Kong, Vietnam and Indonesia this year. (Brazil already has 2.4 million users, and there are 1.5 million in Mexico.) That’s a lot of potential customers.
So, what do you get for your twenty bucks? Not a lot, is the short answer. The test lasts about 18 minutes. There are four sections, and adaptive software analyses the testee’s responses to determine the level of difficulty of subsequent questions. The first section requires users to select real English words from a list which includes invented words. The second is a short dictation, the third is a gapfill, and the fourth is a read-aloud task which is recorded and compared to a native-speaker norm. That’s it.
Duolingo claims that the test scores correlate very well with TOEFL, but the claim is based on a single study by a University of Pittsburgh professor that was sponsored by Duolingo. Will further studies replicate the findings? I, for one, wouldn’t bet on it, but I won’t insult your intelligence by explaining my reasons. Test validity and reliability, then, remain to be proved, but even John Lehoczky , interim executive vice president of Carnegie Mellon University (Duolingo was developed by researchers from Carnegie Mellon’s computer science department) acknowledges that at this point [the test] is not a fit vehicle for undergraduate admissions.
Even more of a problem than validity and reliability, however, is the question of security. The test is delivered via the web or smartphone apps (Android and iOS). Testees have to provide photo ID and a photo taken on the device they are using. There are various rules (they must be alone, no headphones, etc) and a human proctor reviews the test after it has been completed. This is unlikely to impress authorities like the British immigration authorities, which recently refused to recognise online TOEFL and TOEIC qualifications, after a BBC documentary revealed ‘systematic fraud’ in the taking of these tests.
There will always be a market of sorts for valueless qualifications (think, for example, of all the cheap TEFL courses that can be taken online), but to break into the monopoly of TOEFL and IELTS (and soon perhaps Pearson), Duolingo will need to deal with the issues of validity, reliability and security. If they don’t, few – if any – institutions of higher education will recognise the test. But if they do, they’ll need to spend more money: a team of applied linguists with expertise in testing would be a good start, and serious proctoring doesn’t come cheap. Will they be able to do this and keep the price down to $20?
Adaptive Learning in ELT 
In my context (Kazakhstan) exams like IELTS are extremely expensive. Even more for those who live outside the major cities because they have to travel and stay two nights (the speaking is on a Thursday, the rest of it on a Saturday morning). There are also the preparation classes and materials to buy. Cambridge, the British Council and the franchise holders make an awful lot of money out of it.
What Duolingo are trying to do with their exam is appealing because if they are successful it will have a transformational effect on access to opportunity in places like Kazakhstan. Clearly though, they have a long way to go. Your comment about them needing to employ some language professionals could, unfortunately, be said about most (if not all) ed-tech companies involved in language learning.
In terms of security, perhaps some sort of hybrid approach between mobile and fixed (i.e. IELTS and TOEFL) could work? Universities and schools could organise exams at a local level. A group of say 15 max students could all sit in a room and do an exam on their tablets with remote software checking for signs of cheating. The university/school would be responsible for creating exam conditions. Security could be enhanced by the obligatory use of webcams (which could swivel and zoom in and out) with microphones in the room, microphones on each individual device and the use of a live remote proctor. A similar set-up could be used to test speaking with students walking into the room one-at-a-time. This solution would be much less labour intensive than current standardised exams so greatly reducing cost, it could also be administered anywhere in the world with a decent internet connection, but would retain the tight security of the current exams.The individual university or school could pay a flat fee for its students to take the test knowing that any problems would invalidate every student’s test. That would ensure an incentive against colluding to cheat.
Hi Thomas,
I wouldn’t disagree with anything you say. There’s a nice blog post by Jennifer MacDonald (http://jenmacdonald.wordpress.com/2014/05/01/duolingo-enters-the-testing-business/) which sums up in a way that I’d go along with:
‘Don’t get me wrong; I’m no fan of the big business of the TOEFL/IELTS testing industrial complex and the way it has pulled the focus of much EAP teaching, for example, away from real language development toward acing the standardized exam. But the research and development that have gone into TOEFL and IELTS over the years is phenomenal, and as evil and profit-driven as they may be, they test reading, writing, speaking and listening in a generally valid and reliable manner. (Not without major limitations and heaps of criticism, of course! But nonetheless…)’
For twice the price ($39.95) you can take Pearson’s Versant test (once called PhonePass), which tests you on four distinct bands of competence, including spoken fluency, and claims to have been validated by a number of research studies (https://www.versanttest.com/technology/validation.jsp) . I have to say that its Spanish version was fairly impressive, given the ease with which it can be taken (20 minutes or so by phone), but there is no interactive element, nor, obviously, any measure of reading or writing skills. I hate to be endorsing Pearson, but all I’m saying is that Duolingo are going to have to work very hard to catch up.
You have to wonder if the catching up will be possible with an app-based business model. It certainly allows quick and regular updates, but verification of validity and reliability takes time. Which means that every time the test is changed, new research is needed. Pearson, ETS and IELTS can swiftly bring their research people into gear, but Duolingo simply doesn’t have this kind of capability.
This is a very good analysis. You are right that security is the critical issue here. With all the protocols in place, we still get incidents of students who did not take IELTS/TOEFL themselves. Will sure be an uphill battle. At http://www.dailythemes.org, we are telling students that language (in particular writing) is a life long skill and should not be learnt solely for the purpose of passing tests.
I find it frightening how our profession is being subverted by software geeks who obviously have no idea of how languages are learned – as if the last 50 years of research and practice in ELT did not exist.
Hahaha Jill -do those in our industry have any idea how languages are learnt? 😉