WEBVTT 00:00:00.001 --> 00:00:06.057 [Dorothy Bishop] I'm going to talk today about evaluating alternative intervention approaches to dyslexia. 00:00:06.059 --> 00:00:07.462 [slide with talk title] 00:00:07.463 --> 00:00:13.667 The conventional approaches that you get really tend to rely on the fact that we've now got of evidence 00:00:13.668 --> 00:00:18.409 that most children with dyslexia have problems in what is called phonological awareness, 00:00:18.410 --> 00:00:23.440 that is, they don't necessarily hear all the different sounds in speech, 00:00:23.441 --> 00:00:28.380 and therefore have difficulty relating them to letters when they are trying to read. 00:00:28.381 --> 00:00:37.516 And most of the interventions that are mainstream these days would focus on trying to train children to identify sounds in words and relate them to letters. 00:00:38.971 --> 00:00:45.414 But this sort of intervention has been shown to be effective, and there have been a number of large scale studies. 00:00:45.415 --> 00:00:51.725 But nevertheless, it has to be fairly prolonged for some children, and there are children for whom, 00:00:51.725 --> 00:00:57.870 even though they can learn this way to actually sound out words and read, they don't necessarily read fluently. 00:00:57.912 --> 00:01:03.306 It's still an effort for them, and they don't sort of get to the degree of automaticity that you might expect, 00:01:03.307 --> 00:01:07.683 And it's certainly the case that methods that work for many children don't work for all children, 00:01:07.684 --> 00:01:10.756 and there is a hard core of children who remain very hard to treat. 00:01:10.757 --> 00:01:15.943 It is for this reason that many parents do get very concerned about whether there is something else they should be doing 00:01:15.944 --> 00:01:21.997 if they are finding that their child is either not getting intervention, or that the intervention doesn't seem to be working very well. 00:01:21.998 --> 00:01:29.446 And there are a whole load of things out there that are on offer, and the problem for the parents, I think - 00:01:29.447 --> 00:01:33.982 and/or indeed for adults who themselves, might want to have further intervention for dyslexia - 00:01:33.983 --> 00:01:37.691 is that they want to know, "how do I distinguish something that might work for me", 00:01:37.691 --> 00:01:41.641 from something that is just some sort of snake-oil merchant who is out there to make money. 00:01:41.642 --> 00:01:44.482 And that's what I want to try and address today. 00:01:44.482 --> 00:01:51.544 principally from the perspective of how you might evaluate scientific evidence that people put forward. 00:01:51.544 --> 00:01:57.486 But perhaps before going onto that, it's worth going into some relatively commonsense things. 00:01:57.618 --> 00:02:06.612 I would say that there are certain things that should ring alarm bells if people are advertising some sort of treatment for child dyslexia. 00:02:06.649 --> 00:02:13.064 The first thing is if the intervention has been developped by somebody who has no academic track record, 00:02:13.065 --> 00:02:17.834 no experience of doing research in this field, and hasn't published anything in this field, 00:02:17.834 --> 00:02:24.541 if the intervention isn't endorsed by people in the mainstream dyslexia field, 00:02:24.542 --> 00:02:27.633 that should also sound a note of caution. 00:02:27.634 --> 00:02:29.820 Of course, the mainstream people aren't always right. 00:02:29.821 --> 00:02:32.971 It's possible that somebody with no background will develop something marvelous. 00:02:32.972 --> 00:02:37.474 But if that were the case, you would expect it to be pretty quickly picked up by people in the mainstream, 00:02:37.475 --> 00:02:40.798 who are really, on the whole, pretty keen to find things that will work. 00:02:40.798 --> 00:02:47.811 And you obviously want to look at whether somebody is asking for a lot of money for something that hasn't been proven. 00:02:47.812 --> 00:02:56.216 And what is also, to my mind, a worrying sign, is if somebody promoting a treatment is relying heavily just on testimonials 00:02:56.217 --> 00:03:04.154 from individuals who claimed to have been cured, rather than having any sort of proper scientific evaluation or kind of controls. 00:03:04.155 --> 00:03:10.613 And it's worth noting that human beings have a tendency to be terribly impressed by testimonials, 00:03:10.613 --> 00:03:16.749 and even myself, as somebody with a scientific training, I find if, you know, I've got headaches and somebody comes along and says: 00:03:16.750 --> 00:03:20.603 "I was cured by such and such, and I went to my herbalist and it worked", 00:03:20.603 --> 00:03:27.094 you know, you're often very tempted to be much more swayed by that sort of evidence than by a pagefull of numbers and figures. 00:03:27.095 --> 00:03:34.677 And this just a human tendency: we are naturally built to really take advice from other people and to rely on what they tell us. 00:03:34.677 --> 00:03:38.494 But in the contexts of these sorts of interventions, that's really quite dangerous, 00:03:38.494 --> 00:03:44.271 because, when somebody gives a testimonial, that's just one person, their only individual experience, 00:03:44.272 --> 00:03:48.273 And people you don't hear from tend to be the people who tried it, and it didn't work. 00:03:48.274 --> 00:03:51.881 And you don't know how many of them there are: there may be thousands of them. 00:03:51.882 --> 00:03:56.049 But they're not going to publicize the fact that they tried it and it didn't work. 00:03:56.050 --> 00:04:01.945 And so, testimonials are often very much at odds with more scientific evaluations. 00:04:01.945 --> 00:04:10.096 .... to turn out that when somebody says there is scientific evidence for what they're doing, how you should interpret that. 00:04:10.097 --> 00:04:14.964 And that's jolly difficult even for scientists sometimes: there is disagreement - let alone for the general public. 00:04:14.965 --> 00:04:20.727 But again, I think, there are some sort of general rules of thumb that you can go by 00:04:20.728 --> 00:04:24.020 for telling that a treatment is likely to be effective. 00:04:25.450 --> 00:04:38.229 And when I discuss this, I'm going to illustrate it by taking the example of the Dore treatment - that's DORE, named after Wynford Dore, it's originator. 00:04:38.230 --> 00:04:46.360 ANd I'm picking on this largely because it is a non-mainstream treatment that isn't widely accepted by the experts, 00:04:46.361 --> 00:04:49.899 and yet it does claim that there is some scientific evidence to support it, 00:04:49.900 --> 00:04:54.277 which has lead the scientists to look at it quite critically and quite carefully, 00:04:54.277 --> 00:04:57.654 which is what we would do with any scientific evidence that comes along: 00:04:57.655 --> 00:05:04.834 once it's out in the public domain and published, people tend to go and look at it as carefully as they possibly can. 00:05:04.834 --> 00:05:11.944 Now, the Dore method is interesting to us, because it does illustrate the case where there is disagreement 00:05:11.945 --> 00:05:15.197 as to whether the evidence is showing that its' effective or not. 00:05:15.198 --> 00:05:20.587 And so, what I want to explain is why it is the case that despite this published evidence, 00:05:20.588 --> 00:05:25.324 most of the experts are not impressed of the efficacy of the Dore treatment. 00:05:25.325 --> 00:05:32.227 But the general points that I'll make would apply to any other treatment that was out there, whether (?) there was evidence being produced. 00:05:32.751 --> 00:05:39.961 So, first of all, what is the Dore method? Well, it's a method that has been proposed for curing problems 00:05:39.962 --> 00:05:44.774 that are thought to originate in the part of the brain called the cerebellum, which is at the back of the brain, 00:05:44.774 --> 00:05:49.999 and it was developed by Wynford Dore as a method for helping his dyslexic daughter. 00:05:49.999 --> 00:05:53.578 He has written a book about the history of how this came to became (?) about, 00:05:53.579 --> 00:05:58.973 and he was a classic instance of a parent who was rather desperate to help his daughter who, for many years, 00:05:58.973 --> 00:06:03.513 had been through the educational system and failed, and was getting increasingly depressed. 00:06:03.514 --> 00:06:10.842 And he tried various things, he talked to various experts, and he ended up with this program that's been put forward, 00:06:10.842 --> 00:06:15.541 which is an individualized program, where the child follows various sorts of exercises, 00:06:15.542 --> 00:06:21.019 which are done for about ten minutes twice a day, over quite a long period of time, 00:06:21.019 --> 00:06:25.265 varying, depending on the severity of the problem, from maybe 6 months to 2 years. 00:06:25.928 --> 00:06:30.984 And the child is assessed at regular intervals and different exercises may be prescribed. 00:06:30.985 --> 00:06:38.550 Now, the theory behind the Dore method is that dyslexia and other learning difficulties - 00:06:38.550 --> 00:06:43.661 it's not just dyslexia it claims to help, but also Attention Deficit problems ...(?) hyperactive - 00:06:43.661 --> 00:06:49.043 are thought to arise within the cerebellum: the cerebellum just doesn't develop normally, 00:06:49.043 --> 00:06:54.474 and the argument is that you can have different cerebellar impediments in different people, 00:06:54.475 --> 00:06:57.198 and that's why you can get this range of different symptoms, 00:06:57.199 --> 00:07:04.061 but that you can diagnose them by specific tests of test of mental and physical coordinations. 00:07:04.062 --> 00:07:12.679 And what you are then supposed to do is these exercises, which are not anywhere fully described in the public domain, 00:07:12.679 --> 00:07:17.623 because they are commercially sensitive, but there are some examples given, and it's clear that what they do 00:07:17.623 --> 00:07:22.714 is focused largely on training balance and hand-eye coordination in children. 00:07:22.715 --> 00:07:29.890 So you might be asked to stand on a cushion on one leg, or to throw a bean bag from one hand to another 00:07:29.891 --> 00:07:37.296 while you are doing that, just stand on a wobble board (?) and balance, or to follow something with your eyes in a particular way. 00:07:39.188 --> 00:07:48.267 So, the idea is that these are all things that the cerebellum is involved in, by training up the cerebellum, you may improve its general abilities. 00:07:48.727 --> 00:07:53.435 So, what is the evidence for this underlying theory? 00:07:53.436 --> 00:07:58.405 Well, it's not a proven theory, but there is some support for it. 00:07:58.405 --> 00:08:06.920 Certainly, people trying to look at what is going on in the brain in dyslexia have proposed many different theories 00:08:06.921 --> 00:08:09.314 about what the underlying causes might be. 00:08:09.315 --> 00:08:14.735 If you look at the brain in a brain scanner of somebody with dyslexia, it typically looks totally normal. 00:08:14.735 --> 00:08:19.256 There's certainly no big holes in the head or anything like that, that you are going to see on a scanner. 00:08:19.257 --> 00:08:24.865 But the argument is being made that there may be regions of the cerebellum that are perhaps slightly smaller than they should be 00:08:24.866 --> 00:08:28.046 or not functioning quite as they should be. 00:08:28.047 --> 00:08:33.820 And this theory has some support, although not everybody would agree with it 00:08:33.820 --> 00:08:38.843 and there is certainly other theories equally plausible at the moment that are around. 00:08:39.444 --> 00:08:44.664 The notion - the cerebellum is important for getting things automated. 00:08:44.665 --> 00:08:50.675 So you can - when you learn to drive a car, first of all, it's very slow and effortfull, and you have to think about everything you do. 00:08:50.675 --> 00:08:55.185 By the time you are a skilled driver, it's no longe the case that you have to do that, 00:08:55.186 --> 00:08:58.061 you just drive around without thinking about it. 00:08:58.061 --> 00:09:00.063 You can do all sorts of other things while you are driving. 00:09:00.064 --> 00:09:07.321 So, the argument is that with reading, most people, similarly, become very automatic in how they learn to read: 00:09:07.322 --> 00:09:14.902 you do it without thinking about it, but for the dylexic it remains effortfull because the cerebellum is not functioning normally 00:09:14.902 --> 00:09:19.387 and it's the cerebellum that helps you get your skills automatized. 00:09:19.388 --> 00:09:26.874 And in support of this, it has been argued that in many people with dyslexia, there are some associated problems with motor coordination, 00:09:26.875 --> 00:09:32.792 ..... (?) physical skills and so on, and that too could be a sign of a problem with the cerebellum. 00:09:32.793 --> 00:09:39.876 Again, that's fairly controversial, it's not being found in all children, and the arguments go to and fro. 00:09:39.876 --> 00:09:47.492 But this is not a sort of theory that is particularly disapproved of by the mainstream. People are debating it. 00:09:47.493 --> 00:09:55.052 The difficult stumbling block, though, for the Dore approach to treatment comes with the idea that 00:09:55.053 --> 00:10:02.531 if you train the motor skills, that is a sort of coordination between different muscles and movements 00:10:02.532 --> 00:10:09.374 and between their eyes and hands, that this will somehow have a knock-on effect on things like reading. 00:10:09.375 --> 00:10:16.401 And indeed, David Randall and colleagues, who published this initial study on the treatment, 00:10:16.401 --> 00:10:23.410 describe it as something of a leap of faith, because the cerebellum is actually known to be a very complicated organ, 00:10:23.411 --> 00:10:27.249 with lots of different regions, which are fairly independent from one another. 00:10:27.250 --> 00:10:34.590 So there is no real reason to suppose that if you train one part of the cerebellum, it will have somehow a generalized benefit. 00:10:34.591 --> 00:10:40.750 And indeed, you could say: "Well, if it were the case that this is true, if you'd had a chance, you would go to skateboarding, 00:10:40.751 --> 00:10:48.710 or playing ping-pong, or things like that, ....... (?) or perhaps ballet dancing, things that require balance and coordination, 00:10:48.711 --> 00:10:52.653 that should protect you against dyslexia". There is really not much evidence for that, 00:10:52.654 --> 00:10:57.645 on the contrary, there is some very good sportsmen who - gymnasts and people with dyslexia. 00:10:57.645 --> 00:11:06.790 So it is hard to see how the logic of saying "Train these motor skills and somehow the whole cerebellum function some day improves2 00:11:06.790 --> 00:11:12.876 But what does the published evidence look like? Because the theory might be, you know, questionable, 00:11:12.877 --> 00:11:16.956 but basically, what the parents are going to say is, "What matter is, does it work?" 00:11:18.215 --> 00:11:24.717 Well, there is a published study on the intervention, which claims that it shows that it really does work 00:11:24.717 --> 00:11:28.297 if you compare children who have the intervention and children who don't. 00:11:28.298 --> 00:11:35.999 And two papers have been reported - one from the initial phase of the study, and the other from a subsequent phase - 00:11:36.866 --> 00:11:45.538 And they are reported in the Journal of Dyslexia which, in 2003, published the first paper 00:11:45.663 --> 00:11:53.890 which was on just under - started with a sample statistics on the 300 children who were all attending a .... (?) primary school. 00:11:53.890 --> 00:11:58.021 And the researchers went in and screened all the children on the dyslexia screening test, 00:11:58.021 --> 00:12:01.457 to pull out children who would be suitable for enrolment in the study. 00:12:01.458 --> 00:12:10.647 But the first thing that is more important to note is that these were not children who had a very high rate of diagnosis of dyslexia. 00:12:10.648 --> 00:12:19.265 So, there were 35 in the group, and about a third of those came out as having a strong risk of dyslexia on this dyslexia screening test. 00:12:19.265 --> 00:12:26.744 Another 21% came out with a mild risk, but about half of these children were not really ...... (?) in this category 00:12:26.745 --> 00:12:31.496 and they were just picked because their schools (?) were relatively lower compared to the other children. 00:12:31.497 --> 00:12:38.199 And there were only a total of 6 children who had previously been diagnosed with dyslexia, out of the 35. 00:12:38.200 --> 00:12:42.989 There were a couple with a diagnosis of dyspraxia and one with ADHD diagnosis. 00:12:42.990 --> 00:12:48.464 So this is not really a sample consisting of children really with severe problems on the whole. 00:12:48.464 --> 00:12:50.660 There were few in there with major difficulties. 00:12:50.661 --> 00:12:58.037 Nevertheless, the originators of the treatment would argue even quite mild problems might be worth treating with this 00:12:58.037 --> 00:13:01.890 and so you could argue this study is nevertheless of value. 00:13:01.891 --> 00:13:07.834 So what they did, they started out well in this study: they divided the children randomly in treated and untreated groups, 00:13:07.834 --> 00:13:13.687 which is, as I am going on to explain later, is an important part of a good study. 00:13:13.687 --> 00:13:20.727 And if you look at the results that are described on the promotion materials of the DORE organization, 00:13:20.727 --> 00:13:28.476 they are all in Dore's book that he published, "Dyslexia, the miracle cure", he described the results as stunning 00:13:28.477 --> 00:13:35.039 and said that reading age increased threefold, comprehension age increased almost fivefold 00:13:35.040 --> 00:13:40.640 and writing skills by what he described as "an extraordinary 17-fold". 00:13:40.641 --> 00:13:47.856 Of course, everybody reading that think "Wow, my child is going to take off like a rocket if we put him on this intervention." 00:13:47.856 --> 00:13:56.806 But unfortunately, these figures are really a classic instance of how statistics can be manipulated in a very misleading way. 00:13:57.414 --> 00:14:04.912 So, for a start, they were not based on any comparison between the control children and the untreated children - 00:14:04.913 --> 00:14:09.177 sorry, the control children and the treated children. 00:14:09.178 --> 00:14:17.568 They were - instead, they just took all the children who would be treated and looked at how they did on a group reading test 00:14:17.569 --> 00:14:21.044 that had been administered by the school every year. 00:14:21.045 --> 00:14:30.727 And the children had had this on two occasions prior to the intervention - so, 3 months before it started and a year before that - 00:14:30.727 --> 00:14:35.345 and on two occasions after the intervention, after this whole long 4-year period. 00:14:35.346 --> 00:14:44.263 And what the researchers did was to really just plot the average schools of the group over these 4 time periods 00:14:44.264 --> 00:14:49.639 and show that if you compared the amount of change from the first time point to the second, 00:14:49.640 --> 00:14:54.394 which was before they had had any treatment, it was a certain amount 00:14:54.395 --> 00:15:00.347 and if you then compared the second to the third time point, so the treatment had been going on (?) between those two, 00:15:00.347 --> 00:15:02.663 there was a different amount of change. 00:15:02.664 --> 00:15:06.672 And then they divided one by the other and showed that there was this threefold improvement. 00:15:06.673 --> 00:15:13.301 But it's a very, very misleading way of depicting these data, because if you look at them on a graph, here, 00:15:13.301 --> 00:15:18.168 you can see that the only odd thing about the data - well, there's two odd things about the data: 00:15:18.169 --> 00:15:23.171 one is that at most time points, these children are reading at absolutely normal levels. 00:15:23.172 --> 00:15:27.214 So it's not clear why they are regarded as having risk for dyslexia; 00:15:27.214 --> 00:15:32.947 and the one time point when they're not, is the time point 3 months before they are involved in the study, 00:15:32.947 --> 00:15:39.363 where there is a bit of a drop. But it's really not an impressive demonstration of change 00:15:39.364 --> 00:15:46.794 and this division of one time period by another is very misleading, because it just gives double weighting 00:15:46.794 --> 00:15:52.357 to this one low period of three months before the treatment started. 00:15:52.358 --> 00:15:57.610 And they did the same thing again with these other figures of massive increases that they talk about, 00:15:57.611 --> 00:16:06.830 using data from the SATS tests administered by teachers, which are not really regarded as particularly precise or rigorous tests, 00:16:06.831 --> 00:16:14.025 and really group children in a fairly global way at level 2, 3 or 4. 00:16:14.026 --> 00:16:20.494 Level 2 is average for 7 year old, 3 is average for 9 year old, and 4 is average for an 11 year old. 00:16:20.495 --> 00:16:27.876 And to give you an idea of the sort of misleading nature of these massive changes they talk about, 00:16:27.876 --> 00:16:34.125 on the writing test, where there is this incredible change that they talk about, of a 17-fold increase, 00:16:34.126 --> 00:16:40.792 the score at age 8, the average score was 2.5, which is about what you'd expect from a 8 year old. 00:16:40.792 --> 00:16:47.103 At age 9, it was 2.56, which is a little bit better, but not much. 00:16:47.104 --> 00:16:53.486 And then, they argue, the intervention came in, and at age 10, the children scored 2.95. 00:16:53.486 --> 00:16:56.888 They are still rather below where they ought to be at the age of 10. 00:16:56.888 --> 00:17:01.372 It looks as if on this particular writing assessment, the children were just rather creeping along. 00:17:01.373 --> 00:17:09.878 But because the difference between 2.53 and 2.56 is less than the difference between 2.56 and 2.95, 00:17:09.879 --> 00:17:17.935 they make a big computation of dividing one by the other, actually coming out with the number 17, which is a wrong number (?): it's actually 13. 00:17:17.935 --> 00:17:24.646 So there is a 13-fold change. But if you look at the overall numbers, this is really not so an impressive game at all. 00:17:24.647 --> 00:17:27.761 It's really a very misleading way of presenting the numbers. 00:17:27.762 --> 00:17:36.509 So, most people would say, this is really smoke in mirrors in terms of using statistics in a way that isn't really valid. 00:17:36.925 --> 00:17:42.129 The other thing that is of notice is that all these results that have given so much publicity 00:17:42.130 --> 00:17:47.697 in promoting the treatment about these massive changes, haven't talked about the control group at all. 00:17:47.697 --> 00:17:51.428 They've just talked about, "Well, we've got these children, before treatment they did this, 00:17:51.428 --> 00:17:53.800 and after treatment they did that, and it has all gone up". 00:17:53.801 --> 00:17:59.717 And of course, if schools do go up after treatment, it's not necessarily because the treatment works: 00:17:59.717 --> 00:18:02.299 There are lots of other reasons you need to bear in mind. 00:18:02.951 --> 00:18:07.555 And the first of which is just, on some things, you get better because you get older, 00:18:07.556 --> 00:18:13.635 so that if you were to measure shoe sizes before the DORE treatment and after it, it would go up, 00:18:13.636 --> 00:18:16.663 but it wouldn't mean that it made your feet grow bigger. 00:18:16.664 --> 00:18:24.339 Now, clearly, that's a silly example in most cases, because people try to use measures that don't necessarily change with age, 00:18:24.340 --> 00:18:26.643 or that are adjusted in some way for age. 00:18:26.644 --> 00:18:31.742 But it's important to bear that in mind when people are talking about changes on things like - 00:18:31.743 --> 00:18:38.029 the DORE program, they talk about changes on balance, balance improves dramatically after the program. 00:18:38.029 --> 00:18:43.498 These are measures that have not been adjusted for age at all, and so, some of these changes could well be due to the fact 00:18:43.499 --> 00:18:48.309 that the children are getting older and getting better at doing these things because of that. 00:18:48.310 --> 00:18:56.125 Another uninteresting reason why schools may improve is that the children may be having some other sort of special help. 00:18:56.126 --> 00:19:03.315 So, if a child is having reading difficulty, they may very well be getting some special help at the school, in addition to following this program. 00:19:03.315 --> 00:19:10.043 And that may be what's causing the change, rather than the particular intervention you are interested in. 00:19:10.044 --> 00:19:14.869 What's very well known, of course, is the placebo effect, which is a sort of concept coming from medicine, 00:19:14.870 --> 00:19:20.641 which also says that you can get better just because you think you are going to get better, because you think somebody has done something effective. 00:19:20.642 --> 00:19:25.611 And in the case of educational treatments, you can see effects where - 00:19:25.612 --> 00:19:31.881 because the teachers and the parents and the children themselves are all full of expectations of how this is going to improve them - 00:19:31.882 --> 00:19:38.911 there is more motivation: everybody gets positive attention and this itself can cause positive effects. 00:19:40.150 --> 00:19:45.830 The fourth reason, which is often neglected, because it really doesn't affect things in medicine so much, 00:19:45.831 --> 00:19:50.071 but in education, it's actually rather important, using the sort of thing like reading tests: 00:19:50.072 --> 00:19:55.595 you can have practice effects. So you can get better upon some things, just because you've done it before. 00:19:55.596 --> 00:20:00.521 And we've seen this quite a lot with language tests, for example, that we give to children, 00:20:00.522 --> 00:20:05.695 where, the first time you test a child, they don't know what to expect, they don't know what's coming, 00:20:05.696 --> 00:20:09.607 you aske them to do something that's unfamiliar and they are a little bit nervous, maybe. 00:20:09.607 --> 00:20:16.122 You test them again on the same thing a month later: they are much, much better, simply because they've done it before 00:20:16.122 --> 00:20:19.894 and they are calmer about it, they know what to expect, and so on. 00:20:19.894 --> 00:20:26.122 So you can get practice effects that can make quite a difference, just because you know what to expect 00:20:26.122 --> 00:20:30.402 and you are familiar with the whole situation of the test. 00:20:30.403 --> 00:20:37.856 The fifth reason - and the last one, you'll be pleased to hear - why people may improve for no good reason 00:20:37.857 --> 00:20:42.753 is the hardest to explain and it's something known as regression to the mean, 00:20:42.754 --> 00:20:49.400 and it's just a statistical artefact, which has to do with, if you pick somebody because they're bad at something, 00:20:49.401 --> 00:20:54.259 the odds are, when you test them on a second occasion, they'll be a little bit better (?). 00:20:54.260 --> 00:20:59.498 The converse is also true: if you pick somebody who is very good, they tend to get a little bit worse when you test them a second time. 00:20:59.499 --> 00:21:08.743 Why should that be? The reason why this occurs is because our measures are not entirely perfect an accurate - 00:21:08.744 --> 00:21:13.444 I'm showing a graph here, where we have a measure that is almost perfect, 00:21:13.445 --> 00:21:21.953 and you test people on two occasions, and you just will see that their scores on time 1 and time 2 are identical: we are assuming that there is no genune change. 00:21:21.954 --> 00:21:26.963 If you do that, then you don't get regression to the mean, because the measure is perfect 00:21:26.964 --> 00:21:31.039 and if you test them a second time, they'll get exactly the same sort of score. 00:21:31.040 --> 00:21:39.782 And what you can see on the right hand side of the graph here, is people divided up according to the average score they started with. 00:21:39.782 --> 00:21:45.397 So we've put people into groups who were very poor to start with, who were medium, less good and so on. 00:21:45.397 --> 00:21:49.491 And these are just fictitious data made up to illustrate the point. 00:21:49.492 --> 00:21:56.420 So you just generate these numbers by saying, "We've got a measure that has this particular characteristic 00:21:56.420 --> 00:22:01.095 that if you measure on one occasion, on another occasion it remains pretty much the same". 00:22:01.096 --> 00:22:06.777 So then, you don't get regression to mean and you get people to maintain their position across time. 00:22:06.778 --> 00:22:10.011 So if you then see change, you can say "Well, it's genuine change." 00:22:10.011 --> 00:22:15.591 But most of our measures are not like that, most of our measures are not perfectly correlated: 00:22:15.591 --> 00:22:21.641 that means, you measure them on one time, and another time, and they actually change because of all sorts of things. 00:22:21.641 --> 00:22:29.725 Things like the particular test items that you're using, whether you are in a good mood, whether you've made a lucky guess in some items. 00:22:29.726 --> 00:22:36.125 And what you can see is that if you do that, that some people's scores go up with time, some people's go down with time. 00:22:36.126 --> 00:22:44.983 But on average, if you start with a low score, the odds are, you come a little bit closer to the average when you are tested on another occasion. 00:22:44.983 --> 00:22:47.747 If you start with a high score, you get a little bit worse. 00:22:47.748 --> 00:22:53.566 And this is nothing to do with genuine change: it's just to do with the fact that our measures are imperfect. 00:22:53.567 --> 00:22:59.977 And it has been argued that this is a major reason - all sorts of treatments that work (?) but don't really work. 00:22:59.978 --> 00:23:04.401 It's just that it looks as if you've seen a change, and you tend to attribute it to the treatment. 00:23:04.401 --> 00:23:10.252 Now, this sounded very depressing, because it means there's all sorts of reasons why we can see change, 00:23:10.253 --> 00:23:14.912 and how do we distinguish whether we've got a genuine change due to our treatment? 00:23:14.912 --> 00:23:21.473 But the fact is that you can control for most of these things if you do a study that has a control group. 00:23:21.474 --> 00:23:30.969 That's why those who are trying to do scientific evaluations are really keen to include control groups in studies and argue that they are essential. 00:23:30.970 --> 00:23:37.428 Because if you have another group of children who have been selected to be as similar as possible to your treated group, 00:23:37.428 --> 00:23:42.122 and are the same tests before and after the period where the treated group are treated, 00:23:42.122 --> 00:23:48.540 you are actually controlling for the effects of maturation, the effects of any other intervention they might be having, 00:23:48.541 --> 00:23:53.250 practice effects in particular, and also this dreadful regression to the mean. 00:23:53.251 --> 00:23:56.351 All of those things can be then taken into account. 00:23:56.351 --> 00:24:02.283 And in so far as they have effects, what you would expect to see is that you may see improvement in your control group 00:24:03.212 --> 00:24:06.880 because of these spurious things that we don't really want to see. 00:24:06.880 --> 00:24:13.156 And then you can say, "well there is more improvement in the treated group" (?) and it is that difference that is really critical. 00:24:13.157 --> 00:24:20.931 It doesn't actually control to use - if you have a group who have not been given any treatment - it doesn't control for placebo effects. 00:24:20.931 --> 00:24:27.925 So you've still got the problem that maybe your treated group will improve just because everybody is focusing on them with great excitement. 00:24:27.925 --> 00:24:32.778 But you could actually also have control for that, and it's becoming increasingly popular in this field 00:24:32.779 --> 00:24:38.341 to say that what you should have is a control group who are actually given some alternative treatment. 00:24:38.342 --> 00:24:42.780 So, for example, if you are interested in a treatment that might improve reading, 00:24:42.780 --> 00:24:48.264 you could either get children some standard educational treatment that they are getting anyway 00:24:48.265 --> 00:24:53.582 So if your claim is that you are doing better than a phonological-based treatment, 00:24:53.582 --> 00:24:58.205 you could have a control group given that treatment and see if you are making really that much difference, 00:24:58.206 --> 00:25:03.407 or you might prefer to say, "Well, let's treat something else, let's give children training in something completely different 00:25:03.407 --> 00:25:06.862 that isn't focused on reading, but nevertheless could benefit them in other ways." 00:25:06.863 --> 00:25:10.104 And then you can do that sort of comparison. 00:25:10.105 --> 00:25:18.027 So what about the DORE study, because I mentioned at the outset, when talking about this study, that they did have a control group. 00:25:18.027 --> 00:25:23.981 But so far, talking about the results, are only mentioned (?) the dramatic changes that they saw, 00:25:23.982 --> 00:25:27.122 which ignored the control group. 00:25:27.123 --> 00:25:34.593 The interesting thing is that when you look at their control group, it illustrates perfectly the importance of having a control group. 00:25:34.593 --> 00:25:39.806 So, on they dylexia risk's score, where a high score is bad, 00:25:39.806 --> 00:25:47.228 they had a change in the treated group, from 0.74 to 0.34. 00:25:47.228 --> 00:25:51.060 So you think: "Wow, that's great, these children's risks for dyslexia have really come down." 00:25:51.061 --> 00:25:56.440 In the control group, the average score changed from 0.72 to 0.44. 00:25:56.440 --> 00:25:59.037 Now, you could say: "Well, it's not so big a change." 00:25:59.038 --> 00:26:03.849 The trouble is, with groups this size, you can't really tell whether that's meaningful. 00:26:03.850 --> 00:26:07.967 But certainly, what is clear is that both groups improved on the dyslexia screening test, 00:26:07.968 --> 00:26:12.089 even though the control group had not had the intervention. 00:26:12.090 --> 00:26:16.607 So, it really illustrates the point very clearly that on a lot of these measures, 00:26:16.608 --> 00:26:21.043 everybody gets better, even if they are not treated. 00:26:21.690 --> 00:26:30.834 Now, if we look at the more precise data that they presented, they presented average scores on the different subtests from the dyslexia screening tests, 00:26:30.835 --> 00:26:36.090 I won't talk about all of them, I have got a fuller presentation 00:26:36.091 --> 00:26:39.408 where I do talk about all the different measures they use 00:26:39.409 --> 00:26:42.662 and I don't want to sort of be accused of delberately hiding things, 00:26:42.663 --> 00:26:46.291 but I think the tests that people would be most interested in are the literacy tests. 00:26:46.291 --> 00:26:52.752 So, you undertake the DORE treatment because you want to get better at reading and writing, if you are a parent of a dyslexic child, at any rate. 00:26:52.752 --> 00:27:03.100 So, looking at the results on those tests, what they found was that there were a total of 4 tests that had to do with literacy directly. 00:27:03.101 --> 00:27:07.784 And on one of those, it looked as if the treated group did better than the untreated group. 00:27:07.785 --> 00:27:13.853 But there is a problem with that, though, because on this reading test, the untreated - 00:27:13.854 --> 00:27:20.498 the control group are actually right on the average score for their age at the start of the treatment - at the start of the study. 00:27:20.498 --> 00:27:27.497 So, in a sense, you could argue, "Is there really room for improvement ...... (?) school absolutely average, 00:27:27.498 --> 00:27:35.425 whereas it just so happened that the children who had treatment started a little bit lower and therefore had more improvement. 00:27:35.426 --> 00:27:38.544 And their improvement was not dramatic, one has to say as well. 00:27:38.544 --> 00:27:43.222 Their school went up from 3 to 3.5, on a scale of 0 to 10. 00:27:43.222 --> 00:27:51.733 On the other measures, again it illustrated that on two of them, everybody improved, regardless of whether they had the treatment. 00:27:51.733 --> 00:27:54.896 And on the third one, nobody changed very much at all. 00:27:54.896 --> 00:28:03.226 So, this is not dramatic evidence of improvement but you could argue: "Well, nevertheless there was one measure that looked a little bit promising." 00:28:03.227 --> 00:28:11.782 But they then, in the second phase of the study, went on to give the control group the same treatment, and they published this in 2007. 00:28:11.782 --> 00:28:16.456 So we now don't anymore have a control group as everybody has been treated: 00:28:16.456 --> 00:28:21.587 one group early on, and the other group with a delayed time scale. 00:28:21.587 --> 00:28:30.068 And they presented the data between time 1 at the start of the study and right at the end of the study, when everybody had had this treatment. 00:28:30.069 --> 00:28:37.502 But when you look at the results there, it's clear that there really is a, you know, no persistent improvement in reading. 00:28:37.502 --> 00:28:41.723 In fact, the mean scores for the children having the delayed treatment on the reading test 00:28:41.724 --> 00:28:45.453 have now really gone down, rather than up, at the end of treatment. 00:28:45.454 --> 00:28:51.185 And the general impression, I would say, is that there is nothing very stunning going on here, 00:28:51.186 --> 00:28:57.615 certainly nothing that matches the description that you get on the promotion materials for the intervention. 00:28:57.616 --> 00:29:08.095 So, overall, I would argue that the evidence for gains associated with this treatment is really not at all compelling. 00:29:08.096 --> 00:29:17.265 First of all, the claims that are made for stunning changes are all coming from analyses where they didn't incorporate the controls 00:29:17.265 --> 00:29:21.820 and they just tried to argue that any change you see at the time must be due to the treatment. 00:29:21.820 --> 00:29:25.091 and not taking into account all these other factors. 00:29:25.092 --> 00:29:30.355 And on reading measures, where there was control group data available, 00:29:30.356 --> 00:29:37.098 there was an initial small gain in the treated group, but it wasn't sustained by the end of the study. 00:29:37.099 --> 00:29:41.359 So, it really doesn't look terribly promising. 00:29:42.420 --> 00:29:46.989 Now, this is why in general, I think it's true to say this: 00:29:46.989 --> 00:29:52.641 I don't know of anybody in the dyslexic community who is an advocate - in the academic community 00:29:52.641 --> 00:29:57.753 who is an advocate of the DORE treatment, other than people that are directly associated with the DORE organization. 00:29:57.754 --> 00:30:05.466 And so, the reason really is just that the evidence is not at all compelling, 00:30:05.466 --> 00:30:09.184 although the study was small and you could argue a larger study should be done. 00:30:09.184 --> 00:30:14.779 There is a real mismatch between the claims that are being made and the evidence that is available. 00:30:14.780 --> 00:30:21.717 But the interesting thing is also why so many people seem to nevertheless regard this as an effective treatment. 00:30:21.717 --> 00:30:29.504 If the testimonials are to be believed, there are many satisfied customers and happy parents who feel that their children have been helped. 00:30:29.505 --> 00:30:35.596 I think there is quite an interesting set of reasons why this may be so. 00:30:35.597 --> 00:30:41.405 And one is that there is a well known - in the psychological field - well known human tendency 00:30:41.406 --> 00:30:47.147 to think that something that you've put in a lot of time and money too, was worthwhile. 00:30:47.148 --> 00:30:55.566 It's called cognitive dissonance, and it means that if you've actually put in the effort, you tend to feel that there was an effect. 00:30:55.567 --> 00:31:01.531 You have to somehow resolve this sort of inconsistency, otherwise, in your mind. 00:31:01.532 --> 00:31:06.852 And this was beautifully illustrated, not by the trial of the DORE treatment, but in another trial, 00:31:06.852 --> 00:31:11.242 which was a very nicely well-conducted trial of something called Sunflower therapy, 00:31:11.243 --> 00:31:19.165 which is a rather holistic approach to intervention for dyslexia that involves kinesiology and physical manipulation, 00:31:19.166 --> 00:31:23.499 massage, homeopathy, herbal remedies and neurolingusitc programming. 00:31:23.500 --> 00:31:30.274 And there was a very rigorous study done for this, and what was interesting about it 00:31:30.275 --> 00:31:39.296 was that, like so many of these things, they didn't really find a lot of evidence for any better change in the clinical versus the control group, 00:31:39.297 --> 00:31:43.357 although, to some extent, both groups were securing (?) their schools were improving. 00:31:43.867 --> 00:31:49.837 What they did find, though, is that the children themselves had higher self-esteem if they had undergone the Sunflower treatment, 00:31:49.838 --> 00:31:56.296 but that also, 57% of the parents did think that Sunflower therapy was effective in treating their child. 00:31:56.955 --> 00:32:04.260 So there is a clear mismatch between what the study showed of the objective evidence on the children's learning difficulties, 00:32:04.261 --> 00:32:06.682 and what the parents actually thought. 00:32:06.683 --> 00:32:12.230 It is possible that this could be related to the fact that the children's scores did improve, 00:32:12.231 --> 00:32:17.595 but if you didn't know that the control children had also improved, you might attribute that to the therapy - 00:32:17.595 --> 00:32:23.572 but also to the fact that people were again being given a lot of encouragement, 00:32:23.573 --> 00:32:29.845 there was a lot invested in that treatment, and then there might well (?) have been some sort of sense of cognitive dissonance there. 00:32:29.846 --> 00:32:41.088 There's also a strong human tendency to be impressed by certain kinds of explanation that get more biological about dyslexia, 00:32:41.088 --> 00:32:48.683 particularly those such as the DORE treatment that get more neurological and claim to be doing something to the brain in treating dyslexia. 00:32:48.684 --> 00:32:54.007 There was a beautiful study done - published - in 2008, not on dyslexia, 00:32:54.007 --> 00:32:59.816 but just more generally on people's tendency to be impressed by scientific eyplanations. 00:32:59.817 --> 00:33:09.268 and what these researchers did was to give people explanations of psychological phenomena that are well known 00:33:09.269 --> 00:33:14.108 and they either gave them a good explanation or they gave them a not very good explanation 00:33:14.109 --> 00:33:21.341 that was more like just a re-description of the effect, and asked people to judge whether this was a good explanation or not. 00:33:21.342 --> 00:33:25.603 And what was fascinating about this study was that in general, people were quite good at doing this: 00:33:25.604 --> 00:33:33.309 even if they had no familiarity or background in psychology, they could distinguish a good explanation from a bad one. 00:33:33.309 --> 00:33:39.629 But what they found was that if they added some verbiage that just talked about the brain in various ways, 00:33:39.629 --> 00:33:45.245 and said, "This result came about because brain scans showed it", or "because we looked at the frontal lobes", 00:33:45.246 --> 00:33:48.371 people were much more impressed with the bad explanations. 00:33:48.371 --> 00:33:53.128 So a good explanation didn't get any better when you added all this neuroscience waffle, 00:33:53.129 --> 00:33:59.340 but if you added neuroscience waffle to a bad explanation, people thought it not so bad. 00:33:59.341 --> 00:34:05.954 And so, there is a tendency to be very impressed by anything that talks about, adds the brain in to an ........ (?) explanation. 00:34:05.955 --> 00:34:12.994 And I think this is used by people who then try and add spurious neuroscience sometimes 00:34:12.995 --> 00:34:18.171 to their accounts of their particular promissing theory. 00:34:18.172 --> 00:34:23.808 And it really is not - we shouldn't allow ourselves to be mislead. 00:34:24.541 --> 00:34:34.228 So I think, to sum up, there are a number of barriers to objective evaluation of intelligence, 00:34:34.229 --> 00:34:38.309 which, to some extent, functions about (?) our human condition, 00:34:38.309 --> 00:34:44.095 that we are not naturally good at taking in lots of numbers and looking at graphs 00:34:44.096 --> 00:34:47.758 and trying to sort of take into account alternative explanations. 00:34:47.759 --> 00:34:52.649 We tend to be impressed when we hear other people tell us that something has worked 00:34:52.649 --> 00:34:57.183 and it's hard - you have to almost guard yourself against the tendency to do that 00:34:57.184 --> 00:35:02.389 and to look rather for the hard evidence, to look for the actual numerical data. 00:35:03.249 --> 00:35:08.301 We have to be very careful when people start giving us explanations that have got a lot of neuroscience in them 00:35:08.301 --> 00:35:12.194 and check out, is this real neuroscience or is it just put in there to impress us? 00:35:13.808 --> 00:35:19.508 We have to be aware of the effect of cognitive dissonance and the tendency to believe some things 00:35:19.509 --> 00:35:23.866 simply because we have invested time and money in it 00:35:23.867 --> 00:35:31.401 and, most importantly, we have to bear in mind that there will be effects on children's performance 00:35:31.401 --> 00:35:37.974 of maturation, of our expectations, of just get practising on things, 00:35:37.975 --> 00:35:42.705 and there are also these dreadful statistical artefacts that can make it look as if a change has occurred, 00:35:42.706 --> 00:35:45.838 when it's really not particularly impressive. 00:35:45.838 --> 00:35:50.363 But I think, if one bears these things in mind, the bottom line is really: 00:35:50.364 --> 00:35:54.575 "Look for evidence from studies that have got adequate controls" 00:35:54.576 --> 00:36:01.508 and if you do, you'll be in - a standard, I think, by how far you can see improvements in children, even if they haven't had the treatment. 00:36:01.508 --> 00:36:06.140 and there are lots of things that will make things a lot better, just with the passage of time. 00:36:06.141 --> 00:36:11.339 But if you really want to demonstrate that there has been an effective treatment, 00:36:11.339 --> 00:36:15.804 you do have to show an improvement relative to a control group, 00:36:15.804 --> 00:36:20.955 rather than just, somebody started out not so good and is now a little bit better after the treatment. 00:36:20.956 --> 00:36:29.953 I hope that that might give you some useful indicators when trying to look at new treatments that are out there and on offer, 00:36:29.954 --> 00:36:35.111 and for a more detailed account of some of this work, there are various - 00:36:35.112 --> 00:36:40.104 there is a powerpoint presentation with notes on my website on this topic.