[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.00,0:00:06.06,Default,,0000,0000,0000,,[Dorothy Bishop] I'm going to talk today about evaluating alternative intervention approaches to dyslexia.
Dialogue: 0,0:00:06.06,0:00:07.46,Default,,0000,0000,0000,,[slide with talk title]
Dialogue: 0,0:00:07.46,0:00:13.67,Default,,0000,0000,0000,,The conventional approaches that you get really tend to rely on the fact that we've now got of evidence
Dialogue: 0,0:00:13.67,0:00:18.41,Default,,0000,0000,0000,,that most children with dyslexia have problems in what is called phonological awareness,
Dialogue: 0,0:00:18.41,0:00:23.44,Default,,0000,0000,0000,,that is, they don't necessarily hear all the different sounds in speech,
Dialogue: 0,0:00:23.44,0:00:28.38,Default,,0000,0000,0000,,and therefore have difficulty relating them to letters when they are trying to read.
Dialogue: 0,0:00:28.38,0:00:37.52,Default,,0000,0000,0000,,And most of the interventions that are mainstream these days would focus on trying to train children to identify sounds in words and relate them to letters.
Dialogue: 0,0:00:38.97,0:00:45.41,Default,,0000,0000,0000,,But this sort of intervention has been shown to be effective, and there have been a number of large scale studies.
Dialogue: 0,0:00:45.42,0:00:51.72,Default,,0000,0000,0000,,But nevertheless, it has to be fairly prolonged for some children, and there are children for whom,
Dialogue: 0,0:00:51.72,0:00:57.87,Default,,0000,0000,0000,,even though they can learn this way to actually sound out words and read, they don't necessarily read fluently.
Dialogue: 0,0:00:57.91,0:01:03.31,Default,,0000,0000,0000,,It's still an effort for them, and they don't sort of get to the degree of automaticity that you might expect,
Dialogue: 0,0:01:03.31,0:01:07.68,Default,,0000,0000,0000,,And it's certainly the case that methods that work for many children don't work for all children,
Dialogue: 0,0:01:07.68,0:01:10.76,Default,,0000,0000,0000,,and there is a hard core of children who remain very hard to treat.
Dialogue: 0,0:01:10.76,0:01:15.94,Default,,0000,0000,0000,,It is for this reason that many parents do get very concerned about whether there is something else they should be doing
Dialogue: 0,0:01:15.94,0:01:21.100,Default,,0000,0000,0000,,if they are finding that their child is either not getting intervention, or that the intervention doesn't seem to be working very well.
Dialogue: 0,0:01:21.100,0:01:29.45,Default,,0000,0000,0000,,And there are a whole load of things out there that are on offer, and the problem for the parents, I think -
Dialogue: 0,0:01:29.45,0:01:33.98,Default,,0000,0000,0000,,and/or indeed for adults who themselves, might want to have further intervention for dyslexia -
Dialogue: 0,0:01:33.98,0:01:37.69,Default,,0000,0000,0000,,is that they want to know, "how do I distinguish something that might work for me",
Dialogue: 0,0:01:37.69,0:01:41.64,Default,,0000,0000,0000,,from something that is just some sort of snake-oil merchant who is out there to make money.
Dialogue: 0,0:01:41.64,0:01:44.48,Default,,0000,0000,0000,,And that's what I want to try and address today.
Dialogue: 0,0:01:44.48,0:01:51.54,Default,,0000,0000,0000,,principally from the perspective of how you might evaluate scientific evidence that people put forward.
Dialogue: 0,0:01:51.54,0:01:57.49,Default,,0000,0000,0000,,But perhaps before going onto that, it's worth going into some relatively commonsense things.
Dialogue: 0,0:01:57.62,0:02:06.61,Default,,0000,0000,0000,,I would say that there are certain things that should ring alarm bells if people are advertising some sort of treatment for child dyslexia.
Dialogue: 0,0:02:06.65,0:02:13.06,Default,,0000,0000,0000,,The first thing is if the intervention has been developped by somebody who has no academic track record,
Dialogue: 0,0:02:13.06,0:02:17.83,Default,,0000,0000,0000,,no experience of doing research in this field, and hasn't published anything in this field,
Dialogue: 0,0:02:17.83,0:02:24.54,Default,,0000,0000,0000,,if the intervention isn't endorsed by people in the mainstream dyslexia field,
Dialogue: 0,0:02:24.54,0:02:27.63,Default,,0000,0000,0000,,that should also sound a note of caution.
Dialogue: 0,0:02:27.63,0:02:29.82,Default,,0000,0000,0000,,Of course, the mainstream people aren't always right.
Dialogue: 0,0:02:29.82,0:02:32.97,Default,,0000,0000,0000,,It's possible that somebody with no background will develop something marvelous.
Dialogue: 0,0:02:32.97,0:02:37.47,Default,,0000,0000,0000,,But if that were the case, you would expect it to be pretty quickly picked up by people in the mainstream,
Dialogue: 0,0:02:37.48,0:02:40.80,Default,,0000,0000,0000,,who are really, on the whole, pretty keen to find things that will work.
Dialogue: 0,0:02:40.80,0:02:47.81,Default,,0000,0000,0000,,And you obviously want to look at whether somebody is asking for a lot of money for something that hasn't been proven.
Dialogue: 0,0:02:47.81,0:02:56.22,Default,,0000,0000,0000,,And what is also, to my mind, a worrying sign, is if somebody promoting a treatment is relying heavily just on testimonials
Dialogue: 0,0:02:56.22,0:03:04.15,Default,,0000,0000,0000,,from individuals who claimed to have been cured, rather than having any sort of proper scientific evaluation or kind of controls.
Dialogue: 0,0:03:04.16,0:03:10.61,Default,,0000,0000,0000,,And it's worth noting that human beings have a tendency to be terribly impressed by testimonials,
Dialogue: 0,0:03:10.61,0:03:16.75,Default,,0000,0000,0000,,and even myself, as somebody with a scientific training, I find if, you know, I've got headaches and somebody comes along and says:
Dialogue: 0,0:03:16.75,0:03:20.60,Default,,0000,0000,0000,,"I was cured by such and such, and I went to my herbalist and it worked",
Dialogue: 0,0:03:20.60,0:03:27.09,Default,,0000,0000,0000,,you know, you're often very tempted to be much more swayed by that sort of evidence than by a pagefull of numbers and figures.
Dialogue: 0,0:03:27.10,0:03:34.68,Default,,0000,0000,0000,,And this just a human tendency: we are naturally built to really take advice from other people and to rely on what they tell us.
Dialogue: 0,0:03:34.68,0:03:38.49,Default,,0000,0000,0000,,But in the contexts of these sorts of interventions, that's really quite dangerous,
Dialogue: 0,0:03:38.49,0:03:44.27,Default,,0000,0000,0000,,because, when somebody gives a testimonial, that's just one person, their only individual experience,
Dialogue: 0,0:03:44.27,0:03:48.27,Default,,0000,0000,0000,,And people you don't hear from tend to be the people who tried it, and it didn't work.
Dialogue: 0,0:03:48.27,0:03:51.88,Default,,0000,0000,0000,,And you don't know how many of them there are: there may be thousands of them.
Dialogue: 0,0:03:51.88,0:03:56.05,Default,,0000,0000,0000,,But they're not going to publicize the fact that they tried it and it didn't work.
Dialogue: 0,0:03:56.05,0:04:01.94,Default,,0000,0000,0000,,And so, testimonials are often very much at odds with more scientific evaluations.
Dialogue: 0,0:04:01.94,0:04:10.10,Default,,0000,0000,0000,,.... to turn out that when somebody says there is scientific evidence for what they're doing, how you should interpret that.
Dialogue: 0,0:04:10.10,0:04:14.96,Default,,0000,0000,0000,,And that's jolly difficult even for scientists sometimes: there is disagreement - let alone for the general public.
Dialogue: 0,0:04:14.96,0:04:20.73,Default,,0000,0000,0000,,But again, I think, there are some sort of general rules of thumb that you can go by
Dialogue: 0,0:04:20.73,0:04:24.02,Default,,0000,0000,0000,,for telling that a treatment is likely to be effective.
Dialogue: 0,0:04:25.45,0:04:38.23,Default,,0000,0000,0000,,And when I discuss this, I'm going to illustrate it by taking the example of the Dore treatment - that's DORE, named after Wynford Dore, it's originator.
Dialogue: 0,0:04:38.23,0:04:46.36,Default,,0000,0000,0000,,ANd I'm picking on this largely because it is a non-mainstream treatment that isn't widely accepted by the experts,
Dialogue: 0,0:04:46.36,0:04:49.90,Default,,0000,0000,0000,,and yet it does claim that there is some scientific evidence to support it,
Dialogue: 0,0:04:49.90,0:04:54.28,Default,,0000,0000,0000,,which has lead the scientists to look at it quite critically and quite carefully,
Dialogue: 0,0:04:54.28,0:04:57.65,Default,,0000,0000,0000,,which is what we would do with any scientific evidence that comes along:
Dialogue: 0,0:04:57.66,0:05:04.83,Default,,0000,0000,0000,,once it's out in the public domain and published, people tend to go and look at it as carefully as they possibly can.
Dialogue: 0,0:05:04.83,0:05:11.94,Default,,0000,0000,0000,,Now, the Dore method is interesting to us, because it does illustrate the case where there is disagreement
Dialogue: 0,0:05:11.94,0:05:15.20,Default,,0000,0000,0000,,as to whether the evidence is showing that its' effective or not.
Dialogue: 0,0:05:15.20,0:05:20.59,Default,,0000,0000,0000,,And so, what I want to explain is why it is the case that despite this published evidence,
Dialogue: 0,0:05:20.59,0:05:25.32,Default,,0000,0000,0000,,most of the experts are not impressed of the efficacy of the Dore treatment.
Dialogue: 0,0:05:25.32,0:05:32.23,Default,,0000,0000,0000,,But the general points that I'll make would apply to any other treatment that was out there, whether (?) there was evidence being produced.
Dialogue: 0,0:05:32.75,0:05:39.96,Default,,0000,0000,0000,,So, first of all, what is the Dore method? Well, it's a method that has been proposed for curing problems
Dialogue: 0,0:05:39.96,0:05:44.77,Default,,0000,0000,0000,,that are thought to originate in the part of the brain called the cerebellum, which is at the back of the brain,
Dialogue: 0,0:05:44.77,0:05:49.100,Default,,0000,0000,0000,,and it was developed by Wynford Dore as a method for helping his dyslexic daughter.
Dialogue: 0,0:05:49.100,0:05:53.58,Default,,0000,0000,0000,,He has written a book about the history of how this came to became (?) about,
Dialogue: 0,0:05:53.58,0:05:58.97,Default,,0000,0000,0000,,and he was a classic instance of a parent who was rather desperate to help his daughter who, for many years,
Dialogue: 0,0:05:58.97,0:06:03.51,Default,,0000,0000,0000,,had been through the educational system and failed, and was getting increasingly depressed.
Dialogue: 0,0:06:03.51,0:06:10.84,Default,,0000,0000,0000,,And he tried various things, he talked to various experts, and he ended up with this program that's been put forward,
Dialogue: 0,0:06:10.84,0:06:15.54,Default,,0000,0000,0000,,which is an individualized program, where the child follows various sorts of exercises,
Dialogue: 0,0:06:15.54,0:06:21.02,Default,,0000,0000,0000,,which are done for about ten minutes twice a day, over quite a long period of time,
Dialogue: 0,0:06:21.02,0:06:25.26,Default,,0000,0000,0000,,varying, depending on the severity of the problem, from maybe 6 months to 2 years.
Dialogue: 0,0:06:25.93,0:06:30.98,Default,,0000,0000,0000,,And the child is assessed at regular intervals and different exercises may be prescribed.
Dialogue: 0,0:06:30.98,0:06:38.55,Default,,0000,0000,0000,,Now, the theory behind the Dore method is that dyslexia and other learning difficulties -
Dialogue: 0,0:06:38.55,0:06:43.66,Default,,0000,0000,0000,,it's not just dyslexia it claims to help, but also Attention Deficit problems ...(?) hyperactive -
Dialogue: 0,0:06:43.66,0:06:49.04,Default,,0000,0000,0000,,are thought to arise within the cerebellum: the cerebellum just doesn't develop normally,
Dialogue: 0,0:06:49.04,0:06:54.47,Default,,0000,0000,0000,,and the argument is that you can have different cerebellar impediments in different people,
Dialogue: 0,0:06:54.48,0:06:57.20,Default,,0000,0000,0000,,and that's why you can get this range of different symptoms,
Dialogue: 0,0:06:57.20,0:07:04.06,Default,,0000,0000,0000,,but that you can diagnose them by specific tests of test of mental and physical coordinations.
Dialogue: 0,0:07:04.06,0:07:12.68,Default,,0000,0000,0000,,And what you are then supposed to do is these exercises, which are not anywhere fully described in the public domain,
Dialogue: 0,0:07:12.68,0:07:17.62,Default,,0000,0000,0000,,because they are commercially sensitive, but there are some examples given, and it's clear that what they do
Dialogue: 0,0:07:17.62,0:07:22.71,Default,,0000,0000,0000,,is focused largely on training balance and hand-eye coordination in children.
Dialogue: 0,0:07:22.72,0:07:29.89,Default,,0000,0000,0000,,So you might be asked to stand on a cushion on one leg, or to throw a bean bag from one hand to another
Dialogue: 0,0:07:29.89,0:07:37.30,Default,,0000,0000,0000,,while you are doing that, just stand on a wobble board (?) and balance, or to follow something with your eyes in a particular way.
Dialogue: 0,0:07:39.19,0:07:48.27,Default,,0000,0000,0000,,So, the idea is that these are all things that the cerebellum is involved in, by training up the cerebellum, you may improve its general abilities.
Dialogue: 0,0:07:48.73,0:07:53.44,Default,,0000,0000,0000,,So, what is the evidence for this underlying theory?
Dialogue: 0,0:07:53.44,0:07:58.40,Default,,0000,0000,0000,,Well, it's not a proven theory, but there is some support for it.
Dialogue: 0,0:07:58.40,0:08:06.92,Default,,0000,0000,0000,,Certainly, people trying to look at what is going on in the brain in dyslexia have proposed many different theories
Dialogue: 0,0:08:06.92,0:08:09.31,Default,,0000,0000,0000,,about what the underlying causes might be.
Dialogue: 0,0:08:09.32,0:08:14.74,Default,,0000,0000,0000,,If you look at the brain in a brain scanner of somebody with dyslexia, it typically looks totally normal.
Dialogue: 0,0:08:14.74,0:08:19.26,Default,,0000,0000,0000,,There's certainly no big holes in the head or anything like that, that you are going to see on a scanner.
Dialogue: 0,0:08:19.26,0:08:24.86,Default,,0000,0000,0000,,But the argument is being made that there may be regions of the cerebellum that are perhaps slightly smaller than they should be
Dialogue: 0,0:08:24.87,0:08:28.05,Default,,0000,0000,0000,,or not functioning quite as they should be.
Dialogue: 0,0:08:28.05,0:08:33.82,Default,,0000,0000,0000,,And this theory has some support, although not everybody would agree with it
Dialogue: 0,0:08:33.82,0:08:38.84,Default,,0000,0000,0000,,and there is certainly other theories equally plausible at the moment that are around.
Dialogue: 0,0:08:39.44,0:08:44.66,Default,,0000,0000,0000,,The notion - the cerebellum is important for getting things automated.
Dialogue: 0,0:08:44.66,0:08:50.68,Default,,0000,0000,0000,,So you can - when you learn to drive a car, first of all, it's very slow and effortfull, and you have to think about everything you do.
Dialogue: 0,0:08:50.68,0:08:55.18,Default,,0000,0000,0000,,By the time you are a skilled driver, it's no longe the case that you have to do that,
Dialogue: 0,0:08:55.19,0:08:58.06,Default,,0000,0000,0000,,you just drive around without thinking about it.
Dialogue: 0,0:08:58.06,0:09:00.06,Default,,0000,0000,0000,,You can do all sorts of other things while you are driving.
Dialogue: 0,0:09:00.06,0:09:07.32,Default,,0000,0000,0000,,So, the argument is that with reading, most people, similarly, become very automatic in how they learn to read:
Dialogue: 0,0:09:07.32,0:09:14.90,Default,,0000,0000,0000,,you do it without thinking about it, but for the dylexic it remains effortfull because the cerebellum is not functioning normally
Dialogue: 0,0:09:14.90,0:09:19.39,Default,,0000,0000,0000,,and it's the cerebellum that helps you get your skills automatized.
Dialogue: 0,0:09:19.39,0:09:26.87,Default,,0000,0000,0000,,And in support of this, it has been argued that in many people with dyslexia, there are some associated problems with motor coordination,
Dialogue: 0,0:09:26.88,0:09:32.79,Default,,0000,0000,0000,,..... (?) physical skills and so on, and that too could be a sign of a problem with the cerebellum.
Dialogue: 0,0:09:32.79,0:09:39.88,Default,,0000,0000,0000,,Again, that's fairly controversial, it's not being found in all children, and the arguments go to and fro.
Dialogue: 0,0:09:39.88,0:09:47.49,Default,,0000,0000,0000,,But this is not a sort of theory that is particularly disapproved of by the mainstream. People are debating it.
Dialogue: 0,0:09:47.49,0:09:55.05,Default,,0000,0000,0000,,The difficult stumbling block, though, for the Dore approach to treatment comes with the idea that
Dialogue: 0,0:09:55.05,0:10:02.53,Default,,0000,0000,0000,,if you train the motor skills, that is a sort of coordination between different muscles and movements
Dialogue: 0,0:10:02.53,0:10:09.37,Default,,0000,0000,0000,,and between their eyes and hands, that this will somehow have a knock-on effect on things like reading.
Dialogue: 0,0:10:09.38,0:10:16.40,Default,,0000,0000,0000,,And indeed, David Randall and colleagues, who published this initial study on the treatment,
Dialogue: 0,0:10:16.40,0:10:23.41,Default,,0000,0000,0000,,describe it as something of a leap of faith, because the cerebellum is actually known to be a very complicated organ,
Dialogue: 0,0:10:23.41,0:10:27.25,Default,,0000,0000,0000,,with lots of different regions, which are fairly independent from one another.
Dialogue: 0,0:10:27.25,0:10:34.59,Default,,0000,0000,0000,,So there is no real reason to suppose that if you train one part of the cerebellum, it will have somehow a generalized benefit.
Dialogue: 0,0:10:34.59,0:10:40.75,Default,,0000,0000,0000,,And indeed, you could say: "Well, if it were the case that this is true, if you'd had a chance, you would go to skateboarding,
Dialogue: 0,0:10:40.75,0:10:48.71,Default,,0000,0000,0000,,or playing ping-pong, or things like that, ....... (?) or perhaps ballet dancing, things that require balance and coordination,
Dialogue: 0,0:10:48.71,0:10:52.65,Default,,0000,0000,0000,,that should protect you against dyslexia". There is really not much evidence for that,
Dialogue: 0,0:10:52.65,0:10:57.64,Default,,0000,0000,0000,,on the contrary, there is some very good sportsmen who - gymnasts and people with dyslexia.
Dialogue: 0,0:10:57.64,0:11:06.79,Default,,0000,0000,0000,,So it is hard to see how the logic of saying "Train these motor skills and somehow the whole cerebellum function some day improves2
Dialogue: 0,0:11:06.79,0:11:12.88,Default,,0000,0000,0000,,But what does the published evidence look like? Because the theory might be, you know, questionable,
Dialogue: 0,0:11:12.88,0:11:16.96,Default,,0000,0000,0000,,but basically, what the parents are going to say is, "What matter is, does it work?"
Dialogue: 0,0:11:18.22,0:11:24.72,Default,,0000,0000,0000,,Well, there is a published study on the intervention, which claims that it shows that it really does work
Dialogue: 0,0:11:24.72,0:11:28.30,Default,,0000,0000,0000,,if you compare children who have the intervention and children who don't.
Dialogue: 0,0:11:28.30,0:11:35.100,Default,,0000,0000,0000,,And two papers have been reported - one from the initial phase of the study, and the other from a subsequent phase -
Dialogue: 0,0:11:36.87,0:11:45.54,Default,,0000,0000,0000,,And they are reported in the Journal of Dyslexia which, in 2003, published the first paper
Dialogue: 0,0:11:45.66,0:11:53.89,Default,,0000,0000,0000,,which was on just under - started with a sample statistics on the 300 children who were all attending a .... (?) primary school.
Dialogue: 0,0:11:53.89,0:11:58.02,Default,,0000,0000,0000,,And the researchers went in and screened all the children on the dyslexia screening test,
Dialogue: 0,0:11:58.02,0:12:01.46,Default,,0000,0000,0000,,to pull out children who would be suitable for enrolment in the study.
Dialogue: 0,0:12:01.46,0:12:10.65,Default,,0000,0000,0000,,But the first thing that is more important to note is that these were not children who had a very high rate of diagnosis of dyslexia.
Dialogue: 0,0:12:10.65,0:12:19.26,Default,,0000,0000,0000,,So, there were 35 in the group, and about a third of those came out as having a strong risk of dyslexia on this dyslexia screening test.
Dialogue: 0,0:12:19.26,0:12:26.74,Default,,0000,0000,0000,,Another 21% came out with a mild risk, but about half of these children were not really ...... (?) in this category
Dialogue: 0,0:12:26.74,0:12:31.50,Default,,0000,0000,0000,,and they were just picked because their schools (?) were relatively lower compared to the other children.
Dialogue: 0,0:12:31.50,0:12:38.20,Default,,0000,0000,0000,,And there were only a total of 6 children who had previously been diagnosed with dyslexia, out of the 35.
Dialogue: 0,0:12:38.20,0:12:42.99,Default,,0000,0000,0000,,There were a couple with a diagnosis of dyspraxia and one with ADHD diagnosis.
Dialogue: 0,0:12:42.99,0:12:48.46,Default,,0000,0000,0000,,So this is not really a sample consisting of children really with severe problems on the whole.
Dialogue: 0,0:12:48.46,0:12:50.66,Default,,0000,0000,0000,,There were few in there with major difficulties.
Dialogue: 0,0:12:50.66,0:12:58.04,Default,,0000,0000,0000,,Nevertheless, the originators of the treatment would argue even quite mild problems might be worth treating with this
Dialogue: 0,0:12:58.04,0:13:01.89,Default,,0000,0000,0000,,and so you could argue this study is nevertheless of value.
Dialogue: 0,0:13:01.89,0:13:07.83,Default,,0000,0000,0000,,So what they did, they started out well in this study: they divided the children randomly in treated and untreated groups,
Dialogue: 0,0:13:07.83,0:13:13.69,Default,,0000,0000,0000,,which is, as I am going on to explain later, is an important part of a good study.
Dialogue: 0,0:13:13.69,0:13:20.73,Default,,0000,0000,0000,,And if you look at the results that are described on the promotion materials of the DORE organization,
Dialogue: 0,0:13:20.73,0:13:28.48,Default,,0000,0000,0000,,they are all in Dore's book that he published, "Dyslexia, the miracle cure", he described the results as stunning
Dialogue: 0,0:13:28.48,0:13:35.04,Default,,0000,0000,0000,,and said that reading age increased threefold, comprehension age increased almost fivefold
Dialogue: 0,0:13:35.04,0:13:40.64,Default,,0000,0000,0000,,and writing skills by what he described as "an extraordinary 17-fold".
Dialogue: 0,0:13:40.64,0:13:47.86,Default,,0000,0000,0000,,Of course, everybody reading that think "Wow, my child is going to take off like a rocket if we put him on this intervention."
Dialogue: 0,0:13:47.86,0:13:56.81,Default,,0000,0000,0000,,But unfortunately, these figures are really a classic instance of how statistics can be manipulated in a very misleading way.
Dialogue: 0,0:13:57.41,0:14:04.91,Default,,0000,0000,0000,,So, for a start, they were not based on any comparison between the control children and the untreated children -
Dialogue: 0,0:14:04.91,0:14:09.18,Default,,0000,0000,0000,,sorry, the control children and the treated children.
Dialogue: 0,0:14:09.18,0:14:17.57,Default,,0000,0000,0000,,They were - instead, they just took all the children who would be treated and looked at how they did on a group reading test
Dialogue: 0,0:14:17.57,0:14:21.04,Default,,0000,0000,0000,,that had been administered by the school every year.
Dialogue: 0,0:14:21.04,0:14:30.73,Default,,0000,0000,0000,,And the children had had this on two occasions prior to the intervention - so, 3 months before it started and a year before that -
Dialogue: 0,0:14:30.73,0:14:35.34,Default,,0000,0000,0000,,and on two occasions after the intervention, after this whole long 4-year period.
Dialogue: 0,0:14:35.35,0:14:44.26,Default,,0000,0000,0000,,And what the researchers did was to really just plot the average schools of the group over these 4 time periods
Dialogue: 0,0:14:44.26,0:14:49.64,Default,,0000,0000,0000,,and show that if you compared the amount of change from the first time point to the second,
Dialogue: 0,0:14:49.64,0:14:54.39,Default,,0000,0000,0000,,which was before they had had any treatment, it was a certain amount
Dialogue: 0,0:14:54.40,0:15:00.35,Default,,0000,0000,0000,,and if you then compared the second to the third time point, so the treatment had been going on (?) between those two,
Dialogue: 0,0:15:00.35,0:15:02.66,Default,,0000,0000,0000,,there was a different amount of change.
Dialogue: 0,0:15:02.66,0:15:06.67,Default,,0000,0000,0000,,And then they divided one by the other and showed that there was this threefold improvement.
Dialogue: 0,0:15:06.67,0:15:13.30,Default,,0000,0000,0000,,But it's a very, very misleading way of depicting these data, because if you look at them on a graph, here,
Dialogue: 0,0:15:13.30,0:15:18.17,Default,,0000,0000,0000,,you can see that the only odd thing about the data - well, there's two odd things about the data:
Dialogue: 0,0:15:18.17,0:15:23.17,Default,,0000,0000,0000,,one is that at most time points, these children are reading at absolutely normal levels.
Dialogue: 0,0:15:23.17,0:15:27.21,Default,,0000,0000,0000,,So it's not clear why they are regarded as having risk for dyslexia;
Dialogue: 0,0:15:27.21,0:15:32.95,Default,,0000,0000,0000,,and the one time point when they're not, is the time point 3 months before they are involved in the study,
Dialogue: 0,0:15:32.95,0:15:39.36,Default,,0000,0000,0000,,where there is a bit of a drop. But it's really not an impressive demonstration of change
Dialogue: 0,0:15:39.36,0:15:46.79,Default,,0000,0000,0000,,and this division of one time period by another is very misleading, because it just gives double weighting
Dialogue: 0,0:15:46.79,0:15:52.36,Default,,0000,0000,0000,,to this one low period of three months before the treatment started.
Dialogue: 0,0:15:52.36,0:15:57.61,Default,,0000,0000,0000,,And they did the same thing again with these other figures of massive increases that they talk about,
Dialogue: 0,0:15:57.61,0:16:06.83,Default,,0000,0000,0000,,using data from the SATS tests administered by teachers, which are not really regarded as particularly precise or rigorous tests,
Dialogue: 0,0:16:06.83,0:16:14.02,Default,,0000,0000,0000,,and really group children in a fairly global way at level 2, 3 or 4.
Dialogue: 0,0:16:14.03,0:16:20.49,Default,,0000,0000,0000,,Level 2 is average for 7 year old, 3 is average for 9 year old, and 4 is average for an 11 year old.
Dialogue: 0,0:16:20.50,0:16:27.88,Default,,0000,0000,0000,,And to give you an idea of the sort of misleading nature of these massive changes they talk about,
Dialogue: 0,0:16:27.88,0:16:34.12,Default,,0000,0000,0000,,on the writing test, where there is this incredible change that they talk about, of a 17-fold increase,
Dialogue: 0,0:16:34.13,0:16:40.79,Default,,0000,0000,0000,,the score at age 8, the average score was 2.5, which is about what you'd expect from a 8 year old.
Dialogue: 0,0:16:40.79,0:16:47.10,Default,,0000,0000,0000,,At age 9, it was 2.56, which is a little bit better, but not much.
Dialogue: 0,0:16:47.10,0:16:53.49,Default,,0000,0000,0000,,And then, they argue, the intervention came in, and at age 10, the children scored 2.95.
Dialogue: 0,0:16:53.49,0:16:56.89,Default,,0000,0000,0000,,They are still rather below where they ought to be at the age of 10.
Dialogue: 0,0:16:56.89,0:17:01.37,Default,,0000,0000,0000,,It looks as if on this particular writing assessment, the children were just rather creeping along.
Dialogue: 0,0:17:01.37,0:17:09.88,Default,,0000,0000,0000,,But because the difference between 2.53 and 2.56 is less than the difference between 2.56 and 2.95,
Dialogue: 0,0:17:09.88,0:17:17.94,Default,,0000,0000,0000,,they make a big computation of dividing one by the other, actually coming out with the number 17, which is a wrong number (?): it's actually 13.
Dialogue: 0,0:17:17.94,0:17:24.65,Default,,0000,0000,0000,,So there is a 13-fold change. But if you look at the overall numbers, this is really not so an impressive game at all.
Dialogue: 0,0:17:24.65,0:17:27.76,Default,,0000,0000,0000,,It's really a very misleading way of presenting the numbers.
Dialogue: 0,0:17:27.76,0:17:36.51,Default,,0000,0000,0000,,So, most people would say, this is really smoke in mirrors in terms of using statistics in a way that isn't really valid.
Dialogue: 0,0:17:36.92,0:17:42.13,Default,,0000,0000,0000,,The other thing that is of notice is that all these results that have given so much publicity
Dialogue: 0,0:17:42.13,0:17:47.70,Default,,0000,0000,0000,,in promoting the treatment about these massive changes, haven't talked about the control group at all.
Dialogue: 0,0:17:47.70,0:17:51.43,Default,,0000,0000,0000,,They've just talked about, "Well, we've got these children, before treatment they did this,
Dialogue: 0,0:17:51.43,0:17:53.80,Default,,0000,0000,0000,,and after treatment they did that, and it has all gone up".
Dialogue: 0,0:17:53.80,0:17:59.72,Default,,0000,0000,0000,,And of course, if schools do go up after treatment, it's not necessarily because the treatment works:
Dialogue: 0,0:17:59.72,0:18:02.30,Default,,0000,0000,0000,,There are lots of other reasons you need to bear in mind.
Dialogue: 0,0:18:02.95,0:18:07.56,Default,,0000,0000,0000,,And the first of which is just, on some things, you get better because you get older,
Dialogue: 0,0:18:07.56,0:18:13.64,Default,,0000,0000,0000,,so that if you were to measure shoe sizes before the DORE treatment and after it, it would go up,
Dialogue: 0,0:18:13.64,0:18:16.66,Default,,0000,0000,0000,,but it wouldn't mean that it made your feet grow bigger.
Dialogue: 0,0:18:16.66,0:18:24.34,Default,,0000,0000,0000,,Now, clearly, that's a silly example in most cases, because people try to use measures that don't necessarily change with age,
Dialogue: 0,0:18:24.34,0:18:26.64,Default,,0000,0000,0000,,or that are adjusted in some way for age.
Dialogue: 0,0:18:26.64,0:18:31.74,Default,,0000,0000,0000,,But it's important to bear that in mind when people are talking about changes on things like -
Dialogue: 0,0:18:31.74,0:18:38.03,Default,,0000,0000,0000,,the DORE program, they talk about changes on balance, balance improves dramatically after the program.
Dialogue: 0,0:18:38.03,0:18:43.50,Default,,0000,0000,0000,,These are measures that have not been adjusted for age at all, and so, some of these changes could well be due to the fact
Dialogue: 0,0:18:43.50,0:18:48.31,Default,,0000,0000,0000,,that the children are getting older and getting better at doing these things because of that.
Dialogue: 0,0:18:48.31,0:18:56.12,Default,,0000,0000,0000,,Another uninteresting reason why schools may improve is that the children may be having some other sort of special help.
Dialogue: 0,0:18:56.13,0:19:03.32,Default,,0000,0000,0000,,So, if a child is having reading difficulty, they may very well be getting some special help at the school, in addition to following this program.
Dialogue: 0,0:19:03.32,0:19:10.04,Default,,0000,0000,0000,,And that may be what's causing the change, rather than the particular intervention you are interested in.
Dialogue: 0,0:19:10.04,0:19:14.87,Default,,0000,0000,0000,,What's very well known, of course, is the placebo effect, which is a sort of concept coming from medicine,
Dialogue: 0,0:19:14.87,0:19:20.64,Default,,0000,0000,0000,,which also says that you can get better just because you think you are going to get better, because you think somebody has done something effective.
Dialogue: 0,0:19:20.64,0:19:25.61,Default,,0000,0000,0000,,And in the case of educational treatments, you can see effects where -
Dialogue: 0,0:19:25.61,0:19:31.88,Default,,0000,0000,0000,,because the teachers and the parents and the children themselves are all full of expectations of how this is going to improve them -
Dialogue: 0,0:19:31.88,0:19:38.91,Default,,0000,0000,0000,,there is more motivation: everybody gets positive attention and this itself can cause positive effects.
Dialogue: 0,0:19:40.15,0:19:45.83,Default,,0000,0000,0000,,The fourth reason, which is often neglected, because it really doesn't affect things in medicine so much,
Dialogue: 0,0:19:45.83,0:19:50.07,Default,,0000,0000,0000,,but in education, it's actually rather important, using the sort of thing like reading tests:
Dialogue: 0,0:19:50.07,0:19:55.60,Default,,0000,0000,0000,,you can have practice effects. So you can get better upon some things, just because you've done it before.
Dialogue: 0,0:19:55.60,0:20:00.52,Default,,0000,0000,0000,,And we've seen this quite a lot with language tests, for example, that we give to children,
Dialogue: 0,0:20:00.52,0:20:05.70,Default,,0000,0000,0000,,where, the first time you test a child, they don't know what to expect, they don't know what's coming,
Dialogue: 0,0:20:05.70,0:20:09.61,Default,,0000,0000,0000,,you aske them to do something that's unfamiliar and they are a little bit nervous, maybe.
Dialogue: 0,0:20:09.61,0:20:16.12,Default,,0000,0000,0000,,You test them again on the same thing a month later: they are much, much better, simply because they've done it before
Dialogue: 0,0:20:16.12,0:20:19.89,Default,,0000,0000,0000,,and they are calmer about it, they know what to expect, and so on.
Dialogue: 0,0:20:19.89,0:20:26.12,Default,,0000,0000,0000,,So you can get practice effects that can make quite a difference, just because you know what to expect
Dialogue: 0,0:20:26.12,0:20:30.40,Default,,0000,0000,0000,,and you are familiar with the whole situation of the test.
Dialogue: 0,0:20:30.40,0:20:37.86,Default,,0000,0000,0000,,The fifth reason - and the last one, you'll be pleased to hear - why people may improve for no good reason
Dialogue: 0,0:20:37.86,0:20:42.75,Default,,0000,0000,0000,,is the hardest to explain and it's something known as regression to the mean,
Dialogue: 0,0:20:42.75,0:20:49.40,Default,,0000,0000,0000,,and it's just a statistical artefact, which has to do with, if you pick somebody because they're bad at something,
Dialogue: 0,0:20:49.40,0:20:54.26,Default,,0000,0000,0000,,the odds are, when you test them on a second occasion, they'll be a little bit better (?).
Dialogue: 0,0:20:54.26,0:20:59.50,Default,,0000,0000,0000,,The converse is also true: if you pick somebody who is very good, they tend to get a little bit worse when you test them a second time.
Dialogue: 0,0:20:59.50,0:21:08.74,Default,,0000,0000,0000,,Why should that be? The reason why this occurs is because our measures are not entirely perfect an accurate -
Dialogue: 0,0:21:08.74,0:21:13.44,Default,,0000,0000,0000,,I'm showing a graph here, where we have a measure that is almost perfect,
Dialogue: 0,0:21:13.44,0:21:21.95,Default,,0000,0000,0000,,and you test people on two occasions, and you just will see that their scores on time 1 and time 2 are identical: we are assuming that there is no genune change.
Dialogue: 0,0:21:21.95,0:21:26.96,Default,,0000,0000,0000,,If you do that, then you don't get regression to the mean, because the measure is perfect
Dialogue: 0,0:21:26.96,0:21:31.04,Default,,0000,0000,0000,,and if you test them a second time, they'll get exactly the same sort of score.
Dialogue: 0,0:21:31.04,0:21:39.78,Default,,0000,0000,0000,,And what you can see on the right hand side of the graph here, is people divided up according to the average score they started with.
Dialogue: 0,0:21:39.78,0:21:45.40,Default,,0000,0000,0000,,So we've put people into groups who were very poor to start with, who were medium, less good and so on.
Dialogue: 0,0:21:45.40,0:21:49.49,Default,,0000,0000,0000,,And these are just fictitious data made up to illustrate the point.
Dialogue: 0,0:21:49.49,0:21:56.42,Default,,0000,0000,0000,,So you just generate these numbers by saying, "We've got a measure that has this particular characteristic
Dialogue: 0,0:21:56.42,0:22:01.10,Default,,0000,0000,0000,,that if you measure on one occasion, on another occasion it remains pretty much the same".
Dialogue: 0,0:22:01.10,0:22:06.78,Default,,0000,0000,0000,,So then, you don't get regression to mean and you get people to maintain their position across time.
Dialogue: 0,0:22:06.78,0:22:10.01,Default,,0000,0000,0000,,So if you then see change, you can say "Well, it's genuine change."
Dialogue: 0,0:22:10.01,0:22:15.59,Default,,0000,0000,0000,,But most of our measures are not like that, most of our measures are not perfectly correlated:
Dialogue: 0,0:22:15.59,0:22:21.64,Default,,0000,0000,0000,,that means, you measure them on one time, and another time, and they actually change because of all sorts of things.
Dialogue: 0,0:22:21.64,0:22:29.72,Default,,0000,0000,0000,,Things like the particular test items that you're using, whether you are in a good mood, whether you've made a lucky guess in some items.
Dialogue: 0,0:22:29.73,0:22:36.12,Default,,0000,0000,0000,,And what you can see is that if you do that, that some people's scores go up with time, some people's go down with time.
Dialogue: 0,0:22:36.13,0:22:44.98,Default,,0000,0000,0000,,But on average, if you start with a low score, the odds are, you come a little bit closer to the average when you are tested on another occasion.
Dialogue: 0,0:22:44.98,0:22:47.75,Default,,0000,0000,0000,,If you start with a high score, you get a little bit worse.
Dialogue: 0,0:22:47.75,0:22:53.57,Default,,0000,0000,0000,,And this is nothing to do with genuine change: it's just to do with the fact that our measures are imperfect.
Dialogue: 0,0:22:53.57,0:22:59.98,Default,,0000,0000,0000,,And it has been argued that this is a major reason - all sorts of treatments that work (?) but don't really work.
Dialogue: 0,0:22:59.98,0:23:04.40,Default,,0000,0000,0000,,It's just that it looks as if you've seen a change, and you tend to attribute it to the treatment.
Dialogue: 0,0:23:04.40,0:23:10.25,Default,,0000,0000,0000,,Now, this sounded very depressing, because it means there's all sorts of reasons why we can see change,
Dialogue: 0,0:23:10.25,0:23:14.91,Default,,0000,0000,0000,,and how do we distinguish whether we've got a genuine change due to our treatment?
Dialogue: 0,0:23:14.91,0:23:21.47,Default,,0000,0000,0000,,But the fact is that you can control for most of these things if you do a study that has a control group.
Dialogue: 0,0:23:21.47,0:23:30.97,Default,,0000,0000,0000,,That's why those who are trying to do scientific evaluations are really keen to include control groups in studies and argue that they are essential.
Dialogue: 0,0:23:30.97,0:23:37.43,Default,,0000,0000,0000,,Because if you have another group of children who have been selected to be as similar as possible to your treated group,
Dialogue: 0,0:23:37.43,0:23:42.12,Default,,0000,0000,0000,,and are the same tests before and after the period where the treated group are treated,
Dialogue: 0,0:23:42.12,0:23:48.54,Default,,0000,0000,0000,,you are actually controlling for the effects of maturation, the effects of any other intervention they might be having,
Dialogue: 0,0:23:48.54,0:23:53.25,Default,,0000,0000,0000,,practice effects in particular, and also this dreadful regression to the mean.
Dialogue: 0,0:23:53.25,0:23:56.35,Default,,0000,0000,0000,,All of those things can be then taken into account.
Dialogue: 0,0:23:56.35,0:24:02.28,Default,,0000,0000,0000,,And in so far as they have effects, what you would expect to see is that you may see improvement in your control group
Dialogue: 0,0:24:03.21,0:24:06.88,Default,,0000,0000,0000,,because of these spurious things that we don't really want to see.
Dialogue: 0,0:24:06.88,0:24:13.16,Default,,0000,0000,0000,,And then you can say, "well there is more improvement in the treated group" (?) and it is that difference that is really critical.
Dialogue: 0,0:24:13.16,0:24:20.93,Default,,0000,0000,0000,,It doesn't actually control to use - if you have a group who have not been given any treatment - it doesn't control for placebo effects.
Dialogue: 0,0:24:20.93,0:24:27.92,Default,,0000,0000,0000,,So you've still got the problem that maybe your treated group will improve just because everybody is focusing on them with great excitement.
Dialogue: 0,0:24:27.92,0:24:32.78,Default,,0000,0000,0000,,But you could actually also have control for that, and it's becoming increasingly popular in this field
Dialogue: 0,0:24:32.78,0:24:38.34,Default,,0000,0000,0000,,to say that what you should have is a control group who are actually given some alternative treatment.
Dialogue: 0,0:24:38.34,0:24:42.78,Default,,0000,0000,0000,,So, for example, if you are interested in a treatment that might improve reading,
Dialogue: 0,0:24:42.78,0:24:48.26,Default,,0000,0000,0000,,you could either get children some standard educational treatment that they are getting anyway
Dialogue: 0,0:24:48.26,0:24:53.58,Default,,0000,0000,0000,,So if your claim is that you are doing better than a phonological-based treatment,
Dialogue: 0,0:24:53.58,0:24:58.20,Default,,0000,0000,0000,,you could have a control group given that treatment and see if you are making really that much difference,
Dialogue: 0,0:24:58.21,0:25:03.41,Default,,0000,0000,0000,,or you might prefer to say, "Well, let's treat something else, let's give children training in something completely different
Dialogue: 0,0:25:03.41,0:25:06.86,Default,,0000,0000,0000,,that isn't focused on reading, but nevertheless could benefit them in other ways."
Dialogue: 0,0:25:06.86,0:25:10.10,Default,,0000,0000,0000,,And then you can do that sort of comparison.
Dialogue: 0,0:25:10.10,0:25:18.03,Default,,0000,0000,0000,,So what about the DORE study, because I mentioned at the outset, when talking about this study, that they did have a control group.
Dialogue: 0,0:25:18.03,0:25:23.98,Default,,0000,0000,0000,,But so far, talking about the results, are only mentioned (?) the dramatic changes that they saw,
Dialogue: 0,0:25:23.98,0:25:27.12,Default,,0000,0000,0000,,which ignored the control group.
Dialogue: 0,0:25:27.12,0:25:34.59,Default,,0000,0000,0000,,The interesting thing is that when you look at their control group, it illustrates perfectly the importance of having a control group.
Dialogue: 0,0:25:34.59,0:25:39.81,Default,,0000,0000,0000,,So, on they dylexia risk's score, where a high score is bad,
Dialogue: 0,0:25:39.81,0:25:47.23,Default,,0000,0000,0000,,they had a change in the treated group, from 0.74 to 0.34.
Dialogue: 0,0:25:47.23,0:25:51.06,Default,,0000,0000,0000,,So you think: "Wow, that's great, these children's risks for dyslexia have really come down."
Dialogue: 0,0:25:51.06,0:25:56.44,Default,,0000,0000,0000,,In the control group, the average score changed from 0.72 to 0.44.
Dialogue: 0,0:25:56.44,0:25:59.04,Default,,0000,0000,0000,,Now, you could say: "Well, it's not so big a change."
Dialogue: 0,0:25:59.04,0:26:03.85,Default,,0000,0000,0000,,The trouble is, with groups this size, you can't really tell whether that's meaningful.
Dialogue: 0,0:26:03.85,0:26:07.97,Default,,0000,0000,0000,,But certainly, what is clear is that both groups improved on the dyslexia screening test,
Dialogue: 0,0:26:07.97,0:26:12.09,Default,,0000,0000,0000,,even though the control group had not had the intervention.
Dialogue: 0,0:26:12.09,0:26:16.61,Default,,0000,0000,0000,,So, it really illustrates the point very clearly that on a lot of these measures,
Dialogue: 0,0:26:16.61,0:26:21.04,Default,,0000,0000,0000,,everybody gets better, even if they are not treated.
Dialogue: 0,0:26:21.69,0:26:30.83,Default,,0000,0000,0000,,Now, if we look at the more precise data that they presented, they presented average scores on the different subtests from the dyslexia screening tests,
Dialogue: 0,0:26:30.84,0:26:36.09,Default,,0000,0000,0000,,I won't talk about all of them, I have got a fuller presentation
Dialogue: 0,0:26:36.09,0:26:39.41,Default,,0000,0000,0000,,where I do talk about all the different measures they use
Dialogue: 0,0:26:39.41,0:26:42.66,Default,,0000,0000,0000,,and I don't want to sort of be accused of delberately hiding things,
Dialogue: 0,0:26:42.66,0:26:46.29,Default,,0000,0000,0000,,but I think the tests that people would be most interested in are the literacy tests.
Dialogue: 0,0:26:46.29,0:26:52.75,Default,,0000,0000,0000,,So, you undertake the DORE treatment because you want to get better at reading and writing, if you are a parent of a dyslexic child, at any rate.
Dialogue: 0,0:26:52.75,0:27:03.10,Default,,0000,0000,0000,,So, looking at the results on those tests, what they found was that there were a total of 4 tests that had to do with literacy directly.
Dialogue: 0,0:27:03.10,0:27:07.78,Default,,0000,0000,0000,,And on one of those, it looked as if the treated group did better than the untreated group.
Dialogue: 0,0:27:07.78,0:27:13.85,Default,,0000,0000,0000,,But there is a problem with that, though, because on this reading test, the untreated -
Dialogue: 0,0:27:13.85,0:27:20.50,Default,,0000,0000,0000,,the control group are actually right on the average score for their age at the start of the treatment - at the start of the study.
Dialogue: 0,0:27:20.50,0:27:27.50,Default,,0000,0000,0000,,So, in a sense, you could argue, "Is there really room for improvement ...... (?) school absolutely average,
Dialogue: 0,0:27:27.50,0:27:35.42,Default,,0000,0000,0000,,whereas it just so happened that the children who had treatment started a little bit lower and therefore had more improvement.
Dialogue: 0,0:27:35.43,0:27:38.54,Default,,0000,0000,0000,,And their improvement was not dramatic, one has to say as well.
Dialogue: 0,0:27:38.54,0:27:43.22,Default,,0000,0000,0000,,Their school went up from 3 to 3.5, on a scale of 0 to 10.
Dialogue: 0,0:27:43.22,0:27:51.73,Default,,0000,0000,0000,,On the other measures, again it illustrated that on two of them, everybody improved, regardless of whether they had the treatment.
Dialogue: 0,0:27:51.73,0:27:54.90,Default,,0000,0000,0000,,And on the third one, nobody changed very much at all.
Dialogue: 0,0:27:54.90,0:28:03.23,Default,,0000,0000,0000,,So, this is not dramatic evidence of improvement but you could argue: "Well, nevertheless there was one measure that looked a little bit promising."
Dialogue: 0,0:28:03.23,0:28:11.78,Default,,0000,0000,0000,,But they then, in the second phase of the study, went on to give the control group the same treatment, and they published this in 2007.
Dialogue: 0,0:28:11.78,0:28:16.46,Default,,0000,0000,0000,,So we now don't anymore have a control group as everybody has been treated:
Dialogue: 0,0:28:16.46,0:28:21.59,Default,,0000,0000,0000,,one group early on, and the other group with a delayed time scale.
Dialogue: 0,0:28:21.59,0:28:30.07,Default,,0000,0000,0000,,And they presented the data between time 1 at the start of the study and right at the end of the study, when everybody had had this treatment.
Dialogue: 0,0:28:30.07,0:28:37.50,Default,,0000,0000,0000,,But when you look at the results there, it's clear that there really is a, you know, no persistent improvement in reading.
Dialogue: 0,0:28:37.50,0:28:41.72,Default,,0000,0000,0000,,In fact, the mean scores for the children having the delayed treatment on the reading test
Dialogue: 0,0:28:41.72,0:28:45.45,Default,,0000,0000,0000,,have now really gone down, rather than up, at the end of treatment.
Dialogue: 0,0:28:45.45,0:28:51.18,Default,,0000,0000,0000,,And the general impression, I would say, is that there is nothing very stunning going on here,
Dialogue: 0,0:28:51.19,0:28:57.62,Default,,0000,0000,0000,,certainly nothing that matches the description that you get on the promotion materials for the intervention.
Dialogue: 0,0:28:57.62,0:29:08.10,Default,,0000,0000,0000,,So, overall, I would argue that the evidence for gains associated with this treatment is really not at all compelling.
Dialogue: 0,0:29:08.10,0:29:17.26,Default,,0000,0000,0000,,First of all, the claims that are made for stunning changes are all coming from analyses where they didn't incorporate the controls
Dialogue: 0,0:29:17.26,0:29:21.82,Default,,0000,0000,0000,,and they just tried to argue that any change you see at the time must be due to the treatment.
Dialogue: 0,0:29:21.82,0:29:25.09,Default,,0000,0000,0000,,and not taking into account all these other factors.
Dialogue: 0,0:29:25.09,0:29:30.36,Default,,0000,0000,0000,,And on reading measures, where there was control group data available,
Dialogue: 0,0:29:30.36,0:29:37.10,Default,,0000,0000,0000,,there was an initial small gain in the treated group, but it wasn't sustained by the end of the study.
Dialogue: 0,0:29:37.10,0:29:41.36,Default,,0000,0000,0000,,So, it really doesn't look terribly promising.
Dialogue: 0,0:29:42.42,0:29:46.99,Default,,0000,0000,0000,,Now, this is why in general, I think it's true to say this:
Dialogue: 0,0:29:46.99,0:29:52.64,Default,,0000,0000,0000,,I don't know of anybody in the dyslexic community who is an advocate - in the academic community
Dialogue: 0,0:29:52.64,0:29:57.75,Default,,0000,0000,0000,,who is an advocate of the DORE treatment, other than people that are directly associated with the DORE organization.
Dialogue: 0,0:29:57.75,0:30:05.47,Default,,0000,0000,0000,,And so, the reason really is just that the evidence is not at all compelling,
Dialogue: 0,0:30:05.47,0:30:09.18,Default,,0000,0000,0000,,although the study was small and you could argue a larger study should be done.
Dialogue: 0,0:30:09.18,0:30:14.78,Default,,0000,0000,0000,,There is a real mismatch between the claims that are being made and the evidence that is available.
Dialogue: 0,0:30:14.78,0:30:21.72,Default,,0000,0000,0000,,But the interesting thing is also why so many people seem to nevertheless regard this as an effective treatment.
Dialogue: 0,0:30:21.72,0:30:29.50,Default,,0000,0000,0000,,If the testimonials are to be believed, there are many satisfied customers and happy parents who feel that their children have been helped.
Dialogue: 0,0:30:29.50,0:30:35.60,Default,,0000,0000,0000,,I think there is quite an interesting set of reasons why this may be so.
Dialogue: 0,0:30:35.60,0:30:41.40,Default,,0000,0000,0000,,And one is that there is a well known - in the psychological field - well known human tendency
Dialogue: 0,0:30:41.41,0:30:47.15,Default,,0000,0000,0000,,to think that something that you've put in a lot of time and money too, was worthwhile.
Dialogue: 0,0:30:47.15,0:30:55.57,Default,,0000,0000,0000,,It's called cognitive dissonance, and it means that if you've actually put in the effort, you tend to feel that there was an effect.
Dialogue: 0,0:30:55.57,0:31:01.53,Default,,0000,0000,0000,,You have to somehow resolve this sort of inconsistency, otherwise, in your mind.
Dialogue: 0,0:31:01.53,0:31:06.85,Default,,0000,0000,0000,,And this was beautifully illustrated, not by the trial of the DORE treatment, but in another trial,
Dialogue: 0,0:31:06.85,0:31:11.24,Default,,0000,0000,0000,,which was a very nicely well-conducted trial of something called Sunflower therapy,
Dialogue: 0,0:31:11.24,0:31:19.16,Default,,0000,0000,0000,,which is a rather holistic approach to intervention for dyslexia that involves kinesiology and physical manipulation,
Dialogue: 0,0:31:19.17,0:31:23.50,Default,,0000,0000,0000,,massage, homeopathy, herbal remedies and neurolingusitc programming.
Dialogue: 0,0:31:23.50,0:31:30.27,Default,,0000,0000,0000,,And there was a very rigorous study done for this, and what was interesting about it
Dialogue: 0,0:31:30.28,0:31:39.30,Default,,0000,0000,0000,,was that, like so many of these things, they didn't really find a lot of evidence for any better change in the clinical versus the control group,
Dialogue: 0,0:31:39.30,0:31:43.36,Default,,0000,0000,0000,,although, to some extent, both groups were securing (?) their schools were improving.
Dialogue: 0,0:31:43.87,0:31:49.84,Default,,0000,0000,0000,,What they did find, though, is that the children themselves had higher self-esteem if they had undergone the Sunflower treatment,
Dialogue: 0,0:31:49.84,0:31:56.30,Default,,0000,0000,0000,,but that also, 57% of the parents did think that Sunflower therapy was effective in treating their child.
Dialogue: 0,0:31:56.96,0:32:04.26,Default,,0000,0000,0000,,So there is a clear mismatch between what the study showed of the objective evidence on the children's learning difficulties,
Dialogue: 0,0:32:04.26,0:32:06.68,Default,,0000,0000,0000,,and what the parents actually thought.
Dialogue: 0,0:32:06.68,0:32:12.23,Default,,0000,0000,0000,,It is possible that this could be related to the fact that the children's scores did improve,
Dialogue: 0,0:32:12.23,0:32:17.60,Default,,0000,0000,0000,,but if you didn't know that the control children had also improved, you might attribute that to the therapy -
Dialogue: 0,0:32:17.60,0:32:23.57,Default,,0000,0000,0000,,but also to the fact that people were again being given a lot of encouragement,
Dialogue: 0,0:32:23.57,0:32:29.84,Default,,0000,0000,0000,,there was a lot invested in that treatment, and then there might well (?) have been some sort of sense of cognitive dissonance there.
Dialogue: 0,0:32:29.85,0:32:41.09,Default,,0000,0000,0000,,There's also a strong human tendency to be impressed by certain kinds of explanation that get more biological about dyslexia,
Dialogue: 0,0:32:41.09,0:32:48.68,Default,,0000,0000,0000,,particularly those such as the DORE treatment that get more neurological and claim to be doing something to the brain in treating dyslexia.
Dialogue: 0,0:32:48.68,0:32:54.01,Default,,0000,0000,0000,,There was a beautiful study done - published - in 2008, not on dyslexia,
Dialogue: 0,0:32:54.01,0:32:59.82,Default,,0000,0000,0000,,but just more generally on people's tendency to be impressed by scientific eyplanations.
Dialogue: 0,0:32:59.82,0:33:09.27,Default,,0000,0000,0000,,and what these researchers did was to give people explanations of psychological phenomena that are well known
Dialogue: 0,0:33:09.27,0:33:14.11,Default,,0000,0000,0000,,and they either gave them a good explanation or they gave them a not very good explanation
Dialogue: 0,0:33:14.11,0:33:21.34,Default,,0000,0000,0000,,that was more like just a re-description of the effect, and asked people to judge whether this was a good explanation or not.
Dialogue: 0,0:33:21.34,0:33:25.60,Default,,0000,0000,0000,,And what was fascinating about this study was that in general, people were quite good at doing this:
Dialogue: 0,0:33:25.60,0:33:33.31,Default,,0000,0000,0000,,even if they had no familiarity or background in psychology, they could distinguish a good explanation from a bad one.
Dialogue: 0,0:33:33.31,0:33:39.63,Default,,0000,0000,0000,,But what they found was that if they added some verbiage that just talked about the brain in various ways,
Dialogue: 0,0:33:39.63,0:33:45.24,Default,,0000,0000,0000,,and said, "This result came about because brain scans showed it", or "because we looked at the frontal lobes",
Dialogue: 0,0:33:45.25,0:33:48.37,Default,,0000,0000,0000,,people were much more impressed with the bad explanations.
Dialogue: 0,0:33:48.37,0:33:53.13,Default,,0000,0000,0000,,So a good explanation didn't get any better when you added all this neuroscience waffle,
Dialogue: 0,0:33:53.13,0:33:59.34,Default,,0000,0000,0000,,but if you added neuroscience waffle to a bad explanation, people thought it not so bad.
Dialogue: 0,0:33:59.34,0:34:05.95,Default,,0000,0000,0000,,And so, there is a tendency to be very impressed by anything that talks about, adds the brain in to an ........ (?) explanation.
Dialogue: 0,0:34:05.96,0:34:12.99,Default,,0000,0000,0000,,And I think this is used by people who then try and add spurious neuroscience sometimes
Dialogue: 0,0:34:12.100,0:34:18.17,Default,,0000,0000,0000,,to their accounts of their particular promissing theory.
Dialogue: 0,0:34:18.17,0:34:23.81,Default,,0000,0000,0000,,And it really is not - we shouldn't allow ourselves to be mislead.
Dialogue: 0,0:34:24.54,0:34:34.23,Default,,0000,0000,0000,,So I think, to sum up, there are a number of barriers to objective evaluation of intelligence,
Dialogue: 0,0:34:34.23,0:34:38.31,Default,,0000,0000,0000,,which, to some extent, functions about (?) our human condition,
Dialogue: 0,0:34:38.31,0:34:44.10,Default,,0000,0000,0000,,that we are not naturally good at taking in lots of numbers and looking at graphs
Dialogue: 0,0:34:44.10,0:34:47.76,Default,,0000,0000,0000,,and trying to sort of take into account alternative explanations.
Dialogue: 0,0:34:47.76,0:34:52.65,Default,,0000,0000,0000,,We tend to be impressed when we hear other people tell us that something has worked
Dialogue: 0,0:34:52.65,0:34:57.18,Default,,0000,0000,0000,,and it's hard - you have to almost guard yourself against the tendency to do that
Dialogue: 0,0:34:57.18,0:35:02.39,Default,,0000,0000,0000,,and to look rather for the hard evidence, to look for the actual numerical data.
Dialogue: 0,0:35:03.25,0:35:08.30,Default,,0000,0000,0000,,We have to be very careful when people start giving us explanations that have got a lot of neuroscience in them
Dialogue: 0,0:35:08.30,0:35:12.19,Default,,0000,0000,0000,,and check out, is this real neuroscience or is it just put in there to impress us?
Dialogue: 0,0:35:13.81,0:35:19.51,Default,,0000,0000,0000,,We have to be aware of the effect of cognitive dissonance and the tendency to believe some things
Dialogue: 0,0:35:19.51,0:35:23.87,Default,,0000,0000,0000,,simply because we have invested time and money in it
Dialogue: 0,0:35:23.87,0:35:31.40,Default,,0000,0000,0000,,and, most importantly, we have to bear in mind that there will be effects on children's performance
Dialogue: 0,0:35:31.40,0:35:37.97,Default,,0000,0000,0000,,of maturation, of our expectations, of just get practising on things,
Dialogue: 0,0:35:37.98,0:35:42.70,Default,,0000,0000,0000,,and there are also these dreadful statistical artefacts that can make it look as if a change has occurred,
Dialogue: 0,0:35:42.71,0:35:45.84,Default,,0000,0000,0000,,when it's really not particularly impressive.
Dialogue: 0,0:35:45.84,0:35:50.36,Default,,0000,0000,0000,,But I think, if one bears these things in mind, the bottom line is really:
Dialogue: 0,0:35:50.36,0:35:54.58,Default,,0000,0000,0000,,"Look for evidence from studies that have got adequate controls"
Dialogue: 0,0:35:54.58,0:36:01.51,Default,,0000,0000,0000,,and if you do, you'll be in - a standard, I think, by how far you can see improvements in children, even if they haven't had the treatment.
Dialogue: 0,0:36:01.51,0:36:06.14,Default,,0000,0000,0000,,and there are lots of things that will make things a lot better, just with the passage of time.
Dialogue: 0,0:36:06.14,0:36:11.34,Default,,0000,0000,0000,,But if you really want to demonstrate that there has been an effective treatment,
Dialogue: 0,0:36:11.34,0:36:15.80,Default,,0000,0000,0000,,you do have to show an improvement relative to a control group,
Dialogue: 0,0:36:15.80,0:36:20.96,Default,,0000,0000,0000,,rather than just, somebody started out not so good and is now a little bit better after the treatment.
Dialogue: 0,0:36:20.96,0:36:29.95,Default,,0000,0000,0000,,I hope that that might give you some useful indicators when trying to look at new treatments that are out there and on offer,
Dialogue: 0,0:36:29.95,0:36:35.11,Default,,0000,0000,0000,,and for a more detailed account of some of this work, there are various -
Dialogue: 0,0:36:35.11,0:36:40.10,Default,,0000,0000,0000,,there is a powerpoint presentation with notes on my website on this topic.