Randomised Controlled Trials (RCTs) - Method and Practice

0:00 um i'm helen lawson tazo's research programs manager and i'm just facilitating today's webinar um presenting today we have rain sherlock who is our evaluation manager and sarah chapel who is our senior research officer and without further ado i will hand you over to rayne super thank you helen um so yeah welcome everyone good to have lots of you here and today we are going to talk to you through kind of an introduction to randomized control trials both the theory and the background and then also a practical example so you can see here how our session's going to run we'll kind of done a quick welcome and

0:45 intro i am going to cover the theory and background so i'll speak for kind of probably 30 35 minutes and then hand over to sarah for the second half of the webinar to talk you through the summer schools project and some very early findings that we have and then as helen has mentioned we'll have lots of time for q a at the end so please do drop your questions as you go or drop them down and we'll we'll come to them in a little bit so um i think lots of people would have seen this slide and kind of heard the preamble before um but i do like to start these webinars with kind of situating us within the what we call the evaluation cycle so at taser we think of evaluation as kind of iterative or secular

1:30 with each of these four phases kind of informing the development of the next phase so we start with diagnosing the development of a theory of change we move on to plan where we think through our kind of research questions and the methodologies that we're using we then think about how to measure those um questions so collect data and the type of analysis that we use and then of course in the final phase the sort of reflection and dissemination of our findings today's session is situated within plan so we're going to be talking about randomized control trials which are one methodology available to us one approach to evaluation that is an option as we plan our research for evaluation i also often use this slide to help kind

2:16 of situate where we are in terms of the evidence landscape so it's worth noting that the taser aligns to the ofs the office for students standards of evidence um which you can see here on screen in this little diagram and for those of you who also align to the ofs or have a kind of access and participation plan i think it's worth kind of noting where we are so we've got good type one evidence i think many people are using kind of theories of change we have good narrative evidence we're able to describe the interventions that we're doing the programs that we run and what we intend their impact to be i think we have some type 2 evidence which we call kind of more empirical research we're getting better at collecting data and estimating

3:02 correlations um we're getting better at kind of thinking about the counterfactual and comparison groups and then i think we're basically still missing the kind of type three or causal evidence within the he sector in our kind of current landscape so um what we would call causal evidence or research and evaluation that allows us to draw causal inference and this is really where rcts come in randomized controlled trials so they are a methodology that typically allows us to get at causality this kind of cause and effects relationship that said um i've added this slide because i also think it's worth pausing and reflecting on the fact that rcts are just one tool in our toolbox or one kind of

3:49 methodology available to us um there's a number of approaches that we use for evaluation and for research and many of them complement each other and kind of can be used alongside each other and i think as researchers we need to be kind of adaptable to pulling in what's most useful at any given time so randomized control trials serve a purpose they're very useful for this kind of call of attribution causal research um but we need to be flexible and open to using different methodologies and particularly with randomized controlled trials we need to think about kind of pairing them with or also using qualitative research so perhaps as part of an implementation and process evaluation which we sometimes call ipe

4:34 um and then of course theories of change we start most evaluations with a theory of change and would also couple that with an rct so theory and background to randomize control trials i'm going to start with a very brief kind of whistle-stop tour of the history um i think this might be familiar to some people they seem to be names that kind of sprout up a lot but often rcts are tracked back to about 1747 when james lind who was a scottish doctor um started trying to treat scurvy so he randomized people into different treatment groups and he tried to understand what might remedy scurvy with things like um i think kind of sea water nutmeg

5:19 maybe even cider and then obviously citrus fruit which we now know is a treatment and prevention for scurvy another significant point that science often acknowledges is the i think 1863 austin flint he was an american doctor i believe and he was one of the first if not the first to start using the placebo so again would randomize people into kind of treatment control and placebo um and start yeah the lich trek on placebo tracks back to austin flint um we then come to 1943 so this is when the uk medical research council um conducted its first double-blinded comparative trial with the treatment and control so was beginning to see the idea of randomization into different groups becoming

6:05 more sophisticated and how it's done and also more widespread given the kind of medical sphere at the moment um by 1943 we haven't quite got as far as social science but if we then fast forward to where we are today um we have i think over a million and maybe even more by now uh registered clinical trials in the uk and obviously we're increasingly using randomization and rcts for social sciences so we see it in education social work healthcare etc um so to look at a couple of examples of where rcts have allowed us as researchers or social scientists to observe causal evidence of an intervention working well there are loads of examples um of this

6:52 particularly in the u.s literature and i've tried to just pick a couple that feel relatable to us and our work in he so one example um is a u.s study focused on expanding college opportunities and particularly for high achieving but low income students um so i think previous research or kind of existing literature has suggested that a kind of majority so a large proportion of high achieving low-income students were unlikely to be reached by traditional methods for informing students about how to make an application how to finance or fund an application etc so this study used a randomized control trial to test the impact of personalized information on the college application process so

7:38 allocated students obviously into kind of intervention or treatment and control and students who were in the intervention group received customized so personalized information as part of a portal helping them understand how they personally could apply to or put in an application for college um and then the control group just received the usual information so it wasn't personalized or tailored in any way um and what we saw here is that high the high achieving but low income group of students or our kind of target audience submitted 19 more applications and those applications were actually accepted into 12 more colleges than those that didn't receive the information so this is just an example of how an rct um has allowed us to

8:24 create a kind of causal understanding of how the personalized information contributed to a change in outcomes so applications number of applications and also submissions into into he into college the second example is an rct that was run in india again on educational outcomes so this was a partnership with an education um ngo in india and again researchers i think in the us and they wanted to trial a kind of low-cost what they call a remedial education program it was actually a tutoring program in a school in india so they had 200 schools on board again they randomly allocated um a tutor to schools that were in the intervention group and then schools who were in the

9:12 the control group didn't receive the tutor um the tutors i guess because it's helpful information the tutors were um like teaching staff so not necessarily like peers or students that were already at university but they were kind of professional teaching stuff and again the results of the tutoring program showed that the students who received a tutor had an increase in attainment of level three grades in india so they took level three grades for the intervention group and a control group and those who were in the intervention and received a tutor have a significant effect and this program i think was actually deemed so successful that they have rolled it out across different districts in india and it's now become a kind of national tutoring program

9:59 so again just a way or an example of how rcts have allowed us to better understand the relationship between our kind of our activity our program our intervention and the outcome that we're interested in i also want to spend a moment here thinking about studies or examples of where randomized control trials have allowed us to observe an intervention that does not work so these are two more well-known or kind of widely discussed studies or trials so the one on the left here is called scared straights and this was a program run in the u.s that organized visits to prisons or prison facilities for children or young people that were that were at risk of offending or criminal activity so the idea being if we've got students

10:44 who for whatever reason give early indicators that they may end up offending we introduce them to the prison system to the facilities and the idea is that they will kind of experience what it's like and then it will deter them from any criminal activity in the future um so researchers set this up as a randomized control trial people were randomized into an intervention group and a control group obviously those in the intervention did visit the prison facilities and those in the control did not but actually what it found is that visiting the prisons was not only not going to deter people from committing crimes so it didn't have the intended impact that it hoped but actually those in the country those in the treatment group were more likely to

11:31 end up offending later than those in the control group so it backfires um in terms of what the intended impact was meant to be what we observed was actually the opposite another example of something similar but slightly less clear results is the unconscious bias training again some people might be familiar with this but it is a kind of once-off training course or training activity the idea being that by going through an unconscious bias training we're able to help sort of get humans as irrational thinkers to overcome unconscious bias so when we sift cvs as part of our recruitment process um or we are doing research into different populations there is evidence

12:17 to suggest that an unconscious bias can take over our kind of locus of control so we might unconsciously bias based on gender or kind of other factors or variables um and what this what the findings from rcts run on this trial has found is that actually giving people information about their unconscious bias isn't enough to combat the effect so this training doesn't have a an impact on whether or not people become less biased um i think previously this training has been used quite a lot across government and different sectors but due to the evidence that's emerging now based on these trials um the uk government's actually decided to stop using it because it's it's not a good use of kind of time money or resource given that it doesn't have an impact so what i'm trying to

13:05 point out in all of these examples i guess is that rcts allow us to test hypotheses so we have a kind of theory behind the change that we expect to see and by running an rct we have the confidence to kind of estimate a causal effect so does the intervention impact the outcome or the behavior that we're interested in um and you can see examples of where it has and examples of where it hasn't and so it's really important that we understand this causal relationship or we kind of build this evidence bank so that we can make more impactful decisions on the type of activity or programs that we run but also to save money time and resource right there's just no point doing something that's costing a lot and not having the impact that we expect it to

13:52 alrighty so coming to the kind of basic rct so what in all of these examples that we've just gone through what is it that we're actually talking about and how does it work so what you can see on screen here is the most um kind of basic concept of an rct i guess so we take our population which is the big circle on the left hand side and is the kind of sample of people that we're interested to examine or study or investigate and we randomly assign them to two groups so the intervention group at the top here and then the control group at the bottom and then over time we will measure the key outcome that we're interested in or observe the behavior that we're interested in between the two groups

14:38 and we will be able to establish whether there is any difference to the main outcome between the intervention and the control group so sarah is going to talk you through a summer school's example in lots of detail later but just quickly to kind of situate us here the the sample population on the left would be everyone that we're interested to to examine or study so the whole kind of sample we would then randomly assign people either to receive the summer school at the top or to not receive the summer school at bottom and then on the far right hand side we would we would observe them over time and try to understand how many of those students end up in higher education so the little red dots would represent the people who do and the green dots would represent the people who don't now one of the key things that the

15:25 control group gives us or the kind of unique thing about an rct is that it allows us to observe what we call the counterfactual so the counterfactual is kind of simply what would happen if we hadn't done the thing if we hadn't implemented the summer school implemented the intervention changed the information et cetera so it's just what would happen if the the condition of the intervention wasn't present um and by introducing the control where we're able to observe that in a way that some other methodologies don't allow us to um so statistical power i i guess a kind of um caveat up front is that we aren't going to go into loads of detail on power i

16:11 know we we kind of reference it in previous taser webinars um and i'm definitely promising a whole session on power at some point because it does require a full hour really um however we can't really give a whole session on rcts and not mention power so i'm going to give you a quick overview and hopefully it's enough to get us through today but don't worry if it does also feel a little overwhelming because it it is something that should and will have a full hour at a later point um essentially to be able to conduct a randomized control trial we need to think about statistical power um and statistical power is a measure of likelihood so it's just or it's kind of essentially the likelihood that a researcher will find statistical significance in a sample

16:56 if that effect does actually exist in the real world the real population so i sometimes think of statistical power or power as the kind of the muscle that we need to be able to to detect a difference between the the two groups that we're studying to the treatment and control and ensure that we don't miss finding any differences so accidentally think that the effect doesn't exist just because we've we've missed it in the group that we're observing so what you can see on the slide here is that power is a function of three primary factors at the bottom um and then one secondary factor so the primary factors are obviously sample size um so how many people are we studying the effect size so how big is the effect or

17:44 the impact of what we're trying to implement as an intervention or activity and then what we call the significance level or the p level so the p value um and then of course we also need to understand the power of the statistic used so are we using a t-test are we using a regression are we using a chi-squared test and what's the power of that now in our world because we're working with with real-life students and we can't kind of endlessly recruit to our trials we usually have a kind of set sample of the students that exist in a context that we're observing um we can't just increase our sample as we like and recruit to our trial until we reach the number that we want in the way that some social scientists or social researchers can

18:30 because of this power becomes quite a serious consideration when we're setting up trials and this is definitely relevant to the example that sarah will talk through later with summer schools so we need to have an adequate sample size um as well as all the other factors on this slide obviously to be able to run a trial properly or in a robust way and being underpowered is risky um and definitely not advised because it means that we're vulnerable to what we call a type one error which is also sometimes called a false negative and quite simply means that we fail to detect a true effect or a true kind of difference in our outcome when actually that difference does exist and that's quite simply because we just don't have enough people in our sample to observe it so we typically power our trials to 80

19:18 or 0.8 don't worry you don't need to kind of remember that now and really what this means is that if we were to run a trial over and over again one in five times we would still not find the effect that we're hoping to observe um because we're only powered by 80 and 20 is still not accounted for if we reduce the power so if we were to power ourselves to like 50 or 40 we increase the likelihood of not detecting that true effects um and observing a false negative so you can see why in order to observe an effect that does exist in the largest in the larger kind of wide population we we need to be well powered essentially that means we need to have a sample size that's big enough

20:03 i hope that doesn't leave people too confused um but also if it does don't worry as i said we will share readings later happy to take questions on it and we'll also do a proper session on this at some point in the near future um so let's think through a couple of reasons or the different explanations behind why we randomize and why randomization is useful so the first is um because randomization allows us to control for individual differences so by randomly allocating people to an intervention or a treatment and control group we can say that any difference in the outcome or the behavior that we're observing between the two groups is

20:49 caused by the intervention so the thing that we did or the thing that we changed um is what caused the outcome to change and that's quite simply because everything else between the groups is on average the same and by randomizing into treatment and control we have the confidence that they are on average the same this is in line with the law of large numbers which comes back to sample size i guess essentially it works with big populations so if we have lots and lots of people um small fluctuations or any kind of small differences are going to be ironed out over the course of the whole population because there are so many people if we were studying really small groups so kind of two or three people in treatment or two or three people in

21:35 control um any of those small differences or small variances wouldn't be ironed out so they would have quite an impact on on the outcome that we're interested to measure so essentially by randomizing we can assume that the treatment and treatment and control are on average the same and that is the beauty of randomization and one of the kind of key things the second reason here is that randomization allows us to control for external or sometimes what we call unexpected factors in the control group so again when we randomly allocate people from our sample to both intervention or control we can have confidence that the groups are similarly vulnerable to any anything that's happening in the external world or the external environment anything that might

22:21 impact the outcome that we're interested in over time i was struggling to think of an example here and i hope this one is kind of intuitive and makes sense but um essentially if you imagine that we are running a trial that promotes healthy breakfast for school-aged children so you can see um those in the intervention group on the on the left hand side here would get kind of information and training on what a healthy breast breakfast can do for our children but at the same time and coincidentally we didn't know this there's a sort of national tv and radio campaign going on that's also also promoting promoting healthy eating um for school-aged children but also for the population general so there's just a big kind of push for healthy eating

23:08 well because we will randomly allocate people to our intervention and control we can assume that the likelihood of them seeing or hearing the national campaign about healthy eating is the same for both groups now if you imagine for a moment that we hadn't randomized so maybe we had let parents and school-aged children self-select and kind of join the trial if they wanted to it it's very possible that we end up in a scenario where those who opted into the trial are already quite health-conscious maybe already seek out information about healthy eating they might actually be involved in that national campaign that's going on because they've already got an interest in healthy eating and so this is likely to kind of bias the group that end up in the treatment

23:54 essentially what i'm saying here is that having a control group and the nature of randomly allocating people to these two groups allows us to neutralize external factors so we can assume that both groups are equally vulnerable to those external factors that might happen over time the third and kind of final key reason then for why we randomize is that and i should say when it's applied correctly and again linking back to power but assuming that it is applied correctly and it is well powered it allows us to estimate causal effect of our interventions um so we have activity or program a and we can say with confidence that it has a causal link to outcome or behavior b so this

24:40 ability to estimate the causal effect or sometimes we would refer to that as infer causality it really means the same thing that is due to the random assignment of study participants so all similarities and differences between groups are assumed to exist by chance and the the reason that we use randomization is to account for these confounding variables so both measured and unmeasured variables are accounted for when we use randomization so a little a little kind of um i guess refresh or recap here um what does all of this mean in terms of us being able to answer our research questions or our

25:25 evaluation questions so the first cycle on the left here do rcts tell us what works the simple answer is yes they do so rcts are a kind of what works solution so by using an rct we can establish the relationship between the intervention x on outcome or behavior y and as we saw a moment ago rcts allow us to estimate that causal effect do rtts tell us why the intervention works so i think there's a bit of a question mark here essentially sometimes they do or they can so we can design rcts in a way that help us understand why things work we do this by using interactions or different conditions that isolate a

26:10 certain mechanism but broadly our cts are best suited to tell us what works and then finally do our cts tell us how it works i think the simple answer is no they don't um some people may argue that and may know more than i do i think with kind of very elegantly designed trials with different interactions and controls you can begin to get at the how um so yeah feel free to drop information in the chat or come back to us later in the q a if you know more about that um but essentially rcts are not well suited to telling us how it works and so again i guess what i'm trying to illustrate here is that as researchers and evaluators we have this whole toolkit rcts are one tool in that

26:58 toolkit they're very good at answering what works they're less good at answering the why works and i would argue that it's difficult for them to answer the how it works um and this is where we need to think about the other methodologies that we use so i briefly mentioned implementation and process evaluation which is something that we definitely strongly recommend running alongside an rct and that the ipe the kind of abbreviational acronym will help us understand the why and how in a way that the rct doesn't or can't so um it wouldn't be a session on rct's without talking and talking quotes some of you might have might know of torksen you might have read there's an excellent book called i think it's just called

27:43 designing randomized trials which i frequently revisit and revisited when when preparing for this session um but what they say here and i have to say i think i agree is that the beauty of randomization is that it controls for the known measurable variables the known unmeasurable measurable variables and most importantly the unknown and unmeasurable variables that could affect our outcome um yeah definitely recommend visiting their book if you haven't done so to pause and reflect on the basic kind of pros and cons or benefits and limitations of a basic rct um i should say this isn't the kind of wrap-up we're then going to go through quite a lot of detail all the different ways to randomize

28:29 but in terms of the the most basic design that we've talked through there's obviously a number of benefits some of these we've got at already so it's the most rigorous method for addressing the counterfactual it allows us to observe what would have happened if we hadn't implemented the intervention or changed the thing that we're that we're working on it's an ideal tool for causal inference or this idea about kind of causal effect so due to the ability of an rct to control for the unknown and unmeasurable variables we're able to deter causal inference it's helpful in determining whether or not an intervention works so rcts allow us to establish how big the effect is and whether the value add is worthwhile

29:16 so um reaching ourselves in summer schools again if we were to implement a summer school and observe that it only got two more students into higher education um we'd be able to basically evaluate for ourselves whether or not the value ad is worth it whereas if we were to to run a summer school and it got 200 students into education um again we'd be able to do a kind of value-adds estimation and the the good thing or the kind of beauty of our cities is the size of that effect is observable um experiments typically have high internal validity so the extent to which you can be confident that a cause and effect relationship is um not explainable by

30:02 by kind of external variables or confounding variables means that we have high internal validity um and then it also allows us to address the selection bias so people are not self-selecting or kind of self-opting into the trial they're being randomly allocated which means that we account for that bias that sometimes exists in other methodologies um as with all tools in our toolkit rcts also come with a number of limitations so there are ethical considerations around not giving the intervention to to all groups so the control group typically don't get that intervention there are some ways to to kind of manage or navigate this and i again i think sarah is going to come onto this in a little bit more detail later but often we have programs or

30:47 activities that are over subscribed so there's only a set amount of places anyway and some students won't be able to take part randomization is a good way to allocate places in this scenario rather than a kind of first come first serve basis we also have something called equipoise in ethics uh quite simply equity poise is the kind of idea that we don't actually have evidence on what works yet so there's no real understanding of whether it's the treatment or the control that's better and so therefore we're just randomizing people into two groups um without knowing whether or not one group is going to be missing out on something that might work which helps us kind of dig into and understand the ethics in a bit more detail rcts can be time consuming and expensive definitely a limitation that's that

31:33 needs to be considered um we also have something called the hawthorne effects and linked to that is kind of internal validity concerns so if people know that they're part of an rct it can affect their behavior they think they're being studied or examined and that's definitely something we need to bear in mind um it can be difficult to randomize participants oh zed has crept in there i've gone through and tried to combat the americanism but you might you might see a few zeds um so we will come on to this in much more detail the kind of difficulty with randomizing we're going to spend some time on that in a moment um and then of course the the inverse of the internal validity issue is the external validity consideration so um sometimes an intervention works in one context but it doesn't necessarily apply to all contexts and we need to be

32:18 able to continuously test our mechanisms to make sure that we account for any challenges with external validity we've we've said this before but again i guess another limitation is that rcts alone are often not enough so we need to be able to run an implementation or process evaluation or some poll work alongside an rct to help us get at that how and why that we looked at a moment ago so ways to randomize um there's a number of of different strategies for randomization that we can use in different contexts and we're now going to dig into these in a little bit more detail so the first is true randomization or sometimes called simple randomization

33:03 quite simply means that participants have an equal chance of being allocated to either the intervention or the control group so we have a sample population and everyone is just randomly allocated to either of those two groups that's the kind of exactly what it says on the tin version and what we looked at a moment ago in the example we can also use something called sequential randomization so this is an option when true or simple randomization isn't available to us again i've probably got some funky examples in here just trying to to think of simple ways to help illustrate it um to to everyone so the thing that i thought of here and i hope it makes sense is um imagine that we wanted to test the impact of giving people a brochure or a

33:49 leaflet on supermarket deals um or discounts as they enter the store it would be quite hard to randomize people using simple or true randomization because we don't know when they're going to turn up we don't even know who they are before they come to the store we don't have any information on them so we could use sequential randomization here to quite simply give the first person who enters the store gets a brochure the second person doesn't the third person does the fourth person doesn't the fifth person does etcetera so on and so forth um and by using sequential randomization we're trying to basically mimic true randomization because that isn't available to us in this exact example um very similarly and linked to that is blocked randomization so it's the kind of same logic applies but rather than

34:36 doing it one on one off one on one off we just do it in blocks so maybe the first three people get the leaflet or brochure and the the next three people don't the next redo etc you can see here the introduction of something called a counter balance check or that we can kind of counterbalance the randomization this kind of quite simply it should help us improve the rigor of the randomization given that we're trying to mimic true randomization so it helps us control the sun some of the unobservable factors um and the example i could think of here was imagine that we're using age as the sequential factor so 20 year olds get the 20 year olds go into the control 21 year olds go into the intervention 22 year

35:22 olds control 23 year olds intervention etc etc um and in this example we could introduce a counter balance check with each decade so rather than 19 29 39 year olds all receiving the intervention we can counterbalance it so that people in their 20s get the intervention when they're an odd number and people in their 30s get the intervention when they're an even number the point here is that we're trying to minimize what we call a systematic bias so it might be the case that there's something unobservable that means that 19 29 and 39 year olds which are all ages that are kind of before a landmark birthday so before the turn of a new decade it might mean for some reason there's there's something that makes those people susceptible to our intervention

36:08 um something unobservable that we can't account for by deliberately splitting these up so making it that 19 year olds do get the intervention but 29 year olds don't we are introducing the counterbalance and helping control that unobservable factor so now we have a little bit of a problem um imagine that we have a hundred people the eagle eyed of you will spot a theme here which is that there are rarely as many dots as i'm trying to say there are quite simply couldn't face a hundred dots on one screen um but let's imagine there's a hundred dots here and a hundred people and all of those people all of those dots fit into different groupings or categories so on the one end of the spectrum we've got the very high income earners these

36:54 little red dots and on the other end we've got what we call the very low income earners so this example of grey dots we obviously also have three categories in between if we use simple randomization or what we call true randomization a moment ago and we allocate allocate a hundred people into groups it is possible that we end up with groups that are not representative of the different categories of income learners so if we think that income is an important variable um and that it might impact the intervention that we're observing or the intervention that we're hoping to implement we have a challenge here because we need the groups to be representative of all the different income earners in order to ensure that the groups are

37:40 kind of on average the same so this brings us to something called stratification which is basically a solution to that challenge um so simply what it means is that first we gather participants into categories or or into groupings of income earners and we call these um so stratas as part of the stratification so you can see here we've got very high income earners all the way through to very low income manners and then we randomize one strata at a time we randomize them equally into the intervention and the control group so we first stratified to ensure that high income um people are evenly distributed into the intervention and control group so the top row here shows

38:25 how high income people may be concentrated in group a if we don't use stratification to randomize one strata at a time and basically what it means is that we're assigning participants at random to the intervention control groups within each given stressor so we might also have a strata around um kind of gender or heights or pre-he attainment there's a number of things that we could use as stressor and include in our stratification um it could also be based it's obviously quite easy when it's um kind of categorical in terms of the different categories of income earners it could also be a discrete category so made up of a continuous variable we might have

39:10 um those that sit above or below the medium height for instance or those who attend schools with above or below the average number of students eligible for free school meals um but essentially we do this we use stratification to ensure that what we call covariates or kind of key characteristics of balance between the treatment and the control it's worth noting that stratification helps us understand three things or it serves three kind of key purposes so the first is obviously to ensure that any differences in outcome between treatment and control are balanced across groups um the second is that it means that small groups in particular have access

39:56 kind of equal access to treatment so thinking of an example there we might be particularly interested in mature students for instance and we don't want the mature students to be unequally in a treatment or control so we stratify to make sure they have equal access um the second or sorry the third reason is to allow for subgroup analysis so it means that we can look at sub samples within our wider population that we're particularly interested in again mature learners or um perhaps children with experience of social care any subgroup that we're particularly interested to examine linked to stratification is something called balance checks um again sarah we'll come to this in more detail later there's a lot of that um essentially the goal of a balance check is to indicate

40:42 whether the randomization has produced a balance on all of these characteristics um so we might decide that gender is an important covariate in one of our trials and therefore we want the kind of male female split of pupils to be balanced across treatment and control and by doing balance checks on each of those covariates so gender income children children with experience of social care we can make sure that it's it's even down so now let's say that our our hundreds dots that that wasn't 100 dots is 100 000 dots that also isn't 100 000 dots and imagine that we have a really cool intervention so something that we're that we think is excellent we're sure it's gonna work maybe we've done prior research um and we're really excited to

41:27 try that it could it could also be um something that's really risky so it might be a kind of medical intervention maybe some a covered treatment that we think it carries some risk um the third thing i guess is that it could also be expensive so it might be that we plan to pay students 50 pounds to attend an information advice and guidance session and because we're paying them it every single student is it like racks up and comes quite expensive quite quickly um but for now let's imagine that it's a not so expensive not so risky but just a cool intervention so what we're trying to say is that we have an intervention that we think is really going to work um and and and then a kind of condition

42:14 that we think is less likely to work so here we have an in-person tutoring program versus an online tutoring program and we're pretty sure that the online tutoring program is going to work so as i said perhaps we've done research on this before or perhaps there's kind of existing evidence to suggest that those who attend the online tutoring program are more likely um to have an increase in grades or attainment i should say this isn't this isn't a real example this is kind of made up so i don't think we have conclusive evidence on online versus face-to-face tutoring but let's imagine that we do um so what we what we're kind of need to consider here i guess is that because we're pretty sure that online tutoring is going to work 50 000 students seems like quite a lot

43:01 of people to get an intervention that we're actually sure not sure is the best option well as i said on my previous slide it might be linked more to risk or expense so 50 000 people is a lot of people to pay to be in control or 50 000 people is a lot of people um to give an intervention that we think carries some risk so it could be on any of those kind of factors again we have a solution to this kind of challenge and this is what we call unequal randomization so one of the things that we can do is change what we call the allocation ratio when randomizing so rather than everyone having a 50 50 chance of being in a treatment or a control group next in this example here we're using a kind of

43:46 nine to one allocation ratio so nine people are going to receive the online t spring program for every one that receives the control so the the face-to-face tutoring program equally we can use um an unequal randomization to reduce cost or risk as we've discussed so this visual is just to help present what we mean by unequal randomization and it's worth this worth noting that this is also sometimes called powering down a randomized control trial so we basically reduce that um control group but as you can see here we've got our regular sample population on the left and when we allocate treatment and control we essentially allocate more people to treatment than we do to control because we have the power and

44:32 the numbers to be able to do so and we think it is for some reason beneficial to have more people in the treatment it is worth noting here that we can also have the inverse inverse converse of this so we might have a scenario where we are running a program a tutoring program a summer school any intervention um that has limited spaces so we can only randomize into the treatment as far as how many places we have so if we have 100 places we can have 100 people in the treatment and essentially everybody else who applied who was in our kind of sample population would then go into the control so it might be that we end up with a bigger control than we do treatment but as long as we have allocated the kind of ratio when randomizing and as

45:18 long as we're powered to do so it's not a problem and our unequal randomization can take place so our last section um on the kind of theory before i hand over to sarah and again i guess it wouldn't be a taser webinar without one of these questions so do feel free to to kind of drop your thoughts on the chat but what i would like to know is what all three of these things have in common so on the far left we've obviously got cockroaches in the middle we've got coconuts and on the right hand side we've got a galaxy can anyone tell me what you think these three things have in common bearing in mind that we're talking about

46:03 rcts any thoughts in the chat let us know i asked a friend this over the weekend when i was trying to finish off these slides and they said uh you don't want them to fall on you which i actually think is a really good answer i wouldn't want a cockroach a coconut or a galaxy to fall on me but it's not the answer we're looking for the color brown we've had that as a suggestion the color brown is another excellent guess yeah maybe there's multiple correct answers it's not the answer we're looking for any that's probably very accurate a coconut i think definitely a cockroach in a

46:49 galaxy well the answer i'm looking for is that they all come in in my mind famous clusters so we have cockroach clusters for any harry potter fans a type of confectionery in hp we have coconut clusters one of the kind of vegan superfoods that i see all the time and then we have clusters of galaxies we've struggled a bit to find a good third cluster but i do think galaxy's common clusters unsurprisingly this brings us to cluster rcts so the kind of final area of rcts that we're going to look at um so again let's start with a kind of scenario or a little thought experiment

47:37 what would so what we're looking at here is the impact of happiness or the impact of presence on happiness there is actually a wild literature on this that i've discovered again people might know more than i do about it but um let's imagine that we want to understand what happens if we had a family with multiple children in it and we gave one child loads of presents to try and understand the impacts of presence on happiness so my assumption or essentially i think the treatment so giving one child lots of presence is likely to spill over and impact the other child so by giving a present to the kids the the child on the left here um it's likely that the child on the right it looks like his brother is going to

48:22 get stuck into the fun might help open the presents he's almost certainly going to end up playing with the presents or whatever and that is inside it um and it's quite hard to isolate the effect of that present on the one child that we're looking at so this brings us to something that we call the assumption of independence and there's two parts the assumption of independence so the first is that each observation is independent so each participant is only treated once an example that i thought of here was um again in a supermarket not not sure why my head's going there all the time but imagine that we're trying to test the impact of a public service announcement on a kind of supermarket tannoy um and we randomize the time that we do this so we make an announcement at 10 we

49:08 make an announcement 11 12 1 whatever the problem with this is that it's it's very possible that someone will be in the supermarket at 10 and still there at 11 doing their weekly shop and so they would essentially be treated twice by by the announcement that's being made and this would violate the assumption of independence so people only being treated once um the second thing that we need to consider under this assumption of independence is the spillover and that's really what we're seeing with the children and the presence here so the impact of the intervention on one person is going to spill over and impact other people who are close in kind of proximity to them and therefore we don't have a true control so back to our example of the the

49:54 children and the gifts if we give one child a gift the other child's gonna get gonna get stuck in and enjoy the fun um and so what can we do to make sure that we kind of account for or combat this spillover well the the kind of most feasible solution i guess is that we randomize at the level of the family unit instead of at the level of the child so rather than giving one child in the family presence we put that whole family into the treatment or control and both both children as a family you the units either do or do not do not get the presence we can also do this when thinking about schools as as examples so let's say that we're looking at schools and we're trying to

50:40 improve teaching quality within a school um our unit of measurement can't be the school people because a teacher having an impact on one school is like sorry on one pupil it's likely to pillow spill over and affect another people um so what we end up doing is randomizing at the school level so you can see here we've got school one two three and four and each of these are randomly allocated to either be treatment schools or control schools although we're randomizing at the school level we still measure at the individual level so we still look at the impact of the individuals and perhaps on entertainment or wellbeing or quality of education so changing the unit of randomization

51:25 from individual to the cluster whatever that cluster is and in our scenario now it's just schools when thinking about cluster rcts it's really important to think about variation so here you can see that one cluster has more variation at the individual level so you can see the box on the right has different colored dots or more variants in terms of how the dots look and it's possible for a cluster to have either low variation and it's also possible for a cluster to have high variation at the individual level so individual variance is something that's really important to cluster rcts but similarly so is group variance so

52:11 on the left hand side now we have four classrooms or clusters that look quite different to the four classrooms or kind of clusters on the right hand side really what i'm trying to illustrate here is that it's it's possible to have variation both within clusters but also um between separate clusters so within the cluster um the people might not not necessarily be the same so we might be looking at a school classroom but all the students all the pupils within that classroom are quite different um but equally we might have schools that at the meta level look quite different or quite varied and then so when thinking about kind of individual rcts or rcts that are randomized at the individual level we don't typically pay so much attention to this this question of variance or this

52:56 factor of variance um but it is really important when we're thinking about cluster rcts and it does come back to this idea of statistical power so it's important um to think about increasing our sample size and kind of increasing the number of clusters that we have for a cluster rct um so why does all of this matter and i i should say i was definitely relearning this when i was preparing the slides in terms of what we call an icc which is a intra cluster correlation um so similar to power i think don't be too put off if this feels a bit difficult to understand it is difficult to understand equally if there's people who have good resources or anyone watching who's more

53:42 familiar with this do feel free to drop anything in the chat because yeah it is a little complex um but basically what we are trying to understand when thinking about the icc is the measure of relatedness of a cluster so we're looking at the extent to which the variation within a cluster so within a school relates to the variation between a cluster so between the different schools and our unit of randomization mathematically on the right hand side all that we're saying here all this equation is spelling out is that to get the icc so the intracluster correlation we look at the between cluster variability and we divide that by the sum here at the bottom

54:27 of the within cluster variability and the between cluster variability together so this plugs straight into our power calculation for a cluster rct and it is an essential part of thinking through whether or not we have enough schools to be able to conduct a rigorous rct or if not schools than whatever our our unit perhaps families with presence um so let's say we have two scenarios we have ten schools with eight pupils on the left hand side and we have four schools with 16 pupils on the right hand side and what we want to know is which one of these scenarios is likely to give us more statistical power or a more rigorous rct so it's the school on the um left-hand side

55:13 it's the ten schools with eight pupils um and this is because of a kind of first rule of thumb for variation in cluster rcts which is the increasing the number of clusters so the number of schools gives us more power than increasing the group size so four schools with more pupils is not actually better than more schools with less pupils here we have another scenario so in the group with lots of dots and lots of variants on the right hand side and a group with less variance or less different colored dots on the left hand side and again we want to know which which scenario is likely to give us more power um

55:59 so the answer here is that it's school two on the right hand side and this brings us to our second rule of thumb which is that the the more people look like each other within a group the less power we're going to have so this essentially brings us to a kind of i guess a conclusion for cluster rcts which is that in general as a principle we want lots of small groups that look less like each other so more clusters with people who are less similar and we're likely to have more power when conducting our cluster rct so to sum up um cluster rcts we randomize at the group level so the the kind of unit level the school the family etc rather than the individual level

56:45 we still measure at the individual level so we're still looking at the increase in attainment or confidence or sense of belonging per pupil rather than at the school level and the biggest thing to get our head around is the intracluster correlation um and really what this means is that we're just trying to work out a the sum and ratio of the variance between the clusters and within the clusters that is really the main three things we need to know for cluster rcts obviously it it kind of simplifies it in terms of actually how we would conduct one but they're the three principles that we need to be aware of um when conducting a cluster rct so just to finish off quickly some examples of where we use us clustered

57:31 rcts obviously in schools where we can't randomize individuals so we do it in classrooms or whole schools often used in hospitals or medical trials so we'll we'll look at um yeah different hospitals or different wards within hospitals sometimes and then credit unions or banks are also used um to reduce spillover effects between people that use the same bank on the same system i think that is the end of you listening to my voice a bit of a yeah a rapid tour i'm now going to hand over to sarah who's going to ground you in some hopefully easier to understand practical examples of how we have used rcts with summer schools and then we'll

58:18 come to you for a kind of q a at the end so over to you sarah ah sorry sorry to try i think i'm just going to suggest that just a 10 10 minute 5-10 minute comfort break if people would like to get a uh go to the loo get cup of tea whatever um and just say thank you very much rain for that um interesting informative and extremely um extremely good um introduction to our cts just to say that we have had some we have had some questions in the q a in the chat i think some of those have been picked up already and i think they're going to be further picked up in sarah's session when we're looking at the practical side of um implementing an rct in the real world but we will we'll come back to them um at the end of the session as well so it's 1103. now we've just come back

59:06 reconvene at 11 10 um we'll see you there

1:06:09 hi uh hello everyone um welcome back um so i will now pass you over to sarah we'll be talking about rcts and summer schools thanks very much helen and thank you rain that was a really good intro and sets us up nicely for this practical example so i know rain has used sort of lots of examples throughout her talk but this is um a real life project that we're currently doing um on summer schools um so we think it will be helpful just to show you how how this actually works in the real world and and how to run our cts in the real world uh so rain is controlling my slides so hopefully it'll be very smooth but you might hear me occasionally say next slide please or

1:06:56 something like that thought um so let's say to start with uh we've got a year 12 summer school and that that many of you might run yourselves um in your various roles at universities or providers um so we've got our summer school that's particularly for year 12 students that's for their first year in post-16 education and it has a limited number of places in it so this the an advert might go out to colleges or sixth form schools in the area inviting students to apply to the summer school and and from those groups you get some students that apply our group on the left and some students that choose not to apply

1:07:42 and then we might look at our outcomes for these students so to see in the end how many of those students that applied and didn't apply actually enrolled in higher education in their future so we see on the left that those that applied and attended the summer school we had 80 percent that enrolled in he whereas if we look at our group that didn't apply only 50 enrolled in he so if we look here we might say brilliant our summer school is doing exactly what it's meant to be doing majority of students that are attending are enrolling in hg so we're doing our job the summer school is working all good but unfortunately if we if we did take that conclusion there would be some some

1:08:28 problems basically so there's there's going to be key differences between these groups um some might be observable so these might be to do with gender so for example we might find that in our students that attended the summer school majority of the students were female and we know from from other research that more females are more likely to enroll in he than males so perhaps that actually caused the differences in he enrollment between the two groups attainment is a really good example as well so we might have found that the students that attended the summer school had higher attainment at gcse a-level or b-tech so again that was the reason why they were more likely to enroll in he because they got the grades needed

1:09:14 so there's some sort of really obvious observable or demographic differences but there's also a bunch of unobservable differences so for example we might find that those that applied to the summer school had more family support they might have had parents themselves who'd gone into he they might have had siblings that have just been to he that really encouraged them to apply again they might be more individually motivated themselves the fact that they applied to a university summer school implies that they're sort of already interested in he so they might have had higher motivation than our group that didn't attend so there's some really clear differences that we might have missed when we judged on whether our summer school was effective so yep just to sort of illustrate that

1:10:01 point as we can see those who did apply to the summer school they did have higher attainment than the other group they did have more family support and they did have more motivation to attend to enter he in the first place so this is why we have to be really careful with sort of making those assumptions based on raw strat royal stats we really need to know what differences might exist between our two groups before we can say that and this links to something that that rain sort of covered in her introduction which is that correlation doesn't mean causation so at the moment we've got a lot of this sort of type 2 evidence which is building our understanding on summer schools it's showing that attending summer school is associated or

1:10:46 correlated with um more positive attitudes to he and in some cases higher enrollment rates but what we don't have is this causality we don't know for sure that it's the summer schools that actually had this impact rather than sort of key differences that already exist between two groups which might make them more likely to attend h.e in the first place so we need this sort of type 3 evidence to uncover the fuller picture and that's where our summer schools project comes in so we want to try and assess the impact of university summer schools this current evidence that we've got is very much lying in the type 2 section so we've got this limited evidence on causality that's what we really want to

1:11:32 do we've also got limited evidence that actually links summer schools to our behavioural outcomes so a lot of the evidence out there at the moment is linking summer schools to attitudes or confidence levels so we're really missing this behavioural outcome on the on the impact on he enrollment rates so that's what we're trying to uncover with our own summer schools evaluation so the project i'm delighted to see that we've got some of our project partners here and i noticed in our attendees so i hope i do this description justice as you're all watching um but thanks for coming along so we've got multiple university partners in this project some of which run summer schools aimed at pre-16 students so these will be year 9 or year 10

1:12:19 students and some that run summer schools aimed at post-16 students so these will be year 12 like the the example i just used so sort of first year of post-16 education so we actually started this subject this project some time ago um and it started looking at summer schools that were taking place in the summer of 2021 so as we know um we were in the midst of a pandemic then and unfortunately this meant that summer schools had to be delivered online which we know isn't normal practice so what we've actually done is an extension of the project so we're also going to be looking at summer schools which take place this year in a face-to-face residential setting so that

1:13:05 we can measure normal practice and and the impact on he enrollment that because we have um our online project going on as well it's quite nice that we can do a sort of comparison of online delivery first versus face-to-face delivery and with our with our summer school partners uh the the usual way that i'm sure most people are familiar with and that they'll um allocate their places to a summer school is through a ranking system so on the whole um summer schools will be oversubscribed so you'd have a set number of places perhaps 150 places on your summer school but you might get 250 students that apply so not everyone that can apply

1:13:50 that applies sorry can get the place so what universities or providers tend to do normally is and they'd apply a ranking system so they might put students that maybe meet two or three widening participation criteria they ensure that they have have places and and they rank students

…