Test Multiple Variables at Once to Optimize Anything

0:00 hi everyone today we will be learning about a simple experimental method that will dramatically improve your results and understanding of any experiment you happen to try maybe you want your garden plants to grow healthier or to improve a recipe for bread you could even use this technique to optimize the format of YouTube videos to make them more entertaining when I first tried this method I thought my initial experiment was going to be a total failure thinking I must have done something wrong because the information I got out of it was so surprising that I didn't trust it it turns out that this new method had revealed something that I had never even considered in 10 years of tinkering on the same project when I finally did trust the results it led to the fastest

0:47 improvements I've ever made there's one thing that I learned very early on when I first became interested in various sciency inventive projects when working on a complic ated experiment only change one variable at a time that way when you see the results you know exactly the thing that changed and whether that one thing had a positive or negative effect if you change two things at once you would never know which change is responsible for your result in fact one of the two changes might be helpful to your experiment while the other is so destructive that it makes the entire test look like a failure you don't learn anything by doing this and you actually concealed information that might have helped you so I have used the method of

1:33 only changing one variable at a time for about the last 15 years and you can see on my channel that it has served me well in creating lots of successful projects recently however I found myself working on a project Head and Shoulders more complicated than most of the things I've attempted in the past that is developing a new process for making calcium carbonate microspheres by changing only one variable at a time I realized through great effort and longsuffering that optimizing a fairly simple chemical reaction can quickly add up to many hundred hundreds of individual experiments I was out to dinner with some friends and got to complaining about the difficulty I was having to which they asked are you only changing one variable at a time uh yeah I said

2:20 how else would I know which of my changes worked it turns out that these friends of mine used to have jobs working on the optimization of factory processes and were familiar with experimental methods that I had never heard discussed in the realm of hobby projects or even among other amateur chemists like myself that needs to change this experimental process is not complicated and the results are awesome so to explain how this works I thought we might actually try a real experimental task we'll start with something simple let's find out if the common advice for making boiled eggs peel easier really works now we could find many different tips for this if we looked online but I decided to try just

3:05 three different things two are related to temperature whether it matters how cold the egg is when it goes into the boiling water and also if it matters whether we drop the cooked egg in ice water while it's still hot the last variable is an additive vinegar which supposedly helps loosen the shell if a small amount is added to the water so we have three different variables and a variety of ways that we could test them we could run four different tests starting with a control where the eggs are room temperature going into the water with no vinegar added and then are also allowed to cool at room temperature as well this would give us a baseline to compare three more tests where we try each of our changes if any of these makes the eggs easier to peel than the control test then we know what does and

3:52 does not work but here's where it gets complicated what if adding vinegar to the boiling water does not make the eggs easier to peel by itself but perhaps if the egg is first treated with vinegar and then also dropped into cold water the combination of these two variables has an effect where individually they fail we would not discover that with our series of four tests so we need to add four more to account for all possible combinations it's not entirely unreasonable to do eight different tests but that number quickly spirals out of control if you add a fourth variable or a third setting to each of the existing variables this is called a full factorial analysis because it explores all possible combinations of an experiment giving you the most data

4:39 possible in exchange for an enormous amount of work in certain types of experiments this may be worthwhile but for investigating the peeling characteristics of boiled eggs no thank you fortunately we can get most of the relevant data from a much smaller set of experiments one that looks like this you can see that we still have a control group which the other tests can be compared to but after the first run every other test has two different changes occurring at once there are similar tables that can be used for experiments with many more variables but we'll get to that later with this arrangement we only need to do four tests so let's get started I'll do each run with four eggs each so that we have a decent sample size to know what works

5:24 experiment one will start with eggs at room temp and experiment two will start in ice water these are then placed into the boiling water for 11 minutes and then the eggs marked with a one are left to cool at room temp while the others go back into the ice for the second pair of tests I'll add a/4 cup of vinegar to the water and according to our experiment table the third test is meant to start at room temp while the fourth starts in the ice another 11 minutes in the boiling water and then number three goes into the ice while number four cools in open air and now we record our our results this is the most difficult part of running an experiment like this how exactly do you record the difficulty of peeling a boiled egg as an objective

6:11 number you might say that this egg was extremely easy to peel so we'll record it as a 10 and another was difficult to peel so maybe it's a five but that's just an arbitrary estimation it's hard to be objective about assigning accurate numbers to such an organic and messy proc process so it's very important to identify something about the results that can be measured with scientific Precision in this case I decided to assign three different levels of success in the ease of peeling an egg if the shell slips off cleanly and easily that scores a three if some pieces of the egg are left stuck to the Shell that scores a two and if large pieces tear off the egg more than a/ in square that scores a

6:56 one I might also give a partial score of 1.5 or 2.5 if pieces break off in one place but most of the egg peels cleanly in this way we can assign a mostly objective number value to an otherwise messy process which is essential for accurately interpreting the results now I encountered a problem here because every egg except one peeled perfectly which reveals that other factors besides those that we are testing have an effect on our results age of the eggs is one factor that is known to have effect so I had to repeat this test with fresher eggs that are normally much more difficult to peel now we should have some usable data so here is the score of all 16 eggs from all four of my tests

7:43 totaling these together we end up with a single value for the result of each run so how does this solve the original problem of testing two variables at once if we take any one of these tests and compare it to the Baseline we do see a difference between the two but as expected it's impossible to tell which change is responsible for the increased value however the other two results are more revealing let's look at the egg starting temperature first if we take the two tests where the eggs started at room temp and add their scores together we get a total of 19.5 now taking the other two tests where the eggs started in ice water and totaling their scores we also get a total of 9

8:29 19.5 when we look at the combined values of tests where the eggs started at room temp and those which started in ice there was exactly zero difference between the results so what does that tell us the temperature of our eggs as they go into the boiling water made no significant difference in how easily they peeled likewise we can add together the values of tests with and without cooling the eggs in ice water after cooking and see a difference of one and with an without vinegar in the water for a difference of four that is significant it seems that the vinegar in the water really does help the eggs peel easier as for dropping in ice water it might help but only barely and in fact it's very possible that this value of one is just

9:17 due to a single egg being slightly easier to peel than average to be more accurate we could use 10 or 20 eggs for each test but these results are good enough to say with confidence that vinegar makes a difference and the other two things are at most comparatively insignificant already we have learned all of the information that we would have by testing each variable individually and in fact we've done so to a greater degree of confidence because each change was tested twice but that's not all if you look closely every variable was tested against every other variable so we know that the combination of a cold egg going into the boiling water and then being plunged back into ice water has no compounding effect

10:04 because there's little difference when compared to the Baseline likewise vinegar does not seem to have a compounding effect with either of the other variables because if that were the case we would expect one of these values to be significantly different from the other in fact we learned almost everything from just these four tests that one could hope to learn from a full array of all possible configurations so how does this particular set of experiments managed to give us so much information without some of it getting buried by the double variable changes the table we used is called an orthogonal array sometimes called by the name of its inventor a Uchi array the rules for assembling this type of experimental Arrangement are fairly simple every level of each

10:50 variable should be tested an equal number of times so if we test the addition of vinegar twice we also need to do two tests at a Baseline level of vinegar which in this case was none a second rule is that every level of each variable should be tested against every level of the others also an equal number of times both of these rules combined means that if we look at any variable on our table say the vinegar even though multiple things are being tested at once look for both tests that the vinegar is high the changes in the other columns are opposite to one another meaning if you add the results of these two tests together together the effect of the other variables cancels out the same is true in these two rows where the level of vinegar is constant the other changes

11:38 cancel each other when the results are added together it's as if we tested vinegar all by itself this works for the other variables also where the middle column has a low value the two rows added together cause the other changes to cancel out how awesome is that that we can do four tests and get results from each variable individually and results for what happens when you add the variables together maybe you're not impressed yet because this was a very small test here is what an array looks like with four different variables each with three different settings High neutral and low a complete series of tests for this many factors would require 81 individual experiments this table gets the job done

12:24 in nine before looking at more advanced applications of an orthogonal array I thought we might take a brief pause here to talk about improving our understanding of logic in general made super easy with the help of my sponsor brilliant.org my favorite courses on brilliant are about logic and scientific thinking because the skills you learn through those courses are immediately useful in almost any situation being better equipped to detect what is true and false what is sound reasoning and what rests on fallacy the those are skills that can improve not only our understanding of scientific topics but also refine our opinions in everyday life the courses un brilliant make logic and scientific reasoning very easy to

13:10 learn one of my favorite philosophical psychological topics is the idea of being wrong and specifically how it feels to be wrong about something that is being wrong feels exactly the same as being right we can be very confident in a false idea right until the moment that a logical error is exposed and we realize our confidence was founded on a misconception how can I know that the things that I am confident about are true when others are just as confident that I am wrong you'll see in the second half of this video that I was confidently wrong about an aspect of one of my experiments the solution was to take a step back accept the possibility that my preconceptions might be wrong and see what conclusions logically

13:56 followed from new information one of my goals in life is to become increasingly aware of which of my opinions I can be confident about and which ones require more humility and greater Readiness to accept correction logic and scientific thinking are some of the key tools that I find useful in distinguishing between the two which is why I appreciate those courses on brilliant their other courses are also very useful on thousands of targeted topics like math data science and computer science which they teach very skillfully with interactive problems and challenges you'll have fun with Brilliance courses and find that they can help you learn no matter what skill level you're starting from you can get started with brilliant for free for 30 days using my link brilliant.org Nighthawk and the first 200 to use that

14:44 link will also get 20% off brilliant's annual premium subscription my first attempt at using an orthogonal array used this four variable table I was trying to refine a new formula for a type of Japanese sparkler called aseno Hanabi these are a notoriously difficult firework to perfect and to reach my current level of confidence in making them actually required visiting a master Craftsman in Japan every time one of the ingredients in these sparklers is adjusted every other component needs to be adjusted in response I found this effort to be well worth my time over the years because they're just unbelievably beautiful in person and their function is

15:29 scientifically very unique and fascinating one of the key ingredients is called Lamp black which is basically a fancy version of soot from a flame in Japan they made this soot from decades old pine tree stumps meticulously prepared and burned over the course of several days in a room enclosed by walls of tissue paper when finished the lamp black is scraped off the walls and sold as high-end calligraphy ink and for use in senko hanabe sparklers that is a process which is not at all accessible to the average person and so I've sought ways to make lamp black through easier processes that will still work in these sparklers one way to do that is by burning vegetable oil

16:16 under a metal tray the soot collects on the metal surface and afterward the lamp black can be brushed off the tray which I find enormously satisfying now changing the fuel source in the creation of soot dramatically changes its properties and so this oild derived lamp black no longer works in the same formulas I've used before hence the need for a new experiment another fellow here on YouTube who goes by the username zry is the one who made me aware of this method for producing lamp black and he was kind enough to also share a semi-functioning recipe that works adequately well in these sparklers that's what we'll start with as our Baseline and see if it can be further improved through an orthogonal array of

17:07 tests there are four ingredients in these fireworks one that provides oxygen to the mixture and three which are fuels since I don't know if we need more or less of each of the ingredients we need to test three settings for each one that lowers the ingredient one that keeps it the same and one that raises it the amount that each of these ingredients is adjusted by for each setting is mostly arbitrary I chose small enough changes that I suspected they would still allow the sparklers to function for all of the tests but still be a big enough change to notice if that change has an effect so I mixed up nine different sparkler compositions based on the measurements determined by the four

17:52 variable orthogonal array this table was just pulled out of a list of tables set up for experiments of various sizes you don't need to build these tables yourself just Google to Gucci arrays and you will likely find one that will work for as many variables as you'd like to test with my nine different compositions I then rolled two sparklers with each which is a pretty small sample size ideally I should have made more and added the results between four or five sparklers like we did for the egg experiment but two for each test did end up giving me plenty of usable information the next step is once again the most difficult figuring out a way to measure the results so that we can assign them an objective score I decided the best

18:39 way to do this was to place the sparklers in a fixed position over a grid so that with a camera we could later measure the various effects these sparklers have three distinct sparking stages the first shoots out large bursting Sparks the second produces a ball of small crackling Sparks and finally there's a stage of miniature non- branching Sparks that shoot out of an ember that slowly climbs the paper stem to give these sparklers a score I'll record the distance traveled of the top three bursting Sparks that shoot from the sparkler then I'll measure the average diameter of the crackling spark stage and finally I'll give the last stage a score of one two or three based on three different ways in which that stage can progress one by one these

19:25 sparklers were ignited over the test grid and to be honest I really started to think that this was not going to work the sparklers seemed to behave so randomly and mixtures which I thought would perform really well ended up looking terrible in my head I started rewriting the script for this video explaining the shortfalls of experimental methods and maybe where I went wrong even once I got the footage back onto my computer and started adding up the results I was slipping into some hopelessness of learning anything worthwhile from all this work this is the finished table with the results added up first of all notice how much harder it is to understand these results just by looking at them compared to the egg test it didn't take much brain power

20:10 to notice that vinegar made a big difference when there were only four numbers to compare for this we actually need to process the data to do that remember that we add together the scores of each test with the same variable change so we'll start with the oxidizer potassium nitrate and add together the scores when it's at the low setting this gives us a total of 125 the tests with the nitrate at the high setting add up to 123.5 filling out the results for the rest of the variables looks like this these results so go against my expectations that I still thought they must be wrong my experience with these sparklers so far has taught me incorrectly it turns out that the more

20:57 lamp black you can shove into the recipe the better a result you will get the ingredient that I typically lower to make room for more lamp black without disturbing the ratio of oxidizer to fuel is the charcoal by increasing lamp black it would slow the sparklers burn and so I would balance it by reducing the charcoal and achieve a better result and therefore I believed that increased lamp black generally made for better Sparks in fact what these results are telling me is that the extra lamp black often has a negative effect which is hidden by the overwhelming benefits of reducing the charcoal I never discovered this with a one variable at a time approach because while I did adjust the ingredients only one at a time I would

21:43 always add lamp black first test the burn and then lower the charcoal to compensate if I had thought to lower the charcoal first I might have noticed that that was the real cause of my improved results where adding the lamp black was actually making them worse I know that I'm rambling on about a very Niche project but hopefully this serves to illustrate how powerful this experimental method is my first time using it revealed information that I had completely backwards about a project I've spent more than a decade working on it's amazing when I was finally convinced that these numbers might be accurate I made a few more mixtures stepping down the amount of charcoal in each batch the results of these tests were literally off the scale that I had

22:29 set up one thing that this orthogonal array does not tell us is how far we can push the reduction in charcoal before Things Fall Apart I found that a significant reduction did offer improvements but there's still a balance of burn rate that needs to be maintained also I found that when I reduced the charcoal too much I start losing the last stage of Sparks this is where going back to the test data can help in other ways since I recorded the score for different sparking stages individually I can look at the scores for the last stage which is now giving me trouble what I notice is that in three out of four tests where the last sparking stage had a perfect score the sulfur level was high when the

23:14 sulfur was low the results were poor so if I want to improve this final sparking stage the key according to the numbers is to increase the sulfur this multivariable exper experimental method very rapidly gave me this information to drastically improve my sparklers really that's what this method is very good at pointing you in the correct direction for improvement now that I have decreased my charcoal and increased my sulfur the results are much better but from here I could do another panel of experiments and it would probably reveal more subtle changes that could be adjusted from this new and improved Baseline after dramatically lowering the charcoal maybe now it would be beneficial icial to add back some lamp black but probably less

24:01 than I would have before I have no idea what a new series of tests would show me you can run these panels over and over again with diminishing returns to come ever closer to perfection now here is the challenge I've just shown how to use a taguchi array for two experiments that were quite different from one another and I did this intentionally I want to demonstrate how you can take completely different tasks like boiling eggs and making sparklers and set up an experiment in a way that results in useful information figuring out how to do the same with whatever project you might be working on will require its own set of measurements in a way this video has been about experimental methods in general most of the challenge is

24:46 figuring out how to quantify organic lived experiences into numbers that can be more easily interpreted that's the real magic of living in a universe that miraculously makes mathematical sense well I know there will be many of you wondering how I could be so surprised by experimental methods that have been common knowledge forever in your industry I found that I know very little of all the information that's out there hopefully this will be helpful to a few of you thank you to all of you who support me on patreon I very much appreciate that you find Value in what I'm doing with this channel lots more to come thank you for watching I'll see you next time and