AI Creates New Trading Indicators with Genetic Programming (Python) [Full Code]

0:00 Hi and welcome back. Today's strategy is something I'm trying for the very first time. One of our viewers actually encouraged me to test this branch of artificial intelligence named genetic programming and apparently we can use it also for trading and the viewer even shared some academic papers on the subject and most of them report pretty promising returns. So first of all, thank you for sharing this very informative idea. In this video, I'll explain what genetic programming is, how it works, and how we can actually code it in Python. I'll show you which libraries to use, how to train the model, and of course, I'll share the full back testing code and the trained model with you. Quick heads up though, training this model took me several days on a Core i9 processor. So yes, it's computationally heavy and we will see in a moment why that is. Just keep in mind

0:45 that it's only the training phase. The inference later on using the model is much faster. So what is genetic programming? Think of it as a branch of AI that borrows ideas from evolution and genetics. So instead of starting with a fixed indicator, we let the computer randomly generate thousands of formulas. Each formula is a potential trading indicator. These formulas and indicators can evolve through generations. And each indicator is back tested on historical data to check its performance. So the first step is to generate a pool of random indicators that we will call the initial population. This can be 5,000, 10,000 indicators or even more. In this example, I took 15,000 as a start. each formula or indicator is of different complexity. Then we will back test each

1:30 of these indicators within the same trading strategy on historical data. And this is why it's computationally expensive because in this example with these numbers that I'm presenting here, we are applying for one iteration 15,000 back tests. So imagine we are running this for around 20 generations. That's 20 * 15,000 back tests in total. In the second step, we identify the most promising indicators through a selection process based on the back test's performance. We will keep the top 10% or any percentage actually. This is just a parameter that can be changed in the simulation. And we simply discard the remaining 90% of these formulas or indicators. And all of this selection is of course based on either the sharp ratio or the returns percentage or the

2:17 risk adjusted returns. Then in the third step, we breed the uh selected indicators together. In other words, we randomly mix parts of one formula with parts of another just like DNA crossover and we add small mutations or random changes. To recap the process, we have our first generation of indicators. We back test these and we select the top 10%. Then we start shuffling parts of these selected indicators to generate a new set of 15,000 indicators. That is the second generation of our mutated formulas that are the descendants of the top 10% selection. So these are supposed to be better set of indicators or the second generation is supposed to be better than the first generation in terms of returns or sharp ratio in the back testing. Then we repeat the back

3:03 test with the new indicators. We select 10% again. We continue this process for few generations and after 10 20 generations for example indicators evolve through these steps and so we expect them to be more performant then we end up choosing the champion indicator from the last generation. It's kind of looking for this one best formula that could survive the entire process. And once we have this champion we test it on completely new unseen data to see if it holds up. We'll go through the details later on in the code. Now here's the twist. These aren't your usual technical indicators. They are not RSI nor MACD nor binger bands. These are brand new machine generated indicators. They are randomly generated every time we run the code. Some papers use this

3:49 genetic approach. Taking it a step further. Instead of using one asset, they feed in prices from multiple assets. For example, they might give the model US dollar, Japanese yen, a British pound, US dollar data, and so on. in order to evolve an indicator to predict the euro US dollar prices based on prices of different assets. And that's only one approach. Another idea is to use classic technical indicators as inputs and let genetic programming shuffle and mutate them into a brand new formula that is supposed to be more performant. As you can see, it can get pretty wild very quickly. I'm not going to cover every single variation mentioned in the papers right now. Instead, we will start with one version, get some results, and then maybe revisit this in future videos based on your

4:35 feedback. I personally find this fascinating. It's fun. It's experimental, and it opens the door to totally new trading ideas. Again, the Python code and the trained model as well as the environment requirements are shared on my GitHub repo. The link is available in the description of this video. So feel free to try it out. Play the parameters, brainstorm new ideas, and share your experiments in the comments. All right, let's jump in and I'll show you how we coded this whole thing in Python. Then we can back test the model and check the results. So this is the uh Python script. It loads the 5m minute time frame candlesticks. These are CSV files of four pairs Euro US dollar. Uh GBP US dollar, Australian dollar and US dollar and Japanese yen as

5:22 well. So I have one, two, three and four files. These are my data files. It standardizes the open, high, low, close um column names. Merges all pairs into one data frame. Then splits the data set into train uh validation and test. These are the dates where the data is being split. We will evolve on train then choose the winner on validation the winner indicator and then report real performance on the test or out of sample data. And here we can see the uh population size and the number of generations. So for this example I took 1,00 uh population 1,000 indicators at first and then 15 generations for uh evolution. The crossing uh percentage is

6:08 90% and the mutation percentage is 15%. Then we have the maximum depth of each indicator. So these are basically the number of mathematical operations that we can apply on the input parameters and the maximum length of each indicator. We have the initial cash cache for the back testing, the commission percentage, the no trade band. I'm going to explain this in a while and the uh position uh grid. So basically each indicator is a math expression tree built from primitives like add, subtract and uh s cosine and mathematical operations that you can see here in this part. So we're going from here down to um line 136 almost. Inputs

6:54 are the open, high, low, close values from all pairs. So the um genetics program can make signals across markets. indicator output is a time series of desired percentage exposure between minus 100 and plus 100%. So um it's clipped to minus 100 and plus 100. This is the desired uh percentage exposure. As we can see here in line 167, it's then converted to a value when we divide it by 100 between minus1 and one. This number represents the desired position size. Plus 100 means fully long or + one means fully long. Minus 100 means fully

7:39 short. Zero means no position. So we convert this into a portfolio weights between minus1 and one. Now to avoid overtrading we apply something called a deadbend. The deadband means if the new position signal meaning if this number between minus 100 plus 100 or between minus1 and one is within let's say 10% around the zero plus or minus 10% the system does nothing. We're not allowing any trades. It has to cross this 10% threshold in a positive or negative direction to go long or short. So trades are only triggered when the signal changes by more than the deadband threshold of 10%. And again this strategy was inspired by the publication

8:24 or the work published by these three authors that was sent to me by one of the viewers on this channel and we are trying to experiment using their method. Now if you have noticed I have two files GP strategy progress vectorbt and GP strategy progress. This one uses back testing.py Pi which is more discreet in the back testing process and this one uses vector BT which is vectorzed and it runs much faster which is needed actually to uh back test and do this kind of heavily uh heavy computation um experiment but my preferred method even if it's slower is this one because it's discrete so using back testing.py because for each trade we know what we are doing and we're not really bulk executing in a vectorzed way. And here you can see how vectorbt is used in this

9:11 file or back testing.py in the previous file. It's the same. So we are using um target percent size type. Uh we have a commission percentage. We have an initial cache and the frequency is 5 minutes. We're using the 5 minutes time frame. Now for each back test using an individual indicator generated by this genetic program algorithm the fitness of the indicator is measured which is the exponential negative the returns. It's seen here. So it's exponential minus total returns meaning for higher returns we get smaller fitness which is better because the genetics algorithm is minimizing the fitness and this again is also based on the paper where we can find the fitness here that's u

9:58 expression two fitness is equal to exponential minus the return then we can move on to the evolution run uh which is defined in this function. So run evolution. We create a population of 1,000 indicators or strategies and evolve over 15 generations. Each generation does evaluate all strategies, all indicators. Keep the best ones in a hall of fame somehow. Select parents somehow a tournament selection. Apply crossover with 90% chance. Apply mutation with 15% chance. I know they don't add up, but it normalizes at some point. and it re-evaluate any modified strategies. So we rerun the back test on any mutated or crossed uh strategies or

10:44 indicators and while running the um evolution we print some progress parameters like the minimum fitness the average fitness and the evaluation uh the evaluation count. So at the end we take the hall of fame top strategies and indicators score them on the validation period pick the best one then we run one final test using the out of sample back test data on the test set meaning which allows us to print uh key stats like return sharp ratio trades and win rate and the final equity and finally we are saving whatever model we just trained into um a file so best_individual D in this case which we can load later on without rerunning the whole evolution process just for inference and using for

11:31 back tests. Now just for the sake of showing you an example how it runs. I'm going to run a small evolution. So this is 1,00 uh population 15. Let's try it out. See how long it will take. Of course you can run it for more than that. You can start with 75,000. This is what the paper is advising. This is what they did. 75,000 uh population size as a start and for 15 generations. So now we are shrinking this because then it would take uh too much time. So Python uh GP strategy this is the vector BT version and we can run this. So you can see it's loading the data now running the evolution and it's going to print the steps in which the program is

12:18 going through. I've also added a Jupyter notebook file where we repeat all of these steps in simple cells making it as concise and as clear as possible including and importing functions whenever needed from the other files. So as you can see we have the u train validation and test. We have all the parameters as well and some helper functions helper utilities here. uh and the back testing part repeats the same uh work we uh we have explained in the other scripts only using a Jupyter notebook file if you want to see the uh different steps when you are running this program. Uh anyway, robustness should be tested definitely this is not a magic u method to have the magic

13:03 indicators. It's simply an experiment. We are trying something new here. Now if you have a model that is already saved, we can also run the trading system load and infer. This uh file just loads a file loads the uh model trading model indicator if you've already uh done your genetics process and generations and so on and it will be using it just for a back test on the testing uh uh period or the testing data. So let me run one test for you just to show you how it's working. So it's loading the data. It gives you the uh fitness of uh the indicator it's going to use which is the best model basically. So it's not really the best model. It's a fitness of 10 to six. So this is a very bad indicator, very bad uh model. Remember the fitness

13:50 should be something very low, but we're going to run it anyway just for the sake of the experiment. We're starting with a million dollar in cache as a start. It's loading from best individual. Dill file, which is right here. I have two other models individual one individual two. So these are different runs with different generations, different settings which is uh very time consuming when you are training but you can save them for uh later use. So now I'm loading this one and it's going to come up with the equity the back test. It's going to show the equity curve and so the back test results here are 46% in return sharp ratio 1.1 and maximum drawdown of minus 12%. So this is kind

14:35 of a lucky indicator or strategy because the fitness is supposed to be very high. So it's not supposed to be a good indicator. Somewhere in the back test it was faulty but actually it's giving a positive uh result on the testing uh data. So it's saved here. I could use it later on maybe on another back test see if it works. But for now it's just a simple example. Now the way you can evolve this is change the strategy. Remember that we are using uh some kind of a percentage indicator. If it's plus 10% and above, we're going long, minus 10%, we're going short and so on. Things can be changed. We can use a take-profit stop-loss like go back to a normal strategy and just use genetics programming to train an indicator. But

15:21 that might be for a next video. For this video, I think this will be it. I hope it inspired your curiosity. I'll be sharing the code files on a GitHub repository so you can download them, experiment with different parameters and see how it works. Until our next one, trade safe and see you next time.