I need help determining the best way to go about creating the target variable for a machine learning model that is trading a financial instrument (stocks, foreign currencies, crpyto, etc).

Below is some sample data to help with answering the question.

Time: When the data was recorded  
Price: Is the current price of the instrument and is the price I would be buying at.  
price_good: Price level that I need to sell at to make a profit  
price_bad: Price level that I need to sell at to minimize losses  
good_id: Returns the ID of the first row where price >= price_good  
bad_id: Returns the ID of the first row where price <= price_bad   
target: If good_id < bad_id, target = 1. Else target = 0  

My ideal scenario is when price_good occurs before price_bad. The issue I'm running into is how to correctly set the target field. I have 2 ways that I can solve this -

Option #1 - This option uses all of the data and has no restrictions on creating the target field but has a slight mismatch on how it would work in production.

id         time       price    price_good   price_bad    good_id   bad_id   target
1          01-01-19   100      110          90           4         nan      1
2          01-02-19   105      115          95           4         nan      1
3          01-03-19   109      120          99           4         nan      1 
4          01-04-19   121      131          111          nan       5        0
5          01-05-19   110      120          97           nan       nan      nan    

Option #2 - This option doesn't use all of the data and doesn't allow a target to be set until the initial row reaches it's target which is how it would work in production.

id         time       price    price_good   price_bad    good_id   bad_id   target
1          01-01-19   100      110          90           4         nan      1
2          01-02-19   105      115          95           nan       nan      nan
3          01-03-19   109      120          99           nan       nan      nan 
4          01-04-19   121      131          111          nan       5        0
5          01-05-19   110      120          97           nan       nan      nan   

"In production" means how it would work after the model is built. For example, If my model predicted that id:1 would hit target_good, I wouldn't be able to make any other trades until that trade reaches price_good or price_bad which aligns with option #2. However, I'm losing the information about all of the other trades that I could of made from option #1 if I had more capital to invest.

If I go with option #1, the model will sometimes "overfit" because it has data that is closer together in time and uses mostly the same information.

If I go with option #2, it is a more accurate representation of how it would work in production but I lose a ton of data points. For example, what if I started at id:2 instead of id:1. I would end up with different data points that the model is built on. I could test starting at multiple places to find the most optimal starting place but that would take a ton of iterations and resources to test.

Any tips on how to model this correctly? Thanks!

  • $\begingroup$ How would option #2 work in production? It still seems like you are cheating by looking into the future (something that will not be possible in production). And by cheating I mean that for the id=1, you get the good_id =4 by looking into the future. $\endgroup$ – Stergios Sep 3 '19 at 16:09
  • $\begingroup$ That would be what I'm predicting in production. The issue is more around training. With option #2, I would be training the model on data points that wouldn't always be available in production if I had an open trade but I still capture that data for training which could be helpful. This is only really a limitation due to capital but if I go with option #1 then I have significantly less data. $\endgroup$ – bbennett36 Sep 3 '19 at 18:05
  • $\begingroup$ I still cannot totally understand it. So, in production you are always going to have time, price, price_good and price_bad. And you are trying to predict the target. Is that right? $\endgroup$ – Stergios Sep 4 '19 at 7:00
  • $\begingroup$ @Stergios sorry for the delayed response. That is correct. My concern is mainly with the training though. For example, I had 5 rows of data that are at 1 minute intervals. In reality, if my model predicted a 1 for the first row, I couldn't make any more trades until that trade hits "price_good" or "price_bad". So for training, I'm trying to determine if I input the training data like that or if I keep all the rows so I'm not missing all of the possible predictions I could of made if I didn't have the capital requirement. When I do 1 trade, I'm using 100% of my capital so thats my restriction $\endgroup$ – bbennett36 Sep 12 '19 at 19:46
  • $\begingroup$ OK I get it now. I would go with option #1. Let me explain. Practically, your target would be 1 (or 0) if the predicted probability is above a certain threshold (say 80%). Therefore, there will be cases where the model is not confident enough. Using option #1, you allow your model to learn from all the data. It may be the case that the model given you a confident (i.e. >80%) prediction very late, but still it's better than never, right? In any way, you should use your trained model and simulate what would happen if you followed your strategy (you can optimize the threshold as well this way). $\endgroup$ – Stergios Sep 13 '19 at 7:43

You've got an interesting, if over analysed question. I would just say your question itself limits your potential profit. You want to give your computer more choice in this matter. the whole price_good, price_bad approach is price_bad. The computer should decide EVERYTHING in this game and you will want to build in a consideration of previous stock history, share earnings, maybe even public sentiment from twitter, etc.

This really seems like the optimal scenario to implement some variant of q-learning. You'll need a lot of data and the stock market is notorious for being difficult for the layman to derive any meaning from.

What is Q-learning?

Q learning is a deep learning method with allows a computer to find the ideal set of actions to maximize some final score. The system is composed of 3 parts: a deep learning network that predicts the target, a q table that contains the state at a given time and a possible action, and the change in the final score.

Read more here: https://towardsdatascience.com/simple-reinforcement-learning-q-learning-fcddc4b6fe56

Really, read that before continuing :)

Step 1:

You create the q-table (a table of many possible actions). In this case your state will be a vector of the number of shares you hold, the liquid funds you have for buying, and n number of consecutive prices for a stock (You'll also need the next price of the stock for a later step). You will then make a choice: how much of the stock do you want to sell? this number can be any real integer. a negative number indicates selling, a positive number indicates buying and a 0 indicates holding.

EDIT: I want to explain a bit more what's going on with the "n number of consecutive prices for a stock". This is where your "timeseries" bit comes into play. It is a vector that looks at a frame of prices at any given time, but it not necessarily the latest frame. It's just a way for you to train on what would have happened in the past if you were making that trade. For example if I have a vector of prices that looks like all_prices below, the frames would look like:

 all_prices = [12, 13, 14, 13, 14, 15, 16, 17, 18]
 frame1 = [12, 13, 14, 13]
 frame2 = [13, 14, 13, 14]
 frame3 = [14, 13, 14, 15]
 frame4 = [13, 14, 15, 16]

For each state you'll have to add rows to the table of every possible action (everything between sell all and buy with as much cash I have). Then for each row you'll calculate the reward for that action. Now, the reward is calculated based on the next price of the stock. It can be negative (bad) or positive (good).

So here's what that table would look like for a single stock:

 *n_shares* | *cash* | *price_at_t=1* | *price_at_t=2* | .... | *last_reward* | *action*
 50         | 1000   | 13.41          | 12.45          | .... | 0             | -10
 50         | 1000   | 13.41          | 12.45          | .... | 0             | -5
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 0
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 5
 50         | 1000   | 13.41          | 12.45          | .... | 0             | 10

Step 2:

You'll train some deep learning network on each row (state) to predict the reward. yay!

Step 3:

When you actually want to make a decision on your data, you create a table of all possible states and actions and have your nueral network predict the reward (now it's unknown). You can select the action which maximizes long term reward.

Some caveats

Now the above really doesn't look at all that much data. You can add in soooo much more and potentially create a better model. I've never done this, but I don't have particularly high hopes for something like this. I'm definitely open to reconsider though. Perhaps if you look at this with each timepoint being the close price, and each day you get one change to sell, buy, or hold, it might be feasible. Otherwise, you'll need an API that allows you to trade (super cool, but I haven't heard of something like this for consumers).

Are there better solutions? Yes, this is just a simple reinforcement learning solution. We can do better, but this seemed like a simple starting point to me.

Am I liable if you loose all your moneeeys? Nope.

Is the above normal q-learning? Not really. I took some shortcuts and glossed over some aspects of it. I do hope that it wasn't too complex and that it helped you understand how to better approach the problem.

Will you become a billionaire after this? ....maybe. I hope so. If it starts working, tell me and I'll invest with you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.