My training dataset is around 5 MB and test dataset is of the same size. I have run some simulations over the whole dataset couple of times. However, in a latter solution, I ran queries on two columns (say A and B). Now, in the calculation, for each row in the test dataset, I have to get the result of the following query. This time, the program takes forever. I can optimize this on my own writing some extra code (sorting will help in my case and in fact I can skip pandas altogether if I want to), but I wanted to check if there is a better built-in solution should I ever require that (also, I don't want to if I don't have to). The query is like this:
def f(A, B, x, y): data = df.loc[df[A].isin([x]) & df.B.isin([y])] return len(data)
Here, df is the DataFrame (training dataset). For each row in test dataset, data find all rows where column A has value x and B has value y. As far as I know, isin is slightly faster, so I used it. However, I think this is why it does not run within a second like it did before because each time I run this query, it needs a lot of time. What I have to do next is something like this this.
for i in range(0,l): ret = f(A, B, test[A].values[i],test[B].values[i])
Clearly, in every loop, it will calculate the result of the query. Other than this, rest of the functions in my code are mathematical that do not need much calculation.
Let me know if the problem is not clear since I did not use direct codes (replaced it with dummy variables and other lines do not have anything that has high complexity).