I used Random Forest and hypertuned the parameters for a binary classification problem on a dataset (dataset A). I got a F1 score of 0.78. I then used a second dataset (dataset B). It was very similar to dataset A (same variables and the distribution of classes in the target variable). I again built and trained a different Random Forest algorithm for dataset B. I expected the f1 score to be around 0.78, but the f1 score for dataset B was 0.50.
Why could there be such a large difference between the f1 scores of the 2 datasets?
Both datasets (A & B) are very similar to each other and I trained separate models on both of them.