You would have to run a set of artificial tests, trying to detect relevant features using different methods while knowing in advance which subsets of input variables affect the output variable.
Good trick would be to keep a set of random input variables with different distributions and make sure your feature selection algos indeed tag them as not relevant.
Another trick would be to make sure that after permuting rows the variables tagged as relevant stop being classified as relevant.
Above said applies to both filter and wrapper approaches.
Also be sure to handle cases when if taken separately (one by one) variables do not show any influence on the target, but when taken jointly reveal a strong dependence. Example would be a well-known XOR problem (check out the Python code):
import numpy as np
import matplotlib.pyplot as plt
from sklearn.feature_selection import f_regression, mutual_info_regression,mutual_info_classif
[ 0. 0. 0.00429746]
So, presumably powerful (but univariate) filtering method (computing of mutual information between out- and input variables) was not able to detect any relationships in the dataset. Whereas we know for sure it's a 100% dependency and we can predict Y with 100% accuracy knowing X.
Good idea would be to create a kind of benchmark for features selection methods, does anyone want to participate?