Significance shifts when cases are excluded

Question

I have a county level dataset with a dichotomous variable which represents whether or not any charter schools exist in a county. I am performing a nested logistic regression. That variable is the response and the explanatory variables are social factors such as education level, language spoken at home, household type, etc. When I include all counties in the U.S. (Including ten states which have no charter authorization laws and thus every county in those states is represented as a zero in the response variable) my results show significant associations between the response and the explanatory variables. When I exclude those ten states the significance disappears.

Any ideas as to why this might be happening?

Answer 1

Sounds like you came across Simpson's Paradox. Roughly speaking you have a result in one group of data but not when you look at different groups or subsets. This often happens when the variables depend on hidden dimensions in the data.

The wikipedia page explains it a bit better and in more detail.