would like to hear people's opinion on the problem. I am working on a project and would like to know if on right path.
Essentially, I have a segment of 100k people from a population i'm trying to predict. For the variables: its binary information on each person, essentially do you have x: do you have a tv (y or n), an Xbox (y or n), play station (y or n), a wife (y or n), college degree (y or n), kids (y or n), are you married (Y or N)? Is your income above X amount(Y or N)? Let's assume they answered everything truthfully. Some of the variables are related and potentially influence each other like the kids and married or the xbox and play station. Essentially, it's a bunch of binary variables indicating if they have something. The total number of binary variables is 100k. It's a coincidence that both are 100k. Interestingly enough, I can look at association rules. For example see that people within this group are 30% likely to have a college degree or 20% to have kids.
Everybody in this Group, is group A. The idea is to be given new data and classify it as Group A or Group B (not group A / not share same characteristics).
Now I have a new group (Group C) of 5 million people that is a mix of people in Group A and Group B. Keep in mind I have no training data for B. Ultimately, the job is to classify Group C into either Group A or Group B.
Any suggestions on clustering all that categorical data? Would be SUPER useful to know which groups of variables gravitate together.
Any advice on creating a classifier? I mean, from basic data science it seems you can use a regularized logistic regression or a decision tree? But feeding 100k binary questions for something of 100,000 training doesn't seem like the right way to work, especially if there is no official training data in Group B. Please correct me if i'm wrong. Are there any semi-supervised algorithmns out there?
I was thinking along the lines of bayesian analysis or association rules or heuristic? Need some help in these domain as it is not my expertise and hard finding tutorials on this. The best link i've found for association classification is: http://rstudio-pubs-static.s3.amazonaws.com/369396_e1dfbaf244f84858a6caabb4b3decc88.html
I was thinking the key to the answer lays in the probabilities association with the variable. Because I don't know better, i was thinking about implementing A basic heuristic model. The heuristic rule can be creating a count for all the number of variables that is statistically significantly greater or less than. For example, people in group A have a 70% yes rate for Question 1, the new group has a 50% yes rate. Run a two proportional z test and see if this variable is statistically significant. Repeat this process for Question 2.... Question 100k and set a threshold of 1% alpha. If I find let's 1000 questions to be significant for group A % > new group%, I look at these variables in Group C and create a count of times the person answered yes to these questions (variable X). The same situation applies if for a given question group A % < new group%, I count up the times a person in Group C does that and I make this new variable into Variable Y.
So for example, 50 questions have been determined to have a greater percentage in Group A than Group C. X would be the number of times the person says yes to these questions. 100 questions have been determined to have a lower percentage in Group A than Group C, and Y would boil down to a scale of 0 to 100, so from 100k variables it melts down to 2.
Ultimately as a result, my data would turn from
Person | Question 1 | Question 2 | Question 3 | ..... | Question 100k
Person A | 1 | 0 |1 | ..... | 0
Person B | 1 | 0 | 0 | ..... | 0
Person | # of Questions they said yes to that's statistically greater | # of Questions they said yes to that's statistically smaller
Person A | 1 | 20
Person B | 50 | 30
I don't know the answer, but this was my idea on how to solve this problem. If you guys have an input, ideas, or suggestions. Would really like your thoughts.