Prediction with non-atomic features

Question

I would like to use non-atomic data, as a feature for a prediction. Suppose I have a Table with these features:

- Column 1: Categorical - House
- Column 2: Numerical - 23.22
- Column 3: A Vector - [ 12, 22, 32 ]
- Column 4: A Tree - [ [ 2323, 2323 ],[2323, 2323] , [ Boolean, Categorical ] ]
- Column 5: A List [ 122, Boolean ]

I would like to predict/classify, for instance, Column 2.

I am making something to automatically respond to questions, any type of question, like "Where was Foo Born?" ...

I first make a query to a search engine, then I get some text data as a result, then I do all the parsing stuff (tagging, stemming, parsing, splitting ... )

My first approach was to make a table, each row with a line of text and a lot of features, like "First Word", "Tag of First Word", "Chunks", etc...

But with this approach I am missing the relationships between the sentences.

I would like to know if there is an algorithm that looks inside the tree structures (or vectors) and makes the relations and extract whatever is relevant for predicting/classifying. I'd prefer to know about a library that does that than an algorithm that I have to implement.

Answer 1

When it comes to dealing with many disparate kinds of data, especially when the relationships between them are unclear, I would strongly recommend a technique based on decision trees, the most popular ones today to the best of my knowledge are random forest, and extremely randomized trees.

Both have implementations in sklearn, and they are pretty straightforward to use. At a very high level, the reason that a decision tree-based approach is advantageous for multiple disparate kinds of data is because decision trees are largely independent from the specific data they are dealing with, just so long as they are capable of understanding your representation.

You'll still have to fit your data into a feature vector of some kind, but based on your example that seems like a pretty straightforward task, and if you're willing to go a little deeper on your implementation you could certainly come up with a custom tree-splitting rule without actually having to change anything in the underlying algorithm. The original paper is a pretty decent place to start if you want to give that a shot.

If you want pseudo-structural data from your text data though, I might suggest looking into doc2vec, recently developed by Google. I don't think there are any good open-source implementations now, but it's a pretty straightforward improvement on the word2vec algorithm, which has implementations in at least C and python.

Hope that helps! Let me know if you've got any more questions.