Stackoverflow API Structure data storage

Question

I'm downloading Stackoverflow questions & answers for a specific tag using Stackoverflow API and Python.

The goal is to perform document clustering to find relevant terms across the documents and find similarity among them.

Example:

curl --compressed -H "Accept-Encoding: gzip" -X GET 'http://api.stackexchange.com/2.2/questions?page_number=1&pagesize=25&order=desc&sort=activity&tagged=amp-html&site=stackoverflow&run=true'

This is the procedure I'm using with Python 2.7 requests library:

• Request Question via API
• Create Question object and assign API response to body property after using BeautifulSoap.
• Find associated answers in Question.
• Request Answers via API
• Create Answer array
• Assign Answer array to parent Question object.
• For each Answer object assign API response to body property after using BeautifulSoap

API question request:

curl --compressed -H "Accept-Encoding: gzip" -X GET 'https://api.stackexchange.com/2.1/questions/37745529?site=stackoverflow&filter=withbody'

API response example:

{"items":[{"tags":["html","input","amp-html"],"owner":{"reputation":314,"user_id":5426326,"user_type":"registered","accept_rate":39,"profile_image":"https://www.gravatar.com/avatar/cbff9d2f96733be04cb022f8a724ad49?s=128&d=identicon&r=PG&f=1","display_name":"Pranav Bilurkar","link":"http://stackoverflow.com/users/5426326/pranav-bilurkar"},"is_answered":true,"view_count":47,"accepted_answer_id":37787460,"answer_count":1,"score":1,"last_activity_date":1465850038,"creation_date":1465553340,"last_edit_date":1465797429,"question_id":37745529,"link":"http://stackoverflow.com/questions/37745529/input-tags-elements-in-amp-html","title":"Input Tags/Elements in AMP-html","body":"<p>we are implementing our current <a href=\"http://articles.mercola.com/sites/articles/archive/2099/12/31/alzheimers-disease-early-detection.aspx\" rel=\"nofollow\">site</a> in <code>AMP version</code> and i am new to this. I have below queries regarding <code>AMP</code> -</p>\n\n<ol>\n<li>How to get user <code>input</code> from user in <code>AMP-HTML</code>.</li>\n<li>In the above mentioned site [<code>desktop version</code>] we have <code>comment section</code> at the bottom of the page. We need to implement the same functionality in AMP.</li>\n<li>Are there any <code>websites</code> which are build and developed in <code>AMP</code>? If yes, i need links to check.</li>\n</ol>\n\n<p>Any suggestion and ideas related to above queries would be appreciated.\nThanks.</p>\n"}],"has_more":false,"quota_max":300,"quota_remaining":298}

Since questions and answers contain HTML, URLs, special characters, quotes, single quotes, commas, etc in body property. I'm looking to convert this text into structured data which can be represented as a single text chunk and be analyzed as tokens and then I use TF-IDF.

Details:

2 Objects: Question and Answers Question object has an array of answers objects as properties. Each Question and Answer objects contain a body property which is a single string containing each text.

What's the recommended way to store this information?

Answer 1

What do you want to define as your document? You could define a document as a single question, each question-answer pair, or a question with all its answers. How you define a document depends on why you want to cluster documents and how you plan to use this information. From reading your question, my guess is that you want to define a question with all its answer as a document.

Next, it sounds like you want to apply bag of words techniques to extract features (e.g. tf-idf). If so, you would first want to concatenate the question title, question body, and each answer body into one string object. Then I would suggest using a Python package like scikit learn or nltk to do the other preprocessing steps. For example in scikit learn you can then apply TfidfVectorizer to convert a string to a vector.

This is unrelated to your question, but it may be easier to use the data explorer instead of the StackOverflow API. The data explorer allows you to download questions and answers in a large batch and does not have a quota. Unless you need to collect data in real time, the data explorer may be more efficient. It is updated every week and you can use SQL queries to output data tables as csv.