I'm downloading Stackoverflow questions & answers for a specific tag using Stackoverflow API and Python.
The goal is to perform document clustering to find relevant terms across the documents and find similarity among them.
Example:
curl --compressed -H "Accept-Encoding: gzip" -X GET 'http://api.stackexchange.com/2.2/questions?page_number=1&pagesize=25&order=desc&sort=activity&tagged=amp-html&site=stackoverflow&run=true'
This is the procedure I'm using with Python 2.7 requests library:
- Request Question via API
- Create Question object and assign API response to body property after using BeautifulSoap.
- Find associated answers in Question.
- Request Answers via API
- Create Answer array
- Assign Answer array to parent Question object.
- For each Answer object assign API response to body property after using BeautifulSoap
API question request:
curl --compressed -H "Accept-Encoding: gzip" -X GET 'https://api.stackexchange.com/2.1/questions/37745529?site=stackoverflow&filter=withbody'
API response example:
{"items":[{"tags":["html","input","amp-html"],"owner":{"reputation":314,"user_id":5426326,"user_type":"registered","accept_rate":39,"profile_image":"https://www.gravatar.com/avatar/cbff9d2f96733be04cb022f8a724ad49?s=128&d=identicon&r=PG&f=1","display_name":"Pranav Bilurkar","link":"http://stackoverflow.com/users/5426326/pranav-bilurkar"},"is_answered":true,"view_count":47,"accepted_answer_id":37787460,"answer_count":1,"score":1,"last_activity_date":1465850038,"creation_date":1465553340,"last_edit_date":1465797429,"question_id":37745529,"link":"http://stackoverflow.com/questions/37745529/input-tags-elements-in-amp-html","title":"Input Tags/Elements in AMP-html","body":"<p>we are implementing our current <a href=\"http://articles.mercola.com/sites/articles/archive/2099/12/31/alzheimers-disease-early-detection.aspx\" rel=\"nofollow\">site</a> in <code>AMP version</code> and i am new to this. I have below queries regarding <code>AMP</code> -</p>\n\n<ol>\n<li>How to get user <code>input</code> from user in <code>AMP-HTML</code>.</li>\n<li>In the above mentioned site [<code>desktop version</code>] we have <code>comment section</code> at the bottom of the page. We need to implement the same functionality in AMP.</li>\n<li>Are there any <code>websites</code> which are build and developed in <code>AMP</code>? If yes, i need links to check.</li>\n</ol>\n\n<p>Any suggestion and ideas related to above queries would be appreciated.\nThanks.</p>\n"}],"has_more":false,"quota_max":300,"quota_remaining":298}
Since questions and answers contain HTML, URLs, special characters, quotes, single quotes, commas, etc in body property. I'm looking to convert this text into structured data which can be represented as a single text chunk and be analyzed as tokens and then I use TF-IDF.
Details:
2 Objects: Question and Answers Question object has an array of answers objects as properties. Each Question and Answer objects contain a body property which is a single string containing each text.
What's the recommended way to store this information?