Timeline for Converting Json file to Dataframe Python
Current License: CC BY-SA 3.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 11, 2017 at 22:13 | comment | added | user40285 | I recommend you split your dictionary into multiple dictionaries, thereby converting each one into separate dataframes. Pandas DataFrame conversions work by parsing through a list of dictionaries and converting them to df rows per dict. If there are too many child structures in your dicts, such as a "list of dicts containing another list of dicts" times 2, then you need to restructure you data model. You may need to include "foreign keys" as extra columns in your df so you can reference data properly. I hope this was helpful advice. | |
| Oct 11, 2017 at 21:31 | vote | accept | iprof0214 | ||
| Oct 11, 2017 at 21:31 | comment | added | iprof0214 | @Dendekker - df = pd.io.json.json_normalize(dict['log_data']) did the magic. However there is one json object that is a list and not dictionary, its failing to flatten this object requestParameters":{"filterSet":{"items":[{"name":"resource-type","valueSet":{"items":[{"value":""}]}},{"name":"tag:User Name","valueSet":{"items":[{"value":""}]}}]}} is being returned in the dataframe as follows requestParameters.filterSet.items [{u'name': u'resource-type', u'valueSet': {u'items":[{"u'value":"*"}] | |
| Oct 11, 2017 at 6:50 | comment | added | user40285 |
@Sneha dict = json.loads(js);df = pd.io.json.json_normalize(dict['Records']) Doesn't this flatten out your multi structure json into a 2d dataframe? You would need more than 2 records to see if the dataframe properly repeats the data within the child structures of your json.
|
|
| Oct 10, 2017 at 22:40 | comment | added | iprof0214 | Thank you Dendekker, I appreciate your response. I tried converting to a csv file and then to data frame. To a certain extent it worked (please see my updates to the question). However the nested json objects are as it is. As per your suggestion, since there are multiple nested objects if we separate each nested object into a separate dataframe then aren't we looking at a much complex solution given the fact that we would have to combine them later? Would you be able to elaborate on your approach? | |
| Oct 10, 2017 at 19:37 | history | edited | user40285 | CC BY-SA 3.0 |
Addressed your specific data structure scenario.
|
| Oct 10, 2017 at 19:33 | review | First posts | |||
| Oct 10, 2017 at 21:28 | |||||
| Oct 10, 2017 at 19:31 | history | answered | user40285 | CC BY-SA 3.0 |