I'm new to python. I have a text file data set in the form of

[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]

I want to extract the complete data from it, sort it and remove duplicates of the same(i.e., the same row should not appear again).

The Output should look like (in rows):


I want this data for machine learning algorithm implementation, Thank you

  • $\begingroup$ Welcome to the site! I saw your comment to the given answer. Could you provide an estimate to the number of datapoints in all files altogether? Either a number of records or cumulative file size would do. I'm trying to get whether all this could be managed in memory, or you need to use external storage of some sort, possibly a key-value database. $\endgroup$ – mapto Feb 26 '19 at 9:51
  • $\begingroup$ Here is a complete file drive.google.com/open?id=1zIFCJP6GIKvvlAZt_kuqxOgTziAptFZ6 , The file size is around 100kb and I am generating another 150 more files of the same size and format. so i have no issue with the memory or computation speed. $\endgroup$ – Vinay Kumar G Feb 26 '19 at 10:10
  • $\begingroup$ Welcome to the site! The pattern that you post as your output is inconsistent. Can you revise and/or explain exactly what you want. You mention sorting, but the third line of your output is not sorted like the others. $\endgroup$ – I_Play_With_Data Feb 26 '19 at 20:42
  • $\begingroup$ Thank you, I've corrected it. I just want to sort the data as shown in output and the same line should not appear throughout the output. $\endgroup$ – Vinay Kumar G Feb 27 '19 at 6:06

I expanded @glhuilli's answer to also handle the following requirements:

  • Allowing for multiple input files
  • Input with ', rather than "
  • Clean repetitions
  • sort output

Performance and memory usage seem not to be an issue. Below is my resulting code, that I tested with python3, but this code should be backwards compatible:

import json 

collected = set()
files = ['data1.txt']

for f in files:
    with open(f, 'r') as f:
        content = f.read()
        cleaned = content.replace("'", "\"")
    for elt in json.loads(cleaned):
        collected.add(''.join(sorted(elt, reverse=True)))

for elt in sorted(collected, reverse=True):

Notice that in the provided dataset the last list has only 3 elements, its position might appear counterintuitive.

| improve this answer | |
  • $\begingroup$ Thank you sir, Initially I got 'IndentationError: unexpected indent' after removing the white spaces the code is working fine. :-) $\endgroup$ – Vinay Kumar G Feb 27 '19 at 9:40
  • $\begingroup$ In the meantime I fixed the whitespaces (SO formatting issues) and removed the leftover hardcoded data1.txt from the loop body. $\endgroup$ – mapto Feb 27 '19 at 10:00

I think with the volume of data you have, you can try something like this (using the file provided above)

import json 
with open('data1.txt', 'r') as f:
    f_json = f.read().replace('\'', '"')
    d = json.loads(f_json) 
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))
| improve this answer | |
  • $\begingroup$ Thank you for the reply sir, But the data is huge with thousands of arrays in it and i have multiple .txt files with the same format. $\endgroup$ – Vinay Kumar G Feb 26 '19 at 8:43
  • $\begingroup$ I updated the code using the file provided, it works really fast as the sort is just inside each sub-list (size m) and then the iteration is through the whole list only once (n), total would be O(n * m * log(m)). As long as you keep m small then it should not be a problem. $\endgroup$ – glhuilli Feb 26 '19 at 19:07
  • $\begingroup$ It is returning an error, "ValueError: No JSON object could be decoded" $\endgroup$ – Vinay Kumar G Feb 27 '19 at 5:56
  • $\begingroup$ @VinayKumarG, did you see @glhuilli's comment replace ' for " in this file for this to work? $\endgroup$ – mapto Feb 27 '19 at 8:07
  • $\begingroup$ @mapto, I did saw that comment but I didn't get it. Can you please edit the code, moreover I want to include one more functionality (i.e., to remove the repeated output lines after sorting) $\endgroup$ – Vinay Kumar G Feb 27 '19 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.