I have a text file data set in the form of

[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]

I want to extract the complete data from it, sort it and remove duplicates of the same (i.e., the same row should not appear again).

The output should look like (in rows):


I want this data for machine learning algorithm implementation.

  • $\begingroup$ Welcome to the site! I saw your comment to the given answer. Could you provide an estimate to the number of datapoints in all files altogether? Either a number of records or cumulative file size would do. I'm trying to get whether all this could be managed in memory, or you need to use external storage of some sort, possibly a key-value database. $\endgroup$
    – mapto
    Feb 26, 2019 at 9:51
  • $\begingroup$ Here is a complete file drive.google.com/open?id=1zIFCJP6GIKvvlAZt_kuqxOgTziAptFZ6 , The file size is around 100kb and I am generating another 150 more files of the same size and format. so i have no issue with the memory or computation speed. $\endgroup$ Feb 26, 2019 at 10:10
  • $\begingroup$ Welcome to the site! The pattern that you post as your output is inconsistent. Can you revise and/or explain exactly what you want. You mention sorting, but the third line of your output is not sorted like the others. $\endgroup$ Feb 26, 2019 at 20:42
  • $\begingroup$ Thank you, I've corrected it. I just want to sort the data as shown in output and the same line should not appear throughout the output. $\endgroup$ Feb 27, 2019 at 6:06

2 Answers 2


I expanded @glhuilli's answer to also handle the following requirements:

  • Allowing for multiple input files
  • Input with ', rather than "
  • Clean repetitions
  • sort output

Performance and memory usage seem not to be an issue. Below is my resulting code, that I tested with python3, but this code should be backwards compatible:

import json 

collected = set()
files = ['data1.txt']

for f in files:
    with open(f, 'r') as f:
        content = f.read()
        cleaned = content.replace("'", "\"")
    for elt in json.loads(cleaned):
        collected.add(''.join(sorted(elt, reverse=True)))

for elt in sorted(collected, reverse=True):

Notice that in the provided dataset the last list has only 3 elements, its position might appear counterintuitive.

  • $\begingroup$ Thank you sir, Initially I got 'IndentationError: unexpected indent' after removing the white spaces the code is working fine. :-) $\endgroup$ Feb 27, 2019 at 9:40
  • $\begingroup$ In the meantime I fixed the whitespaces (SO formatting issues) and removed the leftover hardcoded data1.txt from the loop body. $\endgroup$
    – mapto
    Feb 27, 2019 at 10:00

I think with the volume of data you have, you can try something like this (using the file provided above)

import json 
with open('data1.txt', 'r') as f:
    f_json = f.read().replace('\'', '"')
    d = json.loads(f_json) 
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))
  • $\begingroup$ Thank you for the reply sir, But the data is huge with thousands of arrays in it and i have multiple .txt files with the same format. $\endgroup$ Feb 26, 2019 at 8:43
  • $\begingroup$ I updated the code using the file provided, it works really fast as the sort is just inside each sub-list (size m) and then the iteration is through the whole list only once (n), total would be O(n * m * log(m)). As long as you keep m small then it should not be a problem. $\endgroup$
    – glhuilli
    Feb 26, 2019 at 19:07
  • $\begingroup$ It is returning an error, "ValueError: No JSON object could be decoded" $\endgroup$ Feb 27, 2019 at 5:56
  • $\begingroup$ @VinayKumarG, did you see @glhuilli's comment replace ' for " in this file for this to work? $\endgroup$
    – mapto
    Feb 27, 2019 at 8:07
  • $\begingroup$ @mapto, I did saw that comment but I didn't get it. Can you please edit the code, moreover I want to include one more functionality (i.e., to remove the repeated output lines after sorting) $\endgroup$ Feb 27, 2019 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.