# How to extract data, sort and remove duplicates from an array using python 2.7

I'm new to python. I have a text file data set in the form of

[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]


I want to extract the complete data from it, sort it and remove duplicates of the same(i.e., the same row should not appear again).

The Output should look like (in rows):

11090702
11090404
07060202
09090606
09090604


I want this data for machine learning algorithm implementation, Thank you

• Welcome to the site! I saw your comment to the given answer. Could you provide an estimate to the number of datapoints in all files altogether? Either a number of records or cumulative file size would do. I'm trying to get whether all this could be managed in memory, or you need to use external storage of some sort, possibly a key-value database. – mapto Feb 26 at 9:51
• Here is a complete file drive.google.com/open?id=1zIFCJP6GIKvvlAZt_kuqxOgTziAptFZ6 , The file size is around 100kb and I am generating another 150 more files of the same size and format. so i have no issue with the memory or computation speed. – Vinay Kumar G Feb 26 at 10:10
• Welcome to the site! The pattern that you post as your output is inconsistent. Can you revise and/or explain exactly what you want. You mention sorting, but the third line of your output is not sorted like the others. – I_Play_With_Data Feb 26 at 20:42
• Thank you, I've corrected it. I just want to sort the data as shown in output and the same line should not appear throughout the output. – Vinay Kumar G Feb 27 at 6:06

I expanded @glhuilli's answer to also handle the following requirements:

• Allowing for multiple input files
• Input with ', rather than "
• Clean repetitions
• sort output

Performance and memory usage seem not to be an issue. Below is my resulting code, that I tested with python3, but this code should be backwards compatible:

import json

collected = set()
files = ['data1.txt']

for f in files:
with open(f, 'r') as f:
cleaned = content.replace("'", "\"")

for elt in sorted(collected, reverse=True):
print(elt)


Notice that in the provided dataset the last list has only 3 elements, its position might appear counterintuitive.

• Thank you sir, Initially I got 'IndentationError: unexpected indent' after removing the white spaces the code is working fine. :-) – Vinay Kumar G Feb 27 at 9:40
• In the meantime I fixed the whitespaces (SO formatting issues) and removed the leftover hardcoded data1.txt from the loop body. – mapto Feb 27 at 10:00

I think with the volume of data you have, you can try something like this (using the file provided above)

import json
with open('data1.txt', 'r') as f:
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))

• Thank you for the reply sir, But the data is huge with thousands of arrays in it and i have multiple .txt files with the same format. – Vinay Kumar G Feb 26 at 8:43
• I updated the code using the file provided, it works really fast as the sort is just inside each sub-list (size m) and then the iteration is through the whole list only once (n), total would be O(n * m * log(m)). As long as you keep m small then it should not be a problem. – glhuilli Feb 26 at 19:07
• It is returning an error, "ValueError: No JSON object could be decoded" – Vinay Kumar G Feb 27 at 5:56
• @VinayKumarG, did you see @glhuilli's comment replace ' for " in this file for this to work? – mapto Feb 27 at 8:07
• @mapto, I did saw that comment but I didn't get it. Can you please edit the code, moreover I want to include one more functionality (i.e., to remove the repeated output lines after sorting) – Vinay Kumar G Feb 27 at 8:26