How to extract data, sort and remove duplicates from an array using python 2.7

Question

I'm new to python. I have a text file data set in the form of

[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]

I want to extract the complete data from it, sort it and remove duplicates of the same(i.e., the same row should not appear again).

The Output should look like (in rows):

11090702
11090404
07060202
09090606
09090604

I want this data for machine learning algorithm implementation, Thank you

Answer 1

I expanded @glhuilli's answer to also handle the following requirements:

• Allowing for multiple input files
• Input with ', rather than "
• Clean repetitions
• sort output

Performance and memory usage seem not to be an issue. Below is my resulting code, that I tested with python3, but this code should be backwards compatible:

import json 

collected = set()
files = ['data1.txt']

for f in files:
    with open(f, 'r') as f:
        content = f.read()
        cleaned = content.replace("'", "\"")
    for elt in json.loads(cleaned):
        collected.add(''.join(sorted(elt, reverse=True)))

for elt in sorted(collected, reverse=True):
    print(elt)

Notice that in the provided dataset the last list has only 3 elements, its position might appear counterintuitive.

Answer 2

I think with the volume of data you have, you can try something like this (using the file provided above)

import json 
with open('data1.txt', 'r') as f:
    f_json = f.read().replace('\'', '"')
    d = json.loads(f_json) 
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))