Extracting data, sorting and removing duplicates from an array using python 2.7

Question

I have a text file data set in the form of

[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]

I want to extract the complete data from it, sort it and remove duplicates of the same (i.e., the same row should not appear again).

The output should look like (in rows):

11090702
11090404
07060202
09090606
09090604

I want this data for machine learning algorithm implementation.

Answer 1

I expanded @glhuilli's answer to also handle the following requirements:

• Allowing for multiple input files
• Input with ', rather than "
• Clean repetitions
• sort output

Performance and memory usage seem not to be an issue. Below is my resulting code, that I tested with python3, but this code should be backwards compatible:

import json 

collected = set()
files = ['data1.txt']

for f in files:
    with open(f, 'r') as f:
        content = f.read()
        cleaned = content.replace("'", "\"")
    for elt in json.loads(cleaned):
        collected.add(''.join(sorted(elt, reverse=True)))

for elt in sorted(collected, reverse=True):
    print(elt)

Notice that in the provided dataset the last list has only 3 elements, its position might appear counterintuitive.

Answer 2

I think with the volume of data you have, you can try something like this (using the file provided above)

import json 
with open('data1.txt', 'r') as f:
    f_json = f.read().replace('\'', '"')
    d = json.loads(f_json) 
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))