0
$\begingroup$

I have a automation system which manages files sent by multiple client inside S3. It so happens client sends file on a regular basis. Let's not consider the content of the file for time being as it can be something of new data or append on old data.

Typically my files are arranged on a pattern:

i) client_a/data_type1/20180610/file_name1_{some_random_text}.csv
ii) client_a/data_type1/20180611/file_name1_{some_random_text}.csv
iii) client_a/data_type1/20180610/file_name2_{some_random_text}.csv
iv) client_a/data_type1/20180611/file_name2_{some_random_text}.csv
v) client_b/data_type2/20180610/file_name3_{some_random_text}.csv
vi) client_b/data_type2/20180611/file_name3_{some_random_text}.csv
vii) client_b/data_type1/20180610/file_name4_{some_random_text}.csv
viii) client_b/data_type1/20180611/file_name4_{some_random_text}.csv

This is a sample of the file management. Now I want to be able to be able to identify unique files. For example case i and ii are same files received on different date. But may be separated by some_random_text (usually date or something). The primary assumptions here are :

  • File name length varies as per client and datatype
  • Random text maybe at the first or end of file name for different files but for not same file. i.e file_name1 will always have random text after the name
  • File name extension will vary on different files.

So I want to build a system which takes realtime input of the files received and do an analysis of the name and path to determine if newer version of he same file is received and place it accordingly (same in the sense data may be added to the file).

I have very limited understanding of machine learning algorithms as of whole and not sure which set of algorithms this requires. I am thinking of looking at clustering algorithms. I just want to sure i start in the right direction. Suggestions will be highly appreciated.

$\endgroup$
8
  • 1
    $\begingroup$ Can you clarify how do you establish that if newer version of he same file is received? $\endgroup$ Feb 19, 2018 at 10:10
  • $\begingroup$ Well we only have the identifier which is file name's initials when we exclude the date. So case i and case ii are same files as the file path is same and file initials are same file_name1_* . Additionally we have a unique ID which is associated with each file that is appended when our automation works. So if the file has same ID then we consider it the same file. $\endgroup$
    – haedes
    Feb 19, 2018 at 12:17
  • $\begingroup$ What programming language do you want to use? $\endgroup$ Feb 19, 2018 at 14:02
  • $\begingroup$ So, you want to main a list of files without that random number suffix and then on receiving every file, you want to compare if you have received this file before? $\endgroup$ Feb 19, 2018 at 14:02
  • $\begingroup$ @EliasStrehle We'll I just plan to use python to model my system. But eventually we plan to use java to integrate it into overall system. $\endgroup$
    – haedes
    Feb 20, 2018 at 4:54

2 Answers 2

1
$\begingroup$

This does not sound like a machine learning problem. It seems to me that you could come up with a list of rules that determine whether two files belong together.

As S van Balen said, you should take a look at the glob and re packages in Python. (If you are on Linux, a Bash script would probably be much more elegant than a solution in Python).

The following links might be helpful:

You mention that files may have different file types. Be careful when converting between types: Make sure for instance that separators and quote characters are treated correctly.

$\endgroup$
0
$\begingroup$

You are looking for the python packages glob and re. I suggest you model the the logic by hand, of the hip it does't sound like something you want to have searched by a machine learning algorithm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.