0
$\begingroup$

I want to use machine learning and NLP to convert semi-structured data in text files to structured data by predicting the patterns in the files and splitting the fields for example if I have a text file that looks like this :

Input :

2021565267MALL1ETAGE ZARA1st FLOOR 2345561
2022565267MALL2ETAGE ZARA1st FLOOR 2345561
2022565267ANFAPLACE2ETAGECOFEESHOP2345561
20225652634ANFAPLACE2ETAGE 2345561

Desired Output :

2021565267,MALL1ETAGE ZARA1st FLOOR,2345561
2022565267,MALL2ETAGE ZARA1st FLOOR,2345561
2022565267,ANFAPLACE2ETAGECOFEESHOP,2345561
20225652634,ANFAPLACE2ETAGE,2345561

The semi-structured files are not fixed-width so we can not just add col specification in pandas like this ( it can work for the first line for example ) :

col_specification =[(1, 10),.... ]

One of the approaches that I found online is to make a dictionary based on the occurrences of the words in the semi-structured file will that work in this situation if so how can I implement something like that?

$\endgroup$

1 Answer 1

0
$\begingroup$

I am making the following assumptions:

  1. The first column of numbers is the first few digits/numbers in each line
  2. The last column of numbers is the last few digits/numbers in each line
  3. So, in the second column, texts like MALL1ETAGE ZARA1st FLOOR should not have numbers in the front or end. So, 20215652671MALL1ETAGE ZARA1st FLOOR 2345561 will be interpreted as 20215652671, MALL1ETAGE ZARA1st FLOOR, 2345561 and NOT 2021565267, 1MALL1ETAGE ZARA1st FLOOR, 2345561

To this end, you can use the following that makes use of regular expression to capture the first and last group of numbers:

import re

re.findall("^([0-9]+)(.*?)([0-9]+)$", "2021565267MALL1ETAGE ZARA1st FLOOR 2345561")
# [('2021565267', 'MALL1ETAGE ZARA1st FLOOR ', '2345561')]

Applying strip() to each of the result removes the trailing whitespace to get the output you desire.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.