0
$\begingroup$

Is there a SOTA solution for finding texts with contact information (phone numbers, social media links, etc.)?

I know that this task is advised to solve by regular expressions, I've already tried it myself, but there are problems with numbers of this type: "8 994 966 twelve 72", "393003*", "912 five69 O7 OO", "8918ob801ra70sha10s" etc. If you write regular expressions under all of them, they already start to affect regular numbers, which don't need to be deleted. I tried to use NER models, but they don't see phone numbers specifically.

I am not asking to write a universal regular expression, I am specifically interested in a solution using machine learning techniques (I tried NER for example, but it did not work). I also put off using BERT and similar SOTA models, because there are no good embeddings for them and my computational power and time are limited :(.

Thank you in advance.

$\endgroup$
1
  • $\begingroup$ Welcome to DataScienceSE. NER is actually your best bet, but it's not going to be perfect, especially with a pretrained model. If you want the model to work better for your specific data, you need to train the NER model, but this means you have to manually annotate a large amount of data. $\endgroup$
    – Erwan
    Aug 21 at 16:09

1 Answer 1

1
$\begingroup$

Your main problem may come from a lack of incorrect data, rather than the model itself.

That's why you could generate incorrect data with its corrected values quite easily by using random rules.

For instance, to create 1000 wrong phone numbers:

import random

def random_phone_num_generator():
    first = str(random.randint(100, 999))
    second = str(random.randint(1, 888)).zfill(3)
    last = (str(random.randint(1, 9998)).zfill(4))
    while last in ['1111', '2222', '3333', '4444', '5555', '6666', '7777', '8888']:
        last = (str(random.randint(1, 9998)).zfill(4))
    return '{}-{}-{}'.format(first, second, last)


n = 1000


wrong_numbers = []

right_numbers = []


for i in range(0, n):
    right_numbers.append(random_phone_num_generator())
    
    

error_probability = 0.1
phone_number_len = 12


def random_option(options_list = []):
    
    len_options = len(options_list)-1
    
    random_position = random.randint(0, len_options)
    
    return options_list[random_position]


#code to have one char error in a phone number
for i in range(0, len(right_numbers)):
    
    current_phone_number = right_numbers[i]
    
    random_float = random.uniform(0, 1)
    
    random_position = int(random_float * phone_number_len)
    
    current_char_random_position = current_phone_number[random_position]
    
    new_error_char = ''
    if current_char_random_position == '-':
        new_error_char = random_option(['dash','~','/'])
    elif current_char_random_position == '1':
        new_error_char = random_option(['one','One'])
    elif current_char_random_position == '2':
        new_error_char = random_option(['two','tWo'])    
    elif current_char_random_position == '3':
        new_error_char = random_option(['three','Three','E'])            
    elif current_char_random_position == '4':
        new_error_char = random_option(['four','Four'])    
    elif current_char_random_position == '5':
        new_error_char = random_option(['five','Five'])            
    elif current_char_random_position == '6':
        new_error_char = random_option(['six','Six','b'])        
    elif current_char_random_position == '7':
        new_error_char = random_option(['seven','Seven','T'])    
    elif current_char_random_position == '8':
        new_error_char = random_option(['height','Height'])    
    elif current_char_random_position == '9':
        new_error_char = random_option(['nine','Nine'])    
    elif current_char_random_position == '0':
        new_error_char = random_option(['zero','Zero','O'])    
        
    new_error_char = random_option(['',' ']) + new_error_char + random_option(['',' ']) #random spaces around the number
    new_error_string = current_phone_number[:random_position] + new_error_char + current_phone_number[random_position+1:]
    
    wrong_numbers.append(new_error_string)
    

print(wrong_numbers)

Now you can train any model (including NER) with a lot of incorrect and correct data.

Note that you have to generate incorrect data for each scenario you can have.

This wouldn't require a lot of calculation as long as the generated incorrect/correct data is not too important compared to the real data you have (could be about 10%) in order to be close enough to reality.

$\endgroup$
1
  • $\begingroup$ Does it answer your question? If not, please let me know. $\endgroup$ Sep 9 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.