Hot answers tagged

6

Since you were working with RegEx, I will ofer a RegEx solution. I will also show that you need to also take care to first remove punctuation. (I will not go down the rabbit-hole of re-sinserting the punctuation back where it was!) A RegEx solution: import re sentence = 'I need need to learn regex... regex from scratch!' # remove punctuation # the ...


3

Build a dictionary of common words that frequently appear in these documents (e.g., MEDICAL, SEX, AGE, etc). Then, for each word in the output from OCR, check whether it is similar to a word in your dictionary; if so, then replace it with the dictionary word. "Similar" might be defined as "edit distance <= 1". For example, your sample output has the ...


3

It's possible if you define CountVectorizer's token_pattern argument. If you're new to regular expressions, Python's documentation goes over how it deals with regular expressions using the re module (and scikit-learn uses this under the hood) and I recommend using an online regex tester like this one, which gives you immediate feedback on whether your ...


2

To remove multiple white space matches, you will need [\s\n]+, note the inclusion of the + (match one or more). Code: Here is a function which will build the regex automatically from a text snippet: def remove_words(to_clean, words, flags=re.IGNORECASE): regex = r'[\s\n]+'.join([''] + words.split() + ['']) return re.sub(regex, ' ', to_clean, flags)...


2

This splits the string and filters the substrings by length, no regex: string <- "blabla a test this that b á" paste(Filter(function(x) nchar(x) > 1, unlist(strsplit(string, " "))), collapse=" ") # [1] "blabla test this that" (does not work with multiple space characters)


2

nice that you solved it. This would also work: import re def eventdetails(text): match =re.search(r'Event Details :(.*)',text, flags=re.S) return match result = eventdetails(text) print(result.group(0))


2

I got my answer , NLTK is good to go for this problem. You may use sutime with python wrapper : Python wrapper for Stanford CoreNLP's SUTime The usual approach in NLP is to collect a dataset required for training. Process that dataset so that the words in the dataset are converted into numbers. One simple example of converting it into numbers is to ...


2

Well, as you state the problem, it is true that the search for a certain sequence of strings/words is the same as looking for the corresponding n-gram. However, keep in mind that an n-gram, when you use it for ML, (often) is represented as a factor. So a certain sequence of words or strings, is thought of carrying valuable information. Like „John Holmes“ ...


1

From what I understood, doing this with regex rules can be tricky and a bit slow. In python, I use this and it works perfectly : from itertools import groupby line_split = [k for k,v in groupby(line.split())]


1

This doesn't seem like a great task for regex--even your pattern would miss very close typos like COWID-19 or potential OCR mistakes like C0VID-I9. Instead, I'd suggest using the stringdist package to do fuzzy matching, perhaps stringdist::afind to find approximate matches of "COVID-19". You can read a bit about it here. This will let you select ...


1

You need to apply two regexes: first, get r'^test:.*$' with m option, then your original regex on the result of the first.


1

here's what I've tried: import pandas import re KW = pd.read_csv("regex.csv") DF = pd.read_csv("Default-Profile.csv") for i in range (0,KW.kws.size): for j in range (0,DF.field_name.size): if (DF.field_name.match(r"KW.kws[i]") == True && (DF.field_name.match(r"mand_kw[i]" == True)): ...


1

You could look at string similarity measures and TFIDF (usually with cosine). If you want a measure which works at both levels of words and sentences, there are more advanced options such as SoftTFIDF.


1

Is there any better way to validate this piece of python code. ? No, because any automatic method would be equivalent to creating another learner, and there would be no way to know if it's good or bad at the job. So unless you find another annotated dataset (i.e. a set of resumes with labels indicating whether or not they switched jobs in the past year), ...


1

Use strsplit(). If you want to remove the space between the two words (along with 'NSW'): city_clean <- unlist(c(strsplit(city, " NSW"))) Output: [1] "Sydney" "Newcastle" "Liverpool" "Broken Hill" It wasn't clear to me whether you wanted to keep the space or not. If you want to keep the trailing space after the city name city_clean <- ...


1

Try this if you decide to use pandas: readFile = pd.read_csv("C:/Users/siddhesh.kalgaonkar/Desktop/data01.txt",header=None) readFile.columns = ['IP'] readFile['IP'] = readFile['IP'].replace(regex='((?<=[0-9])[0-9]|(?<=\.)[0-9])',value='X') print(readFile) and this without pandas: readFile = open("C:/Users/siddhesh.kalgaonkar/Desktop/data01.txt","r") ...


1

This is pure(almost) python: list(map(lambda x: x[0] + '.'.join(['X' * len(c) for c in x[1:].split('.')]), my_df['IP'])) Explanation: - Use the map to iterate over each row in my_df['IP'] column. - Per each IP value, split into first char and others using the x[0], x[1:] notation. - Get each part in x[1:] using the split method. - For each part get ...


1

PoS tagging works for natural language only and identifies grammatical parts of the sentence, nothing more. LSTM is an algorithm that can be used to predict series. Named Entity Recognition (NER) and Terminology Extraction could work if you have already data to engage in Information Extraction (IE). However, In order to use a these techniques you need to ...


1

If you need optional parts in the pattern then regular expressions were specifically built for those kind of problems. You could probably speed up regular expressions by using loops in Cython, but the time spent programming it is not really worth it. If your pattern is fixed then there are faster ways to do pattern matching. OpenCV uses Descrete Fourier ...


1

I am getting that you want extract numbers with either '$' or even not without a dollar. Why are not using just [0-9]+ for that field? The results will be like some $1 something >> ['1'] some other 115 >> ['115'] $115 some thing >> ['115'] If you want to include '\$' sign, then '\$' can be repeated zero or once or by typo ...


1

No- not if your regex is actually generating the data. If this is the case, the regex (or a complementary one) should be able to classify the resulting text perfectly. That is, you know what the rules for creating a 'happy' or 'sad' piece of text are within the regex, and so you simply apply those rules backwards to classify it. So, if you take what the ...


1

It may not be perfect but does the job almost. import re re.findall(r'(?<=: )\w{2}-\w{3}-\d{4}|(?<=: )\d{2}-\w{3}-\w{2}|(?<=: )\s?\w+\s?\w+\s?\w+',data) #['Thomas Joseph MRNO','DQ026151','Haneef M An','513','Male','19-Feb-2V','IDOGIII','22-Feb-20','E2-Feb-2017']


1

As @spacedman correctly mentioned, this will answered quicker at StackOverflow. But you can use this to create a dictionary like this. There might be a better way but this is a quick work around. # -*- coding: utf-8 -*- import re st = '''Pafient Name : Thomas Joseph MRNO : DQ026151? Doctor : Haneef M An : 513! Gandar : Male Admission Data : 19-Feb-2V'3‘¥T12:...


1

Optical character recognition is a well-studied problem with many possible solutions (ressources). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers: Backpropagation Applied to Handwritten zip code Comparaison of Classifier Methods: A Case Study in Handwritten Digit Recognition Here is a ...


Only top voted, non community-wiki answers of a minimum length are eligible