# how to extract substrings from a dataframe column

I have a dataset as below:

 col1

{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70


I want to add 2 more cols such as col_50 & col 59 as below:

 col2                                        col3
ghgjlj45llj,ljhjlhlhj,clause                jhjhhjxjhj65@,jjjhjhd


I tried for col 2 like below:

   def find_between( s, first, last ):
try:
start = s.index( first ) + len( first )
end = s.index( last, start )
return s[start:end]
except ValueError:
return ""
df['col2']=find_between( df1['col1'].str, "50k:", ":" )
df['col3']=find_between( df1['col1'].str, "59:", ":" )


Such that it extracts all characters between 50k: & untill next ':' appears, however not getting desired result. Can anyone please help?

• Might these strings be much longer? And so with many pattern like 50k:, :59: and so on? Jul 17, 2019 at 9:24
• yes, they can be longer that what I represented. The idea is to extract all the content between "50k:" & the next ':'. Similarly for 59, that appears between '59:' and the next ':'. Jul 17, 2019 at 9:31

In your specific case, when you want to find something between two different markers, you can use the .split(marker) method as follows:

In [1]: s = "{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,j
...: jjhjhd :70"

In [2]: s.split("50k:")[1].split("59:")[0]
Out[2]: ' ghgjlj45llj,ljhjlhlhj,clause '


If you want to get rid of the spaces at the end, just throw in another method at the end: strip():

In [3]: s.split("50k:")[1].split("59:")[0].strip()
Out[3]: 'ghgjlj45llj,ljhjlhlhj,clause'


You can add that to a function as you did with your own code, and put the results into a Pandas Dataframe.

def my_parser(s, marker1, marker2):
"""Extract strings between markers"""
base = s.split(marker1)[1].split(marker2)
part1 = base[0].strip()
part2 = base[1].strip()
return part1, part2


Now use the function to iterate over all thew string you have

results = []    # to collect results of each example
m1 = ...        # put whatever markers you need here
m2 = ...
for line in lines:
results.append(my_parser(line, m1, m2))

# Create a dataframe from a list of lists (i.e. from_records). Each inner list becomes a row
df = pd.DataFrame.from_records(results)


### Edit:

Given you have your strings in a DataFrame already and just want to iterate over them, you could do the following, assuming you have my_col, containing the strings:

for line in df.my_col:
results.append(my_parser(line, m1, m2))    # results is a list as above


So that is what you said you wanted to extract, but it will maybe not generalise well. For example, if there are multiple of those markers in your sentence, you might get unexpected results, or at least only the first occurrence of what you want to extract (if there can be many in your other text examples).

• I tried this before, however for me, the initial col1 is a dataframe column and even though i did df['col1']=df['col1'].astype(str), still it gives and error as : AttributeError: 'Series' object has no attribute 'split' Jul 17, 2019 at 9:37
• @ I guess you took the text as a string & did not consider it as a text content in a dataframe column. could you please help ? Jul 17, 2019 at 9:59
• Have a look at the string accessor for a column: pandas.pydata.org/pandas-docs/stable/user_guide/text.html. There is also a nice extract all method there which might give you more flexibility, as it also accepts regular expressions for pattern matching. Jul 17, 2019 at 11:06
• @sayansen - have a look at my edit. Jul 17, 2019 at 11:17
• By, my_col , you meant col1 as in the data & this new edit is supposed to replace the "for line in lines" part? Jul 17, 2019 at 11:24