how to extract substrings from a dataframe column

Question

I have a dataset as below:

 col1    

{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,jjjhjhd :70

I want to add 2 more cols such as col_50 & col 59 as below:

 col2                                        col3
 ghgjlj45llj,ljhjlhlhj,clause                jhjhhjxjhj65@,jjjhjhd

I tried for col 2 like below:

   def find_between( s, first, last ):
      try:
        start = s.index( first ) + len( first )
        end = s.index( last, start )
        return s[start:end]
     except ValueError:
        return ""
  df['col2']=find_between( df1['col1'].str, "50k:", ":" )
  df['col3']=find_between( df1['col1'].str, "59:", ":" )

Such that it extracts all characters between 50k: & untill next ':' appears, however not getting desired result. Can anyone please help?

Answer 1

In your specific case, when you want to find something between two different markers, you can use the .split(marker) method as follows:

In [1]: s = "{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,j
   ...: jjhjhd :70"

In [2]: s.split("50k:")[1].split("59:")[0]                                        
Out[2]: ' ghgjlj45llj,ljhjlhlhj,clause '

If you want to get rid of the spaces at the end, just throw in another method at the end: strip():

In [3]: s.split("50k:")[1].split("59:")[0].strip()                                
Out[3]: 'ghgjlj45llj,ljhjlhlhj,clause'

You can add that to a function as you did with your own code, and put the results into a Pandas Dataframe.

def my_parser(s, marker1, marker2):
    """Extract strings between markers"""
    base = s.split(marker1)[1].split(marker2)
    part1 = base[0].strip()
    part2 = base[1].strip()
    return part1, part2

Now use the function to iterate over all thew string you have

results = []    # to collect results of each example
m1 = ...        # put whatever markers you need here
m2 = ...
for line in lines:
    results.append(my_parser(line, m1, m2))

# Create a dataframe from a list of lists (i.e. from_records). Each inner list becomes a row
df = pd.DataFrame.from_records(results)

Edit:

Given you have your strings in a DataFrame already and just want to iterate over them, you could do the following, assuming you have my_col, containing the strings:

for line in df.my_col:
    results.append(my_parser(line, m1, m2))    # results is a list as above

---

So that is what you said you wanted to extract, but it will maybe not generalise well. For example, if there are multiple of those markers in your sentence, you might get unexpected results, or at least only the first occurrence of what you want to extract (if there can be many in your other text examples).