This should have an easy solution but I can't understand how to avoid it.

    content1 = soup.find('div', class_='fusion-text fusion-text-6')
    content2 = soup.find('div', class_='fusion-text fusion-text-7')

    for para in content1:
        comp_name = para.find_all('a')['href']

The error comes because I have a list at ['href'] together with find_all.

        comp_name = para.find('a')['href']

doesn't return an error and gives the right output (an URL), but just the first one. Since I want to scrape all of them inside my content1 I wanted to use find_all but inevitably I get the error.

How can I avoid this?

Thank you.

  • $\begingroup$ if the result is a list, you can access list elements like mylist[0] and you can pull out elements from 0 to len(mylist) $\endgroup$ – Peter Jul 29 '20 at 16:25
  • $\begingroup$ @Peter I'm not sure if I understood it. How can I access it if it returns an error? I don't know how to create a list out of find_all. Thanks. $\endgroup$ – Steven Jul 29 '20 at 16:34
  • $\begingroup$ comp_name = para.find_all('a')[0].get('href') or comp_name = para.find('a')['href'] return 1 result, but I can't get all of them. $\endgroup$ – Steven Jul 29 '20 at 17:52

This will give you list of companies then you need an other loop/or list comprehension to extract from the children.

 for para in content1:
    comp_name = para.find_all('a')
    all_comps = [one_para.get("href") for one_para in para]
    print(all_comps ) 
  • $\begingroup$ this is a copy of my answer :( $\endgroup$ – Nikos M. Jul 29 '20 at 20:45
  • $\begingroup$ @Nauman thanks, I kinda get what you wrote but I get this: AttributeError: 'NavigableString' object has no attribute 'get' $\endgroup$ – Steven Jul 29 '20 at 21:15

Use list comprehensions:

comp_name = [link['href'] for link in para.find_all('a')]
  • $\begingroup$ thanks, it works but I can't get the text, only html. I tried .text but as usual find_all doesn't accept that. $\endgroup$ – Steven Jul 29 '20 at 21:17
  • 1
    $\begingroup$ @Steven, change link['href'] to link.get_text() to get the text $\endgroup$ – Nikos M. Jul 29 '20 at 22:43
  • $\begingroup$ alright, thanks. Just a note, with that from url-html I get the other text at the end of the tag and not urls anymore $\endgroup$ – Steven Jul 30 '20 at 6:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.