How can I scrape a URL such as: https://www.indeed.com/jobs?q=data+scientist+%2420%2C000&l=New+York&start=10

where I want to get the following parameters: job title, company name, job description, city, location, and salary.

What are the basic steps I need to take to build a scraper?

  • $\begingroup$ What URL is this you are scraping? $\endgroup$
    – JahKnows
    Oct 11, 2018 at 3:47
  • $\begingroup$ i have given URL now.kindly go through it. $\endgroup$
    – jai sundar
    Oct 11, 2018 at 3:56
  • $\begingroup$ Your code is a mess man. What are you trying to scrape from this URL? $\endgroup$
    – JahKnows
    Oct 11, 2018 at 4:06
  • $\begingroup$ job title,company name,job description ,city,location,salary $\endgroup$
    – jai sundar
    Oct 11, 2018 at 4:09

2 Answers 2


Scraping a website is very particular to the case. To try and make this answer useful to other people I will go through some basic concepts as to how this can be done successfully. Below I will use your URL as an example to put everything together.

First we need a means to access the HTML content of the website. For static websites we can use BeautifulSoup. This is a very powerful package which allows you to navigate an HTML. To get the HTML we will use the requests packages.

from bs4 import BeautifulSoup
import requests

url = 'https://www.indeed.com/jobs?q=data+scientist+$20,000&l=New+York&start=10'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')

BeautifulSoup requires a parser, I have had a lot of luck using lxml, however html.parser is also very popular. The parser is what is used to access the HTML tags and identify its inner elements.

Most websites are quite hard to scrape because they will reuse the same name for multiple element tages. This makes it harder to get the elements and extract their values. This is either due to the developers laziness or their expertise to avoid people abusing their resources by running scrapers on their data. To circumvent this you can use this simple BeautifulSoup trick. You can chain your .find calls to hone into the part of the HTML you want to scrape.

Using the above example URL we can identify where in the HTML we can find the results on the page for each job posting.

len(soup.find_all('div', {'class': 'row'}))


Now we will want to collect data from each of these HTML elements. The data we will want to collect is: job title, company name, job description, city, location, and salary.

Each record of data will be collected into a dictionary with all the desired data as listed above. These dictionaries will be collected into a list which will be used to create the pandas DataFrame.

To find the individual data points you can use your browser and inspect elements to find out the name of the element parameters in order to use BeautifulSoup to get the information.

data = []
for i in soup.find_all('div', {'class': 'row'}):
    job_title = i.find('a', {'data-tn-element': 'jobTitle'})['title']
    company_name = i.find('span', {'class': 'company'}).text.strip()    
    job_summary = ''.join([j.text.strip() for j in i.find_all('span', 
                                                              {'class': 'summary'})])
    location = i.find('span', {'class': 'location'})
    if location is not None:
        location = location.text.strip()

    salary_range = i.find('span', {'class': 'no-wrap'})
    if salary_range is not None:
        salary_range = salary_range.text.strip()

    datum = {'job_title': job_title,
             'company_name': company_name,
             'job_summary': job_summary,
             'location': location,
             'salary_range': salary_range}


df = pd.DataFrame(data)

enter image description here


I would recommend using a Selenium Webdriver object which can be scripted to suit your requirement better. You could then perform the exact actions that the user would (using these scripts) and then get the output in the code itself which can be easily captured. Webdriver has bindings with both Python, Java and a few other popular languages.

JSoup is definitely fast, but Selenium Webdriver is more flexible and resilient to the dynamic nature of websites with lots of Javascript actions


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.