0
$\begingroup$

I have this script:

    import requests
    from requests import get
    from bs4 import BeautifulSoup
    import csv
    import pandas as pd

    f = open('olanda.csv', 'wb')
    writer = csv.writer(f)

    url = ('https://www......')
    response = get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    type(soup)

    table = soup.find('table', id='tablepress-94').text.strip()

    print(table)

    writer.writerow(table.split(), delimiter = ',')

    f.close()

When it writes to a CSV file it writes everything in a single cell like that:

Sno.,CompanyLocation,1KarifyNetherlands,2Umenz,Benelux,BVNetherlands,3TovertafelNetherlands,4Behandeling,BegrepenNetherlands,5MEXTRANetherlands,6Sleep.aiNetherlands,7OWiseNetherlands,8Healthy,WorkersNetherlands,9&thijs,|,thuis,in,jouw,situatieNetherlands,10HerculesNetherlands, etc.

I wanted to have the output in a single column and each value (separated by comma) in a single row.

I tried to use delimiter = ',' but I got:

TypeError: a bytes-like object is required, not 'str'.

Screesnhot of table: table

How can I do this? Thanks!

$\endgroup$
0
$\begingroup$

Please see doc of csv.writer

delimiter param is used when instantiating the writer, not on writerow method.

Furthermore from what I understand, your approach is wrong. For example you want to scrape the html table and store it as csv file.

But what you do is get the text from table

table = soup.find('table', id='tablepress-94').text.strip()

which is a concatenated string of all columns with no structure and try to split by nothing, which is wrong

while you should be getting a dict of table values, eg using:

table = soup.find('table', id='tablepress-94')
# assuming table has a
# <thead><tr><td>Column 1</td>..<td>Column n</td></tr></thead>
# section with column headers, else adjust accordingly
column_headers = [td.get_text() for td in table.find('thead').find('tr').find_all('th')]
print(column_headers)
data = {}
row_index = 0
for row in table.find('tbody').find_all('tr'):
    row_index += 1
    column_index = 0
    columns = row.find_all('td')
    for td in columns:
        if column_index >= len(column_headers):
            column_headers.append('Column_'+str(column_index+1))
        if column_headers[column_index] not in data:
            data[column_headers[column_index]] = []
        data[column_headers[column_index]].append(td.get_text())
        column_index += 1

print(data)
# now output your scarped table data, data into csv properly
writer.writerow(column_headers) # print headers

# print each row of data
for row in range(row_index):
    writer.writerow([data[column][row] for column in column_headers])
$\endgroup$
12
  • $\begingroup$ True, thanks I didn't notice it. However, it doesn't work with .split() because I get writer.writerow(table.split()) TypeError: 'NoneType' object is not callable but I need .split() because I get written html code and not text. $\endgroup$ – Steven Jul 27 '20 at 15:22
  • $\begingroup$ @Steven your approach as far as I see is wrong, see updated answer and hope it helps. If it helps you should consider acceptng it. Cheers! $\endgroup$ – Nikos M. Jul 27 '20 at 19:26
  • $\begingroup$ Thanks, I appreciate it. With a simple print(data) I actually get what I want, but when it comes to print it doesn't work. So I tried your code and I get if column_headers[column_index] not in data: data[column_headers[column_index]] = [] IndexError: list index out of range I was trying to understand why it is out of range but so far I haven't find it out. $\endgroup$ – Steven Jul 27 '20 at 20:31
  • $\begingroup$ @Steven, please print(column_headers) to make sure they are extracted correctcly or are indeed there and let me know. Possibly you have to adjust the code the correctly extract the table headers if you know the table structure and have some understandfing of html and soup $\endgroup$ – Nikos M. Jul 27 '20 at 20:34
  • $\begingroup$ I printed column_headers but it returned the same error. I uploaded the html code in the question. I wanted to extract the company names, eg Karify, and possibly the corresponded url. I don't know why the range is related to the tag I want to extract. $\endgroup$ – Steven Jul 27 '20 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.