Questions tagged [web-scraping]
The web-scraping tag has no usage guidance.
22
questions
0
votes
1answer
22 views
Suggestions for studying *Clickstream* data
I've essentially been handed a dataset of website access history and I'm trying to draw some conclusions from it.
The data supplied gives me the web URL, the datetime for when it was accessed, an the ...
0
votes
0answers
23 views
How to get IAB category for 1M websites
I am writing a publication about websites privacy. Quite a standard for such works [e.g., 1, 2] is to summarize categories of websites in the study, to either prove that the study is not biased, or ...
0
votes
0answers
15 views
What are the different ways to feature engineer webpage data for input into a webpage classification model?
Looking for resources on the different ways that one can manipulate webpage data to input as features into a neural net.
I'm aware of a service called diffbot that claims to use a CV based method to &...
0
votes
1answer
134 views
Best way to store scraped web pages?
I'm going to scrape the HTML code from a large number of ULRs and store them on my computer, for machine learning purposes (basically, I'm going to use Python and PyTorch to train a neural network on ...
0
votes
0answers
9 views
Link Recommendation System
I am trying to build a Recommendation system on my website for recommending similar articles to the user. Eg: Lets say a user is reading an article about sports on a news website. The next article ...
1
vote
1answer
47 views
Beautifulsoup iterating through scraped data
I have this html code that is repeating multiple times:
...
0
votes
2answers
536 views
Beautifulsoup find_all returns TypeError: list indices must be integers or slices, not str [closed]
This should have an easy solution but I can't understand how to avoid it.
...
0
votes
1answer
407 views
1
vote
0answers
49 views
Web scraping using Beatiful Soup
I have this code and I wanna extract holidays, petrol and temperature but I don't know where is the problem. I need your help as soon as possible, please.
I want to add this extraction to my dataset ...
2
votes
2answers
100 views
looking for public dataset for stock market
I want to do some modelling and data visualization on historical stock data, including price, volume, financials, etc. Is there an public dataset available for stock price history? I looked at a few, ...
1
vote
1answer
35 views
Web Scraping: Multiple small files or one large file?
I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each ...
1
vote
0answers
30 views
Good database/plug-in to scrape for academic paper info?
I am trying to scrape the web to find information about a set of academic papers.
Unfortunately, for many of the papers, I only have the author's name, a year, and part of a title.
(For example, [...
2
votes
1answer
35 views
getting error while scrapping Amazon using Selenium and bs4
I'm working on a class project using BeautifulSoup and webdriver to scrap
Disposable Diapers on
amazon for the name of the item, price, reviews, rating.
My goal is to have something like this where ...
1
vote
1answer
65 views
Use Python to Scape Data on website ends with aspx
https://blue.kingcounty.com/Assessor/eRealProperty/default.aspx
This is the website I want to scrape.
For example, I input 500 4TH AVE
I just want to get Parcel Number and Lot size.
I was trying ...
1
vote
0answers
20 views
Cyber crime - data set
I'm doing science project on my university to make an app, which uses AI to detect cyber crimes. I'm looking for sites to make my data set. Do you know any with ads like tobacco, alcohol, ...
1
vote
1answer
26 views
Scraping mixed elements and passing to SQL
I'm running webscrapes via python which are retrieving data from csv's hosted on the web. I'd like to pass the data into a MSSQL database. An issue I have is the mixed elements/data types in the csv....
0
votes
1answer
67 views
Can I download Twitter data via web scraping for research?
I want to do a sentiment analysis using twitter data. Was thinking about hardcoding a cURL script to download data, from a Google Cloud service (I'll run the data on a neural network on the server, to ...
1
vote
1answer
21 views
Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)
I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/ . I would like to have the values described in ...
-1
votes
1answer
16 views
How would one get the HTML structure of a web page as a numeric vector? [closed]
Suppose you want to cluster, or classify, web pages inside a domain.
In the same domain similar web pages always have the same structure more or less. (think of every product page an e-commerce would ...
1
vote
0answers
95 views
Bootstrap Template for a Dash App
Where Can I find a Bootstrap Template for a Dash app?
Are there implemented examples? I cannot find nothing about this.
I am building a Dashboard with Dash/Plotly but I want to improve the style of ...
5
votes
1answer
230 views
Scraping financial web data
I recently started working as a data scientist and I am starting a web scraping and NLP project using Python. The idea is to create a program that searches for public information on the company's ...
1
vote
0answers
24 views
Web scrapping with beautifulSoup is done slowly
I have developed a web scrapping code in Python which takes data from Hattrick.org's matches and returns them in a table so it can be mined, determined likelihood of goals, etc.
I have the difficult ...
