Questions tagged [web-scraping]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

Need help in Webscraping T20 World Cup 2022 data

I have a list of links of match scorecards and i have stored the respective html content of each link in a list. I need to get the csv files I am attaching as the output. Any help would be ...
karan shah's user avatar
1 vote
1 answer
51 views

What is the NLP problem I am solving called and how should i go about solving it?

I am working on a POC where I am required to write a NLP code after web scraping. The prompt to my code is How good is the online Data Science degree offered by MIT? I am required to do web ...
The White Cloud's user avatar
0 votes
0 answers
218 views

How to extract data from a hidden element in HTML mouseover

I need some data for my thesis but the original source reffused to divulge it due to some classified data being mixed in the original dataset. The non-classified info is available on this link, and ...
Igor Salerno Filgueiras's user avatar
0 votes
0 answers
243 views

How to scrape stock moment data from Online web Trading Platform

I recently started a Data Analysis project that involves data relating to stock market prices (i.e currency pair information). An example is AUD/CAD trade. I wish to scrape 1-minute price details from ...
Harrylever's user avatar
1 vote
1 answer
30 views

Detect data (web textual content) age

This is a broad question and maybe does not have an answer but I will try. I have been thinking of some techniques to detect the date of publication of public data in the wild of the internet. Without ...
bacloud14's user avatar
  • 453
0 votes
0 answers
134 views

Data Scraping Government Site

As I know it may be okay or may not be okay to data scrape a government website. Would the following site be okay to data scrape? https://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist I ...
LJ90's user avatar
  • 1
0 votes
1 answer
18 views

Extracting Public Page Posts from Facebook

I am a data science student working on my capstone research project. I am trying to collect posts from a number of public organization pages from facebook. I am looking at the graphs api and it does ...
Reed's user avatar
  • 101
2 votes
0 answers
82 views

Is there a way to scrape tweets in realtime from a list of specified users? [closed]

I am trying to build a scraper that will run continuously and save the tweets from a list of users instantaneously or within seconds of the user tweeting it. It could save the tweet details to a ...
niusoski's user avatar
0 votes
0 answers
332 views

how to scrape images from facebook group

I want to scrape images from several Facebook groups some of them are public and some or not I am new to web scraping but I tried to look for solutions with selenium or BeautifulSoup or scrappy but I ...
Mohamed Amine's user avatar
1 vote
0 answers
79 views

How to Scrape Website Usage/User Data for a Website that Does Not Have an API [closed]

I am an aspiring Data Analyst and I was wondering how do I web scrape website usage/user data such as: total people who visited that website, total visitors per page, how long did each visitor stay ...
Nat's user avatar
  • 11
1 vote
2 answers
121 views

Data extraction using crawlers

I have a rather simple data scraping task, but my knowledge of web scraping is limited. I have a excel file containing the names of 500 cities in a column, and I'd like to find their distance from a ...
Jay's user avatar
  • 13
0 votes
1 answer
111 views

Suggestions for studying *Clickstream* data

I've essentially been handed a dataset of website access history and I'm trying to draw some conclusions from it. The data supplied gives me the web URL, the datetime for when it was accessed, an the ...
user1147964's user avatar
0 votes
1 answer
608 views

Best way to store scraped web pages?

I'm going to scrape the HTML code from a large number of ULRs and store them on my computer, for machine learning purposes (basically, I'm going to use Python and PyTorch to train a neural network on ...
HelloGoodbye's user avatar
1 vote
1 answer
304 views

Beautifulsoup iterating through scraped data

I have this html code that is repeating multiple times: ...
Steven's user avatar
  • 119
0 votes
2 answers
1k views

Beautifulsoup find_all returns TypeError: list indices must be integers or slices, not str [closed]

This should have an easy solution but I can't understand how to avoid it. ...
Steven's user avatar
  • 119
0 votes
1 answer
1k views

Python writing to Excel file: writerow() takes no keyword arguments

I have this script: ...
Steven's user avatar
  • 119
1 vote
0 answers
87 views

Web scraping using Beatiful Soup

I have this code and I wanna extract holidays, petrol and temperature but I don't know where is the problem. I need your help as soon as possible, please. I want to add this extraction to my dataset ...
IKRAM EL MBARKI's user avatar
2 votes
2 answers
159 views

looking for public dataset for stock market

I want to do some modelling and data visualization on historical stock data, including price, volume, financials, etc. Is there an public dataset available for stock price history? I looked at a few, ...
Donald S's user avatar
  • 1,899
1 vote
1 answer
255 views

Web Scraping: Multiple small files or one large file?

I plan to scrape some forums (Reddit, 4chan) for a research project. We will scrape the newest posts, every 10 minutes for around 3 months. I am wondering how best to store the JSON data from each ...
thomasjeeverson's user avatar
1 vote
0 answers
37 views

Good database/plug-in to scrape for academic paper info?

I am trying to scrape the web to find information about a set of academic papers. Unfortunately, for many of the papers, I only have the author's name, a year, and part of a title. (For example, [...
Mox's user avatar
  • 19
2 votes
1 answer
190 views

getting error while scrapping Amazon using Selenium and bs4

I'm working on a class project using BeautifulSoup and webdriver to scrap Disposable Diapers on amazon for the name of the item, price, reviews, rating. My goal is to have something like this where ...
cesco's user avatar
  • 29
0 votes
1 answer
156 views

Use Python to Scape Data on website ends with aspx

https://blue.kingcounty.com/Assessor/eRealProperty/default.aspx This is the website I want to scrape. For example, I input 500 4TH AVE I just want to get Parcel Number and Lot size. I was trying ...
Mr. Z's user avatar
  • 141
1 vote
0 answers
29 views

Cyber crime - data set

I'm doing science project on my university to make an app, which uses AI to detect cyber crimes. I'm looking for sites to make my data set. Do you know any with ads like tobacco, alcohol, ...
Piotr Rarus's user avatar
1 vote
1 answer
34 views

Scraping mixed elements and passing to SQL

I'm running webscrapes via python which are retrieving data from csv's hosted on the web. I'd like to pass the data into a MSSQL database. An issue I have is the mixed elements/data types in the csv....
thesimplevoodoo's user avatar
0 votes
1 answer
109 views

Can I download Twitter data via web scraping for research?

I want to do a sentiment analysis using twitter data. Was thinking about hardcoding a cURL script to download data, from a Google Cloud service (I'll run the data on a neural network on the server, to ...
Delta's user avatar
  • 103
1 vote
1 answer
24 views

Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)

I am trying to do a correlation study about personality traits as described in Hofstede's :https://www.hofstede-insights.com/product/compare-countries/ . I would like to have the values described in ...
MSIS's user avatar
  • 113
-1 votes
1 answer
27 views

How would one get the HTML structure of a web page as a numeric vector? [closed]

Suppose you want to cluster, or classify, web pages inside a domain. In the same domain similar web pages always have the same structure more or less. (think of every product page an e-commerce would ...
Nicolò Gasparini's user avatar
1 vote
0 answers
115 views

Bootstrap Template for a Dash App

Where Can I find a Bootstrap Template for a Dash app? Are there implemented examples? I cannot find nothing about this. I am building a Dashboard with Dash/Plotly but I want to improve the style of ...
Laura's user avatar
  • 161
5 votes
1 answer
326 views

Scraping financial web data

I recently started working as a data scientist and I am starting a web scraping and NLP project using Python. The idea is to create a program that searches for public information on the company's ...
maxime.b28's user avatar
1 vote
0 answers
69 views

Web scrapping with beautifulSoup is done slowly

I have developed a web scrapping code in Python which takes data from Hattrick.org's matches and returns them in a table so it can be mined, determined likelihood of goals, etc. I have the difficult ...
Juan Esteban de la Calle's user avatar