Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
479
questions
2
votes
2answers
260 views
Ways to simulate weather data over several periods (Python or R)?
I have a time series dataset that has several variables for a state/province for fixed periods of time. That is for state A, there are samples from April 2017 to July 2019. Of course, I thought adding ...
1
vote
0answers
19 views
How to estimate and/or determine how much data is enough to train a model?
To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of ...
2
votes
1answer
18 views
How to build a unbiased predictive ML model when the record of the event is less compared to the total number of records?
I am trying to build a model that will predict the communication loss of a wireless device. For now I am using RandomForestClassifier along with Device and Location as the features. I am getting both ...
2
votes
0answers
29 views
Fixed-Width vs Adaptive Binning
I have some continuous variable in my data that I wish to apply binning for. The values range from 0 to 800 but I got motivated by the fact that the data distribution was left skewed as you could see ...
0
votes
1answer
20 views
difference btw data distribution and frquency distribution
I have a dataset with 'n' features and corresponding labels(binary in nature). How can I calculate the data distribution and frequency distribution of the same? What is the difference btw the two?
2
votes
1answer
37 views
How to separate words that are together in a large data set
in twitter data i came across words that are glued together like 'boycottbears'
i want them as 'boycott' 'bears' 'man'
i tried this but this is slow
...
1
vote
0answers
13 views
SEM (Structural Equation Modelling) with Exploratory Factor Analysis
Problem Statement: I need to do some Structural Equation Modelling at work to get the main factors in a marketing survey data-set.
There are no assumed equations to perform SEM on so what would be ...
5
votes
1answer
60 views
0
votes
1answer
24 views
How can I extract the reason of the legal compensation from a court report?
I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report)
Order mister X to ...
2
votes
1answer
19 views
How do I best visualize this voltage data for a science project
I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
1
vote
0answers
29 views
Dataset for musical Instruments recognition
I am looking for a "rich" Dataset to Teach my model to seperate Music intruments .. any Suggestion ? I found the URMP Dataset but it doesn't have many files.. I appreciate your help
thanks and happy ...
1
vote
2answers
26 views
Should I use MongoDB instead of storing data in CSV in python?
I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
0
votes
0answers
22 views
Data Visualization of Feature Distribution Plot
I have been trying my best to plot this type of feature distribution plot using Python. I looked for what they are called to replicate the plot, but couldn't find the name of it?
The Reference of ...
0
votes
0answers
9 views
ValueError while Comparing with Multilevel Index
My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
0
votes
0answers
9 views
access to the didi ditech2016 dataset
a bit off-topic, but I'm trying to locate a famous dataset used in the DiDi DiTech2016 supply-demand challenge. All sources I found point to a single url:
http://research.xiaojukeji.com/competition/...
1
vote
1answer
37 views
What's the best way to train a NER model?
I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used…
To do that I want to ...
2
votes
1answer
19 views
Are parquet files compressed? [closed]
Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
1
vote
0answers
16 views
PointCloud value prediction - regression
I have point clouds for 400 objects. I have also some other features of these objects. For example their weights.
How I can now train model to predict the weight based on the shape(point cloud)?
I ...
0
votes
1answer
20 views
how to know what my data are uninformative and that machine learning will not work with it?
following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I ...
0
votes
1answer
19 views
should I shift a dataset to use it for Time series regression with RNN/LSTM?
I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1
so my ...
2
votes
2answers
78 views
in which case can I say that the data are bad and I ll achieve nothing using machine learning on it
general Infos about my dataset: I have 40k data points and 5 features.
I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
1
vote
0answers
12 views
How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?
I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine ...
0
votes
0answers
12 views
Is syslog/sysmon data considered 'raw data'
I apologize if this is considered a topic of 'discussion,' but I would hope it can be proven factually. Is sysmon data considered raw data? I understand that the data is formatted on output, but in my ...
0
votes
0answers
12 views
football score data, goal events
I am looking for event level football/soccer data. I want to know when a goal happened (minute) - other events are not important. Do you know where I can get this data. Preferably without much ...
1
vote
1answer
23 views
0
votes
0answers
10 views
Rank impact of individual metrics on overall sales performance
I’m interested in learning about different methods to analyse sales data - at both store level and corporate.
Essentially I have lots of performance metrics (~20) that break down the overall sales ...
0
votes
2answers
23 views
Unsupervised encoding of categorical features
I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.
...
1
vote
0answers
11 views
0
votes
0answers
15 views
How can I compare continuous and non-binary categorical data in Excel?
I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
0
votes
2answers
29 views
How to apply data binning on reviews data? [closed]
I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
22 views
Generating random numbers from best probability distributions?
I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test):
...
1
vote
2answers
27 views
Cross validation vs leave one out
I have found the following definitions, but I don't really see the difference.
cross validation
Method for testing classification and prediction models. The data are randomly split into N partitions (...
1
vote
0answers
31 views
How to avoid bias when optimizing time-series model?
I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
1
vote
3answers
52 views
How to handle “year” variable for Machine Learning models
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?
Thanks ...
2
votes
1answer
32 views
Extract data from PDFs
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
0answers
25 views
Anonymizing the CSV data using Python Faker
To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations.
Is it possible to anonymize the ...
0
votes
0answers
15 views
Difference between Big Data and a Data Warehouse?
I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
1answer
28 views
Stata: Trying to merge two data sets with common year observations
I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
1
vote
2answers
43 views
Clearest way to visualise temporal data?
I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
0
votes
0answers
21 views
Transforming Data for Classification
I have a dataset of the type:
Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views
Space and time fourier transform of data
I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
24 views
Free API for historical weather US data?
I am trying to retrieve a free R-Python API that provides historical weather data in US.
In fact wunderground API is no longer available.
Any suggestion?
2
votes
1answer
49 views
Analysis of sales performance
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.
I work for a business, we sell certain products.
We are running a trial currently on ...
0
votes
0answers
15 views
Free historical news data base?
I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
32 views
How to sort of “average” data without smoothing?
I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time.
The problem ...
2
votes
1answer
43 views
When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?
I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
1
vote
1answer
234 views
pd.qcut bins error!
Hi Data Science community!
I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error.
...
1
vote
1answer
19 views
Is there a way to aggregate frequency distribution for many runs?
In the following plot
I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
1
vote
1answer
20 views
Best file format for transfer of EHR data
I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
0
votes
0answers
82 views
Connecting to IEX with Pandas Datareader
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
