Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
476
questions
1
vote
0answers
9 views
SEM (Structural Equation Modelling) with Exploratory Factor Analysis
Problem Statement: I need to do some Structural Equation Modelling at work to get the main factors in a marketing survey data-set.
There are no assumed equations to perform SEM on so what would be ...
5
votes
1answer
52 views
0
votes
1answer
23 views
How can I extract the reason of the legal compensation from a court report?
I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report)
Order mister X to ...
2
votes
1answer
18 views
How do I best visualize this voltage data for a science project
I'm helping my son with his 7th grade science project. We've had a good deal of fun with our experiments with Solar Arrays and charging 12 volt UPS batteries! But, I am not sure how to interpret the ...
0
votes
0answers
26 views
Dataset for musical Instruments recognition
I am looking for a "rich" Dataset to Teach my model to seperate Music intruments .. any Suggestion ? I found the URMP Dataset but it doesn't have many files.. I appreciate your help
thanks and happy ...
1
vote
2answers
26 views
Should I use MongoDB instead of storing data in CSV in python?
I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB ...
0
votes
0answers
22 views
Data Visualization of Feature Distribution Plot
I have been trying my best to plot this type of feature distribution plot using Python. I looked for what they are called to replicate the plot, but couldn't find the name of it?
The Reference of ...
0
votes
0answers
8 views
ValueError while Comparing with Multilevel Index
My dataframe - ncg_vol contains Multilevel index - 'Year' and 'Month' and single column 'std'. I used .apply() on std column to perform operation based on the number of days in that year. I used the ...
0
votes
0answers
9 views
access to the didi ditech2016 dataset
a bit off-topic, but I'm trying to locate a famous dataset used in the DiDi DiTech2016 supply-demand challenge. All sources I found point to a single url:
http://research.xiaojukeji.com/competition/...
1
vote
1answer
33 views
What's the best way to train a NER model?
I am trying to do a project using NLP. My goal is to process Cyber Threat Intelligence articles like this to extract information such as actor’s name, malwares and tools used…
To do that I want to ...
2
votes
1answer
17 views
Are parquet files compressed? [closed]
Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
1
vote
0answers
15 views
PointCloud value prediction - regression
I have point clouds for 400 objects. I have also some other features of these objects. For example their weights.
How I can now train model to predict the weight based on the shape(point cloud)?
I ...
0
votes
1answer
20 views
how to know what my data are uninformative and that machine learning will not work with it?
following this question, I'm making a data analysis again because I tried to use machine learning algorithms like Random forest to predict a value from certain features but it didn't work for me. I ...
0
votes
1answer
18 views
should I shift a dataset to use it for Time series regression with RNN/LSTM?
I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1
so my ...
2
votes
2answers
77 views
in which case can I say that the data are bad and I ll achieve nothing using machine learning on it
general Infos about my dataset: I have 40k data points and 5 features.
I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
1
vote
0answers
12 views
How to extract data from 3d model to a data table that is easy to read for machine learning algorithm?
I want to do something similar to Point Cloud. What I am NOT trying to do is to recognize objects or reconstruct them. I have a 3d file and I want to somehow read the data to use it in machine ...
0
votes
0answers
12 views
Is syslog/sysmon data considered 'raw data'
I apologize if this is considered a topic of 'discussion,' but I would hope it can be proven factually. Is sysmon data considered raw data? I understand that the data is formatted on output, but in my ...
0
votes
0answers
12 views
football score data, goal events
I am looking for event level football/soccer data. I want to know when a goal happened (minute) - other events are not important. Do you know where I can get this data. Preferably without much ...
1
vote
1answer
23 views
0
votes
0answers
10 views
Rank impact of individual metrics on overall sales performance
I’m interested in learning about different methods to analyse sales data - at both store level and corporate.
Essentially I have lots of performance metrics (~20) that break down the overall sales ...
0
votes
2answers
21 views
Unsupervised encoding of categorical features
I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.
...
1
vote
0answers
11 views
0
votes
0answers
15 views
How can I compare continuous and non-binary categorical data in Excel?
I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
0
votes
2answers
24 views
How to apply data binning on reviews data? [closed]
I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
22 views
Generating random numbers from best probability distributions?
I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test):
...
1
vote
2answers
26 views
Cross validation vs leave one out
I have found the following definitions, but I don't really see the difference.
cross validation
Method for testing classification and prediction models. The data are randomly split into N partitions (...
1
vote
0answers
31 views
How to avoid bias when optimizing time-series model?
I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
1
vote
3answers
52 views
How to handle “year” variable for Machine Learning models
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?
Thanks ...
2
votes
1answer
31 views
Extract data from PDFs
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
0answers
17 views
Anonymizing the CSV data using Python Faker
To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations.
Is it possible to anonymize the ...
0
votes
0answers
15 views
Difference between Big Data and a Data Warehouse?
I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
0
votes
1answer
27 views
Stata: Trying to merge two data sets with common year observations
I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
1
vote
2answers
42 views
Clearest way to visualise temporal data?
I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
0
votes
0answers
21 views
Transforming Data for Classification
I have a dataset of the type:
Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views
Space and time fourier transform of data
I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
20 views
Free API for historical weather US data?
I am trying to retrieve a free R-Python API that provides historical weather data in US.
In fact wunderground API is no longer available.
Any suggestion?
2
votes
1answer
49 views
Analysis of sales performance
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.
I work for a business, we sell certain products.
We are running a trial currently on ...
0
votes
0answers
15 views
Free historical news data base?
I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
32 views
How to sort of “average” data without smoothing?
I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time.
The problem ...
2
votes
1answer
43 views
When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?
I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
1
vote
1answer
199 views
pd.qcut bins error!
Hi Data Science community!
I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error.
...
1
vote
1answer
18 views
Is there a way to aggregate frequency distribution for many runs?
In the following plot
I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
1
vote
1answer
19 views
Best file format for transfer of EHR data
I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
0
votes
0answers
73 views
Connecting to IEX with Pandas Datareader
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
6
votes
2answers
306 views
Data anonymization in Python
I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
0
votes
2answers
65 views
Oversampling/Undersampling only train set only or both train and validation set
I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
10 views
Most reliable source for Literacy data
I'm searching for the most reliable source for granular American literacy population data.
I am conducting an analysis, reporting, and data visualization of American literacy. To do this, I am ...
1
vote
0answers
17 views
How can I measure data anonymity when present summary statistics
I am looking to present users with aggregate statistics of a highly sensitive underlying dataset. See example data below:
...
0
votes
0answers
9 views
Reflections about types of bias in comparative analysis
I am working on an analysis on proposal rejections across two different periods in time and I was wondering what reflections I would have to make about the risk of bias.
The reason for this is ...
0
votes
2answers
146 views
Class Imbalance and Cost-Sensitive Learning XGBoost
I'm fairly new to data science and machine learning and have been trying to read a bit more on methods like boosting for one of the projects I am working on. The investigator on this project is ...
