Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
460
questions
0
votes
2answers
936 views
Orange3 summarizing data, grouping data values
Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values?
For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
0answers
8 views
17
votes
1answer
26k views
How does the validation_split parameter of Keras' fit function work?
Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ :
validation_split: Float between 0 and 1. Fraction of the training data
...
0
votes
2answers
36 views
Suggestion of a model for these type of data?
I've got a data set that looks like this
...
2
votes
1answer
47 views
Balance data using different criteria
I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID.
For each ID, I have a ...
0
votes
1answer
241 views
Equivalence of Tidy Data and Third Normal Form
In Hadley Wickham's "Tidy Data" paper, he states that
In tidy data:
Each variable forms a column.
Each observation forms a row.
Each type of observational unit forms a table.
...
0
votes
0answers
12 views
How can I compare continuous and non-binary categorical data in Excel?
I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
0
votes
2answers
20 views
How to apply data binning on reviews data? [closed]
I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
21 views
Generating random numbers from best probability distributions?
I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test):
...
0
votes
0answers
7 views
Scraping from non-identifiable table, please help! [closed]
I'm scraping all the international basketball player data from realgm.com. An example of a page I'm scraping is https://basketball.realgm.com/player/Dominique-Jones/Summary/6412. I'm doing this for ...
0
votes
2answers
32 views
Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?
I have several marketing files that I need to download on a weekly basis.
Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
1
vote
2answers
24 views
Cross validation vs leave one out
I have found the following definitions, but I don't really see the difference.
cross validation
Method for testing classification and prediction models. The data are randomly split into N partitions (...
0
votes
1answer
62 views
pd.qcut bins error!
Hi Data Science community!
I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error.
...
1
vote
0answers
28 views
How to avoid bias when optimizing time-series model?
I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
0
votes
2answers
36 views
How to handle “year” variable for Machine Learning models
I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable?
Thanks ...
2
votes
2answers
151 views
Transform single-label data set into multi-label data set
I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
1
vote
1answer
27 views
Extract data from PDFs
I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
1answer
58 views
PCA in visual Analytics
I m studying visual analytics and i have a theoretical question about this topic.
My professor introduced this schema in him slide
For connect data to visualisation.
Some topic is very easy to ...
1
vote
2answers
35 views
Clearest way to visualise temporal data?
I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
0
votes
0answers
11 views
Anonymizing the CSV data using Python Faker
To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations.
Is it possible to anonymize the ...
0
votes
0answers
13 views
Excel date-time auto fill error [migrated]
I have data of 10 metrics for a 24 hour period which is in 5 minute intervals. I merged all the 10 data metrics into one file and converted it into a .xlsx file.
After opening the excel file I ...
0
votes
1answer
25 views
Stata: Trying to merge two data sets with common year observations
I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
0
votes
0answers
14 views
Difference between Big Data and a Data Warehouse?
I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
2
votes
1answer
446 views
How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library
Sorry for a long story, but it is a long story. :)
I am using h2o library for python to build a decision tree and to extract a decision rules out of it.
I use some data for training where labels get ...
0
votes
1answer
56 views
R - How do I remove a varying number of digits from a date-vector
I want to remove a varying number of digits from a date vector. My date vectors looks like this:
I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
2answers
32 views
Oversampling/Undersampling only train set only or both train and validation set
I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
21 views
Transforming Data for Classification
I have a dataset of the type:
Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views
Space and time fourier transform of data
I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
17 views
Free API for historical weather US data?
I am trying to retrieve a free R-Python API that provides historical weather data in US.
In fact wunderground API is no longer available.
Any suggestion?
11
votes
2answers
8k views
How much data are sufficient to train my machine learning model?
I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.
My colleague (who is a machine ...
2
votes
1answer
47 views
Analysis of sales performance
This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question.
I work for a business, we sell certain products.
We are running a trial currently on ...
1
vote
1answer
33 views
how to match a sentence to a cluster of keywords?
I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below.
<...
6
votes
2answers
168 views
Data anonymization in Python
I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
6
votes
4answers
393 views
What are some of the best practices for sharing data and models with colleagues?
As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
0
votes
0answers
11 views
Free historical news data base?
I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
32 views
How to sort of “average” data without smoothing?
I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time.
The problem ...
1
vote
1answer
294 views
How to determine the weights of features when calculating the weighted average?
I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
2
votes
4answers
295 views
Generate timeseries data
Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
0
votes
1answer
38 views
Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?
I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
1
vote
1answer
116 views
Python equivalent of Wolfram Language Query for JSON?
The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
2
votes
1answer
39 views
When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?
I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
1
vote
1answer
18 views
Is there a way to aggregate frequency distribution for many runs?
In the following plot
I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
0
votes
1answer
38 views
SQL and JSON database - R
I have a database (PostgresSQL) which looks like this:
""results"":
{""WEBS"":
{""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}}
I would ...
1
vote
2answers
64 views
Sentiment Analysis Datasets
I am looking for sentiment analysis data, mostly customer product review.
I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date ...
0
votes
1answer
33 views
Searching prediction from 4 datasets
The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
0
votes
0answers
40 views
Connecting to IEX with Pandas Datareader
My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
1
vote
1answer
34 views
Why is correlation between my independent variables helping my linear regression model?
I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three ...
1
vote
1answer
15 views
Best file format for transfer of EHR data
I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
1
vote
1answer
42 views
How to create a system to detect text structure of a file?
Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
1answer
51 views
How to predict consumer purchase in next 6 months?
I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous ...
