Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
2answers
936 views

Orange3 summarizing data, grouping data values

Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values? For example, instead of plotting a scatter with lots of data points, I would like to ...
1
vote
0answers
8 views
17
votes
1answer
26k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
0
votes
2answers
36 views
2
votes
1answer
47 views

Balance data using different criteria

I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID. For each ID, I have a ...
0
votes
1answer
241 views

Equivalence of Tidy Data and Third Normal Form

In Hadley Wickham's "Tidy Data" paper, he states that In tidy data: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. ...
0
votes
0answers
12 views

How can I compare continuous and non-binary categorical data in Excel?

I'm trying to find the correlation between vehicle days on lot vs different categorical variables (color, make, model, etc). I know it's possible with binary data, but I haven't found a simple way to ...
0
votes
2answers
20 views

How to apply data binning on reviews data? [closed]

I need to apply data binning on a set of reviews, I have searched for some data binning methods for reviews and long-texts and couldn't find anything other than classification. Is NLP or ...
2
votes
1answer
21 views

Generating random numbers from best probability distributions?

I have done some statistical distributions on real-world data. The distribution fitting gives me the below results (lognormal fits the best on data based on chi-squared test): ...
0
votes
0answers
7 views

Scraping from non-identifiable table, please help! [closed]

I'm scraping all the international basketball player data from realgm.com. An example of a page I'm scraping is https://basketball.realgm.com/player/Dominique-Jones/Summary/6412. I'm doing this for ...
0
votes
2answers
32 views

Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

I have several marketing files that I need to download on a weekly basis. Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
1
vote
2answers
24 views

Cross validation vs leave one out

I have found the following definitions, but I don't really see the difference. cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (...
0
votes
1answer
62 views

pd.qcut bins error!

Hi Data Science community! I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error. ...
1
vote
0answers
28 views

How to avoid bias when optimizing time-series model?

I have 4000 days of data. I am trying create a time-series model with parameters P to forecast the value of the target Y using the last N days of data. The parameters P include: lookback window for ...
0
votes
2answers
36 views

How to handle “year” variable for Machine Learning models

I have a "year" variable but I don't know which is the best way to handle it for a ML model, as it is a numerical variable, giving some sequence. Should I treat it as a categorical variable? Thanks ...
2
votes
2answers
151 views

Transform single-label data set into multi-label data set

I received a data set containing a string of text and a label that categorizes that text into one of 50 categories. I'm hoping to build a model that predicts which category a string of text belongs in....
1
vote
1answer
27 views

Extract data from PDFs

I would like to do an experiment. I would like to get the following data: advisor education rank (Ing., Bc. etc), number of pages in the thesis, number of citations etc for each student about thesis ...
0
votes
1answer
58 views

PCA in visual Analytics

I m studying visual analytics and i have a theoretical question about this topic. My professor introduced this schema in him slide For connect data to visualisation. Some topic is very easy to ...
1
vote
2answers
35 views

Clearest way to visualise temporal data?

I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.)....
0
votes
0answers
11 views

Anonymizing the CSV data using Python Faker

To protect the sensitive customer data, I came across a useful python project faker. However, this project only generates the fake values after random iterations. Is it possible to anonymize the ...
0
votes
0answers
13 views

Excel date-time auto fill error [migrated]

I have data of 10 metrics for a 24 hour period which is in 5 minute intervals. I merged all the 10 data metrics into one file and converted it into a .xlsx file. After opening the excel file I ...
0
votes
1answer
25 views

Stata: Trying to merge two data sets with common year observations

I am currently trying to merge two datasets with a common "year" variable using Stata. One dataset is purely for incorporating CPI with an associated year, and so the only variables are "year" and "...
0
votes
0answers
14 views

Difference between Big Data and a Data Warehouse?

I would like to know what is the exact difference between Big Data and a Data Warehouse. I know that Big Data is the technology to receive and work with really large amounts of data and a data ...
2
votes
1answer
446 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
0
votes
1answer
56 views

R - How do I remove a varying number of digits from a date-vector

I want to remove a varying number of digits from a date vector. My date vectors looks like this: I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
0
votes
2answers
32 views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
0
votes
0answers
21 views

Transforming Data for Classification

I have a dataset of the type: Time_from_anchor is the time from when the diagnosis was made, if he had cancer. i.e patient 1 was diagnosed with cancer at 0. He had tests x and y at time -1 and tests ...
0
votes
0answers
12 views

Space and time fourier transform of data

I have discrete data for function $H(x,y,t)$ in the interval $0<x<x_0$, $0<y<y_0$ and $0<t<t_0$. I'm wondering if there is a way to find the function in Fourier space $H_f(q_x,q_y,w)$...
0
votes
1answer
17 views

Free API for historical weather US data?

I am trying to retrieve a free R-Python API that provides historical weather data in US. In fact wunderground API is no longer available. Any suggestion?
11
votes
2answers
8k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
2
votes
1answer
47 views

Analysis of sales performance

This is my first question on this stackexchange - i hope i'm in the right place and i'm asking the right question. I work for a business, we sell certain products. We are running a trial currently on ...
1
vote
1answer
33 views

how to match a sentence to a cluster of keywords?

I have a classification problem. I have clusters called 'Experience', 'Education', 'Abilities' . The labelled data (72,000+ entries with all clusters together) with two columns looks like below. <...
6
votes
2answers
168 views

Data anonymization in Python

I am working on an industrial project which consists of real data. Now, the data contains sensitive information about company operations which could not be disclosed publically. As a result, I need to ...
6
votes
4answers
393 views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
0
votes
0answers
11 views

Free historical news data base?

I spent some time looking for news databases and APIs. I can find a lot of paid ones. The data google provides when searching for news is perfect for what I am after (something with a deep history, ...
0
votes
0answers
32 views

How to sort of “average” data without smoothing?

I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time. The problem ...
1
vote
1answer
294 views

How to determine the weights of features when calculating the weighted average?

I have a dataset of Tweets along with their Retweet Counts and Favorite Counts. After extracting features from the Tweets, I am training a model to predict the Retweet Counts and Favorite Counts. I ...
2
votes
4answers
295 views

Generate timeseries data

Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
0
votes
1answer
38 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
1
vote
1answer
116 views

Python equivalent of Wolfram Language Query for JSON?

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
2
votes
1answer
39 views

When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

I have been through various kernels where scaling is done on y_train and y_test and many where there isn't. Is there any specific rule which should be followed when to or when not to do this?
1
vote
1answer
18 views

Is there a way to aggregate frequency distribution for many runs?

In the following plot I have frequency distribution for each of the fitness values that are shown in thte legend. Each run (x-axis) has a different frequency of each fitness value. The total runs is ...
0
votes
1answer
38 views

SQL and JSON database - R

I have a database (PostgresSQL) which looks like this: ""results"": {""WEBS"": {""WEBS_4"": [{""ip"": ""XXXXX"", ""value"": ""YYYY""}]}} I would ...
1
vote
2answers
64 views

Sentiment Analysis Datasets

I am looking for sentiment analysis data, mostly customer product review. I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date ...
0
votes
1answer
33 views

Searching prediction from 4 datasets

The fourth dataset contains (train_data, test_data, previous_data, and information_history_data). The goal is to search for a user's rating on the loan to the bank. ...
0
votes
0answers
40 views

Connecting to IEX with Pandas Datareader

My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
1
vote
1answer
34 views

Why is correlation between my independent variables helping my linear regression model?

I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three ...
1
vote
1answer
15 views

Best file format for transfer of EHR data

I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
1
vote
1answer
42 views

How to create a system to detect text structure of a file?

Let's say I want to create a Machine Learning system that has a lot of log files of some few types (F1, F2,.. Fn) and I get a new Log file with maybe some errors or ...
0
votes
1answer
51 views

How to predict consumer purchase in next 6 months?

I'm working on a model to predict a customer as being 'in-market' for a product in the next 6 months. The dataset has a wealth of information like lifestyle and demographic variables and previous ...