Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
611
questions
0
votes
0answers
3 views
How to calculate an average “will not close” date on opportunity data?
Please let me know if this question belongs elsewhere.
In my simple data set focused on sales pursuit opportunities, I have the following columns available.
Pursuit Name
Status(Open, Won, Lost)
Date ...
-2
votes
0answers
13 views
Converting aggregates to raw data [closed]
I'm looking for a way to regenerate an artifical society if i know alot about the realy socity parameters: totoal population, gender distribtuion, age distribtuion...etc.
Can anyone help me from where ...
0
votes
1answer
24 views
why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies
Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only ...
0
votes
0answers
17 views
Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`
I want to train a Sequential Neural Net (NN) with Tensorflow.
...
0
votes
0answers
8 views
Why do executable files make weird sounds when converted to audio? [closed]
So, I got into file data, and I downloaded an audio editing software called "Audacity". I realized that you can convert raw data from executable files into audio. I wondered what Audacity ...
0
votes
2answers
26 views
How match output (pred value) to input value
I'm working with data(with 4 columns which are p(product), M(name of the store)), I want predict the demand of store for that I sued SVR on the data by theses formulation:
...
-1
votes
0answers
11 views
What is the p-value of a binomial distribution? [closed]
I understand that p-value is the strength of evidence in support of a null hypothesis.
However, I do not understand what would be different for a binomial distribution?
0
votes
0answers
19 views
R: saving/printing all intermediate results
currently, we have developed a model at work and it is at the verification step. the goal is for another person to receive output from each and every step of calculation to verify.
The issue is that ...
-2
votes
3answers
45 views
How to decrease $R^2$ value and change it to positive value [closed]
I'm working on a data, and use regression , as you see bellow:
from sklearn.svm import SVR
regressor = SVR(kernel = 'linear')
regressor.fit(trainX,trainY)
above ...
0
votes
1answer
32 views
How to use inverse_transform in MinMaxScaler for pred answer in a matrix
I am working on a data, for preding output, I used SVR by bellow code:
...
0
votes
0answers
15 views
How to find the relationship between one dataset and two others?
From a STEM problem, I vary a variable x within a range and calculate quantities $U(x)$, $V(x)$ and $W(x)$. I want to figure out an analytical relationship between unknown $U(x)$ and the other two ...
-1
votes
0answers
12 views
Val_accuracy not showing in tensorflow
I am implementing a TensorFlow model for image classification. while training the model I am not able to see val_accuracy and val_loss, I have attached a screenshot
0
votes
1answer
15 views
How to Impute Top-Censored Data
What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?
1
vote
0answers
9 views
Is there enough datapoints to make a reasonable predcitction?
I am planning to use this dataset (for a school project) in order to determine the most important features in predicting whether a student will receive a placement.
Further I would like to create a ...
1
vote
2answers
31 views
Is it possible to detect drift with real time predictions?
I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or ...
1
vote
0answers
10 views
Comments vs. Files (& how to import w/o packages) for Data Science [closed]
I am trying to settle on an efficient & pythonic workflow for developing scripts in data science work that are scalable and don't turn to spaghetti once things get complicated and parameters are ...
-2
votes
1answer
14 views
What data to collect to create useful insights?
I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
-1
votes
0answers
7 views
How to solve this error 'Input contains NaN, infinity or a value too large for dtype('float64').'
I ran this code:-
imp = SimpleImputer(missing_values='NaN', strategy='mean')
X=imp.fit_transform(features)
X
0
votes
1answer
24 views
When would C become nescessary to do an analysis or manage data?
I use python in my day to day work as a research scientist and I am interested in learning C. When would a situation arise where python would prove insufficient to manipulate data?
2
votes
1answer
27 views
Do i need to handle missing values before EDA?
I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
0
votes
1answer
34 views
Data Cleaning Techniques - What to drop?
I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, ...
0
votes
0answers
27 views
Data calculation/normalisation /data summarisation
Clustering is done based on houses data and say suppose cluster 1 contains 5 houses.
Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
2
votes
0answers
18 views
What is the difference between Pachyderm and Git?
I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:
It holds all your data in a central accessible ...
2
votes
1answer
30 views
How would you build a big production ready image training dataset from scratch?
How would you most likely create a large production ready image training dataset from scratch including annotations for a image classification task?
We will take a large amount of images (~1 million) ...
0
votes
1answer
23 views
Clusterize item set with items as vectors of features
I have to clusterize this dataset in which I have houses and water consumption in this form:
$$
House1 = (x_{1},x_{2}... x_{n});\\
House2 = (y_{1},y_{2}... y_{n});\\
House3 = (z_{1},z_{2}... z_{n});\\
...
0
votes
0answers
31 views
What exactly negative/positive value of Captum's Integrated Gradient mean?
I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ...
Positive attribution score means that the input in that ...
0
votes
1answer
18 views
Microsoft custom vision vs Tensorflow model?
I am planning to implement my own image classifier model using TensorFlow instead of a custom vision platform.
what is the biggest difference between custom vision(https://www.customvision.ai/) vs ...
0
votes
0answers
8 views
What makes a high quality image dataset?
I am constructing a custom dataset for image classification. It is a simple task in the sense that only one class needs to be detected. However it is crucial that the dataset is high quality, and I am ...
2
votes
2answers
23 views
How to determine if dataset is a suitable representation of the context?
How can I determine if the data I have collected is a good enough representation of the context? For example, I am working on an object detection system and have been building an image dataset. How ...
1
vote
3answers
22 views
Deep Learning model for classifying skin diseases
I have planned to create a deep learning model that classifies skin diseases(around 5 to 7 diseases). Please suggest me a good deep learning model to go with. I am planning to integrate this model as ...
0
votes
0answers
12 views
Customer Segmentation - Can I use an ML model for this problem?
I have a seller segmentation problem where I need to segment the top and bottom N customers based on three variables: sales, Average selling price and product category. However, the thresholds for the ...
3
votes
0answers
41 views
SVD Data Requirements [closed]
I'm very very new to data-science, so go easy on me.
I want to incorporate an SVD based game recommendation model (using the Surprise Python Lib) for a full-stack project I'm working on, but before I ...
1
vote
0answers
20 views
how to know if there is a bias in data collection methods
I am collecting data for machine learning models I want to build for some application.
I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
0
votes
1answer
34 views
Does shuffling the training data matter in a K Nearest Neighbors Classifier model?
I am new to machine learning and I have a couple of questions about a project.
So, I created a classifier using the MNIST data set for a ML project that I was working on. I augmented the data by ...
0
votes
0answers
12 views
What is the azure solution for storing and querying Netcdf data on the fly?
We currently run a live web application which utilises standard azure mounted file storage to store 2tb worth of metocean data in netcdf format. On here we run python queries directly on the data e.g. ...
1
vote
1answer
30 views
What are the soft skills required in data science? [closed]
Most discussions center around hard skills, e.g., coding, learning specific languages. I'm curious what are the soft skills one needs to thrive in data science.
Veterans, what would you tell the you ...
0
votes
1answer
24 views
Methods of disaggregating data to smaller units?
I have a relatively straightforward question that I know poses some difficult challenges.
Let's say I have a state-level rate of X. I would like to disaggregate the state-level rate to the county-...
-1
votes
1answer
26 views
Why isn't all data periodic? [closed]
Let's say we have 2 classes (-1 and 1), 1 feature (x), and an arbitrary amount of data points.
Why can't we always find a frequency and phase that fit a sine wave or a square wave to our data points? ...
0
votes
0answers
16 views
How to do parameter estimation on relatively few samples?
Context:
I have a time series on which I've trained an autoencoder for anomaly detection (AD) purposes.
My time-series is 6 years worth of daily data, but for AD purposes, I am looking at one week ...
0
votes
1answer
25 views
How to split data into training, validation, test data sets if the data is non-stationary?
When splitting data into training, validation, test data sets to be fed to machine learning model, the data is ideally expected to be stationary. However, in the real world, some data is non-...
2
votes
1answer
17 views
Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time
First time plotting and interpreting time series data and I have used a line plot for ease of use. I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., ...
28
votes
8answers
5k views
Why is it wrong to train and test a model on the same dataset?
What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?
0
votes
0answers
21 views
Is there a data structure that works similarly to a (binary) tree?
Im currently working with some data, which is part of a very large data set (height, about 6000 elements per timestep - timestep every minute over days at a time). I need to differentiate between ...
0
votes
1answer
28 views
Similarity between two time series with different sampling frequency, different amplitude, and different lengths but taken from the same source?
I have two files with accelerator readings and I want to get some metric/ measurement to get the similarity between these two files. I have tried Pearsonās R coefficient, dtw distance, dtw score. ...
0
votes
1answer
25 views
How can I generate reasonable dummy/artificial data from a pre-existing time-series data?
I have a dataset like this, basically all numerical time-series data.
I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for ...
0
votes
0answers
35 views
Normal vs Uniform Distribution for machine learning
I have a dataset that follows Zipf's law such that the majority of the values are concentrated at one end, with the remaining items containing a very small percentage. Training on the dataset as is ...
-1
votes
1answer
22 views
Similar values cleaning [closed]
can someone know algorithm how to identify account names that are similar enough to be potentially merged and imported as one
Duplicates with different values:
Geico val1 NaN =====>>...
0
votes
0answers
16 views
Ideas on handling large quantities of “waste” data
Hello and thank you for taking time out of your day to help me. I am currently working on developing a machine vision application for production monitoring. The application handles images, about 20,...
-1
votes
1answer
33 views
Can we identify that an academic dataset was used for commercial purpose [closed]
There are many datasets released on the internet. Authors of many of these datasets state that the datasets are strictly for academic usage and not for commercial purposes. Although some datasets are ...
0
votes
1answer
38 views
How to input sets as features
Need advice on the best way to represent the below data to be fed into an ML algorithm (yet to decided on)
This is from the online order processing domain. An order consists of a set of variable ...
