Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views
0
votes
0answers
22 views

Convert a dataset to a binomial distribution in R [closed]

I have an excel file that has COVID-19 data for Indiana. Which can be found here: https://hub.mph.in.gov/dataset/covid-19-county-statistics. The data has the following columns: ZIP_CD (zipcode) ...
0
votes
1answer
17 views

Why are ordered lists considered one dimensional?

At Uni I am currently taking data visualization as a course. While reading thru the script I stumbled upon two examples for one dimensional data. Example one: Bus stops with their name and order (...
0
votes
0answers
11 views

How to organize data in workflows with user input between steps?

I am building a multistep workflow. User provides input files input.csv, parameters_global.yml, ...
0
votes
0answers
18 views

Data visualization multiple choices multiple condition conditions

Say I have a element that base on a certain condition (1, 2, 3 or 4), it chooses a decision (A,B or C). What diagrams can be used to present this data if I have more than one thousand elements so that:...
0
votes
1answer
20 views

What kind of data analysis problem is this?

This might be a dumb question, but I am an absolute beginner. What kind of data analysis problem is the following? Is it an Exploratory Data Analysis problem/Data Mining problem/data analytics problem?...
1
vote
1answer
24 views

Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed ...
0
votes
0answers
11 views

Manually labeled data quality check methods?

In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation ...
0
votes
0answers
22 views

Load data from SQL for Machine Learning

I have an SQL DB which has millions of records(10 years of data) and wanted to implement some ML Models. Most of the courses/tutorials explains using a local file or downloaded CSV. But in my case I ...
0
votes
0answers
21 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
0
votes
1answer
36 views

Image classification with CNN Python

I'm working on image classification using CNN, my dataset contains more than 50 classes (50 folders) which represent the types of car parts, and in each folder we have vehicle brands, each vehicle ...
1
vote
1answer
34 views

Hi, im currently working for a company that has some inventory control problems

First, I was asked by the manager to make a plot showing produced vs received items, its a multistage process so we are only in charge of one of the steps which is designing, I made the plot comparing ...
1
vote
0answers
6 views

Available data sets on basic income pilot studies? [closed]

I'm interested in exploring data relating to any UBI/basic income pilot studies that have been conducted but have been unable to find any so far? thanks
0
votes
0answers
6 views

Question on What I would use for a Piece of Data

I have 3 categories that we can call A, B, and C. In almost every case we B/A = C. B and A are given in the Dataset and C is what I am find an trying to use elsewhere. The higher number C is the ...
0
votes
1answer
38 views

Which data science framework should I start learning? [closed]

I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of ...
0
votes
1answer
18 views

Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series')

Several terms are used to describe the structure of instances of a dataset. For example, an instance can be 'univariate' if it depends on only one variable or 'multivariate' if it depends on 2 or more ...
1
vote
1answer
51 views

Why does an imbalanced data set badly effect distance measures like Mahalanobis?

I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
0
votes
1answer
29 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
0
votes
0answers
8 views

How to Combine Oracle and Snowflake/Redshift data on the fly

Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift. Most ...
1
vote
1answer
19 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
0
votes
0answers
22 views

Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?

I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
0
votes
1answer
18 views

Problem with binning

I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements. My questions are: Which type of ...
0
votes
0answers
17 views

Code in R, and Normally Distrubuted

...
1
vote
0answers
17 views

MCMC algorithm — understanding some paremeters

I am trying to understand an MCMC program. I manage to run it, but I am trying to understand the meaning of the some parameters in the analysis. The code is something like this ...
0
votes
0answers
18 views

Download weather data from a http directory

This link contains weather data with a lot of folders and subfolders. I need to download as a whole without visiting every file inside the folders. https://www.ncei.noaa.gov/data/global-forecast-...
0
votes
0answers
15 views

LSTM Train accuracy decreases in training time during the epoch

I'm training an LSTM model for sentiment analysis on a text corpus. There's a thing that I believe is not normal because I never have seen it in training my models. At the start of the epoch, the ...
1
vote
1answer
13 views

Structuring extensive medical histories + demographic information for prediciting future medical outcomes

I'm looking for advice structuring extensive medical histories for predicting future outcomes, specifically hospital admissions. Let's say I want to predict the whether or not someone will be admitted ...
0
votes
1answer
17 views

How to calculate an average cliff date?

Please let me know if this question belongs elsewhere. In my simple data set focused on sales pursuit opportunities, I have the following columns available. Pursuit Name Status(Open, Won, Lost) Date ...
1
vote
0answers
40 views

why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies

Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only ...
0
votes
0answers
31 views

Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

I want to train a Sequential Neural Net (NN) with Tensorflow. ...
0
votes
2answers
27 views

How match output (pred value) to input value

I'm working with data(with 4 columns which are p(product), M(name of the store)), I want predict the demand of store for that I sued SVR on the data by theses formulation: ...
0
votes
0answers
23 views

R: saving/printing all intermediate results

currently, we have developed a model at work and it is at the verification step. the goal is for another person to receive output from each and every step of calculation to verify. The issue is that ...
-2
votes
3answers
51 views

How to decrease $R^2$ value and change it to positive value [closed]

I'm working on a data, and use regression , as you see bellow: from sklearn.svm import SVR regressor = SVR(kernel = 'linear') regressor.fit(trainX,trainY) above ...
0
votes
1answer
363 views

How to use inverse_transform in MinMaxScaler for pred answer in a matrix

I am working on a data, for preding output, I used SVR by bellow code: ...
0
votes
0answers
18 views

How to find the relationship between one dataset and two others?

From a STEM problem, I vary a variable x within a range and calculate quantities $U(x)$, $V(x)$ and $W(x)$. I want to figure out an analytical relationship between unknown $U(x)$ and the other two ...
0
votes
1answer
15 views

How to Impute Top-Censored Data

What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?
1
vote
0answers
9 views

Is there enough datapoints to make a reasonable predcitction?

I am planning to use this dataset (for a school project) in order to determine the most important features in predicting whether a student will receive a placement. Further I would like to create a ...
2
votes
2answers
44 views

Is it possible to detect drift with real time predictions?

I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or ...
1
vote
0answers
10 views

Comments vs. Files (& how to import w/o packages) for Data Science [closed]

I am trying to settle on an efficient & pythonic workflow for developing scripts in data science work that are scalable and don't turn to spaghetti once things get complicated and parameters are ...
-2
votes
1answer
16 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
0
votes
1answer
24 views

When would C become nescessary to do an analysis or manage data?

I use python in my day to day work as a research scientist and I am interested in learning C. When would a situation arise where python would prove insufficient to manipulate data?
2
votes
1answer
54 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
0
votes
1answer
34 views

Data Cleaning Techniques - What to drop?

I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, ...
0
votes
0answers
29 views

Data calculation/normalisation /data summarisation

Clustering is done based on houses data and say suppose cluster 1 contains 5 houses. Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
2
votes
0answers
27 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
2
votes
1answer
31 views

How would you build a big production ready image training dataset from scratch?

How would you most likely create a large production ready image training dataset from scratch including annotations for a image classification task? We will take a large amount of images (~1 million) ...
0
votes
1answer
24 views

Clusterize item set with items as vectors of features

I have to clusterize this dataset in which I have houses and water consumption in this form: $$ House1 = (x_{1},x_{2}... x_{n});\\ House2 = (y_{1},y_{2}... y_{n});\\ House3 = (z_{1},z_{2}... z_{n});\\ ...
0
votes
0answers
68 views

What exactly negative/positive value of Captum's Integrated Gradient mean?

I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ... Positive attribution score means that the input in that ...
0
votes
1answer
20 views

Microsoft custom vision vs Tensorflow model?

I am planning to implement my own image classifier model using TensorFlow instead of a custom vision platform. what is the biggest difference between custom vision(https://www.customvision.ai/) vs ...
0
votes
0answers
9 views

What makes a high quality image dataset?

I am constructing a custom dataset for image classification. It is a simple task in the sense that only one class needs to be detected. However it is crucial that the dataset is high quality, and I am ...

1
2 3 4 5
13