Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
0
votes
2answers
137 views

Public dataset for account to account payments

I'm looking for a dataset which contains account to account payments (bank transfers). Ideally, this dataset would contain labeled data for transactions or accounts known to be victims of phishing ...
0
votes
0answers
35 views

is padding input images better than resizing image?

I am working with images and was thinking of pre-training VGG19 and EffecientNETB0 model on my dataset. However, a query:- Is ...
0
votes
0answers
26 views

How can I cluster 3-dimensional data?

For a project in my data mining class, I am to perform fuzzy-c means clustering on a data set, where each data point has 3 axes.I'm not exactly sure how I would do so, especially given the clustering ...
0
votes
1answer
50 views

Data Selection according to Feature Values

I am given a dataset consisting of 10 million molecules. Each row contains: The average value predicted by an ensemble of regression models trained to predict a certain property about chemicals (...
1
vote
0answers
9 views
0
votes
1answer
25 views

Why are ordered lists considered one dimensional?

At Uni I am currently taking data visualization as a course. While reading thru the script I stumbled upon two examples for one dimensional data. Example one: Bus stops with their name and order (...
0
votes
0answers
14 views

How to organize data in workflows with user input between steps?

I am building a multistep workflow. User provides input files input.csv, parameters_global.yml, ...
0
votes
0answers
37 views

Data visualization multiple choices multiple condition conditions

Say I have a element that base on a certain condition (1, 2, 3 or 4), it chooses a decision (A,B or C). What diagrams can be used to present this data if I have more than one thousand elements so that:...
0
votes
1answer
24 views

What kind of data analysis problem is this?

This might be a dumb question, but I am an absolute beginner. What kind of data analysis problem is the following? Is it an Exploratory Data Analysis problem/Data Mining problem/data analytics problem?...
2
votes
1answer
47 views

Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?

The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed ...
0
votes
0answers
20 views

Manually labeled data quality check methods?

In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation ...
0
votes
0answers
33 views

Load data from SQL for Machine Learning

I have an SQL DB which has millions of records(10 years of data) and wanted to implement some ML Models. Most of the courses/tutorials explains using a local file or downloaded CSV. But in my case I ...
0
votes
0answers
26 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
1
vote
1answer
37 views

Hi, im currently working for a company that has some inventory control problems

First, I was asked by the manager to make a plot showing produced vs received items, its a multistage process so we are only in charge of one of the steps which is designing, I made the plot comparing ...
1
vote
0answers
8 views

Available data sets on basic income pilot studies? [closed]

I'm interested in exploring data relating to any UBI/basic income pilot studies that have been conducted but have been unable to find any so far? thanks
0
votes
0answers
6 views

Question on What I would use for a Piece of Data

I have 3 categories that we can call A, B, and C. In almost every case we B/A = C. B and A are given in the Dataset and C is what I am find an trying to use elsewhere. The higher number C is the ...
0
votes
1answer
43 views

Which data science framework should I start learning? [closed]

I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of ...
0
votes
1answer
21 views

Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series')

Several terms are used to describe the structure of instances of a dataset. For example, an instance can be 'univariate' if it depends on only one variable or 'multivariate' if it depends on 2 or more ...
1
vote
1answer
64 views

Why does an imbalanced data set badly effect distance measures like Mahalanobis?

I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
0
votes
1answer
257 views

How to specify different columns stacked vertically within CSV using pandas?

I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
0
votes
0answers
13 views

How to Combine Oracle and Snowflake/Redshift data on the fly

Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift. Most ...
1
vote
1answer
19 views

Where can I find a dataset that contains criminal case sentencing data? [closed]

I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as: Type of crime Defendant Age/Sex/Race Plea Verdict Sentence Is there a dataset ...
0
votes
0answers
36 views

Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?

I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
0
votes
1answer
97 views

Problem with binning

I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements. My questions are: Which type of ...
0
votes
0answers
17 views

Code in R, and Normally Distrubuted

...
1
vote
0answers
18 views

MCMC algorithm -- understanding some paremeters

I am trying to understand an MCMC program. I manage to run it, but I am trying to understand the meaning of the some parameters in the analysis. The code is something like this ...
0
votes
0answers
20 views

Download weather data from a http directory

This link contains weather data with a lot of folders and subfolders. I need to download as a whole without visiting every file inside the folders. https://www.ncei.noaa.gov/data/global-forecast-...
0
votes
0answers
26 views

LSTM Train accuracy decreases in training time during the epoch

I'm training an LSTM model for sentiment analysis on a text corpus. There's a thing that I believe is not normal because I never have seen it in training my models. At the start of the epoch, the ...
1
vote
1answer
14 views

Structuring extensive medical histories + demographic information for prediciting future medical outcomes

I'm looking for advice structuring extensive medical histories for predicting future outcomes, specifically hospital admissions. Let's say I want to predict the whether or not someone will be admitted ...
0
votes
1answer
17 views

How to calculate an average cliff date?

Please let me know if this question belongs elsewhere. In my simple data set focused on sales pursuit opportunities, I have the following columns available. Pursuit Name Status(Open, Won, Lost) Date ...
1
vote
0answers
40 views

why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies

Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only ...
0
votes
0answers
255 views

Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

I want to train a Sequential Neural Net (NN) with Tensorflow. ...
0
votes
2answers
32 views

How match output (pred value) to input value

I'm working with data(with 4 columns which are p(product), M(name of the store)), I want predict the demand of store for that I sued SVR on the data by theses formulation: ...
0
votes
0answers
29 views

R: saving/printing all intermediate results

currently, we have developed a model at work and it is at the verification step. the goal is for another person to receive output from each and every step of calculation to verify. The issue is that ...
-2
votes
3answers
53 views

How to decrease $R^2$ value and change it to positive value [closed]

I'm working on a data, and use regression , as you see bellow: from sklearn.svm import SVR regressor = SVR(kernel = 'linear') regressor.fit(trainX,trainY) above ...
0
votes
1answer
4k views

How to use inverse_transform in MinMaxScaler for pred answer in a matrix

I am working on a data, for preding output, I used SVR by bellow code: ...
0
votes
0answers
27 views

How to find the relationship between one dataset and two others?

From a STEM problem, I vary a variable x within a range and calculate quantities $U(x)$, $V(x)$ and $W(x)$. I want to figure out an analytical relationship between unknown $U(x)$ and the other two ...
0
votes
1answer
19 views

How to Impute Top-Censored Data

What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?
1
vote
1answer
14 views

Is there enough datapoints to make a reasonable predcitction?

I am planning to use this dataset (for a school project) in order to determine the most important features in predicting whether a student will receive a placement. Further I would like to create a ...
3
votes
2answers
130 views

Is it possible to detect drift with real time predictions?

I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or ...
1
vote
0answers
15 views

Comments vs. Files (& how to import w/o packages) for Data Science [closed]

I am trying to settle on an efficient & pythonic workflow for developing scripts in data science work that are scalable and don't turn to spaghetti once things get complicated and parameters are ...
-2
votes
1answer
17 views

What data to collect to create useful insights?

I am creating a Social Media app and I want to know what type of data shall I collect to create useful insights for Business accounts or Content Creators? For example if I want to create an insight ...
0
votes
1answer
26 views

When would C become nescessary to do an analysis or manage data?

I use python in my day to day work as a research scientist and I am interested in learning C. When would a situation arise where python would prove insufficient to manipulate data?
3
votes
2answers
477 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
0
votes
1answer
35 views

Data Cleaning Techniques - What to drop?

I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, ...
0
votes
0answers
31 views

Data calculation/normalisation /data summarisation

Clustering is done based on houses data and say suppose cluster 1 contains 5 houses. Appliance 1 is used in 2 houses out of 5 and appliance 2 is used in 4 out of 5 houses.Each appliance produces an ...
2
votes
0answers
39 views

What is the difference between Pachyderm and Git?

I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that: It holds all your data in a central accessible ...
2
votes
1answer
37 views

How would you build a big production ready image training dataset from scratch?

How would you most likely create a large production ready image training dataset from scratch including annotations for a image classification task? We will take a large amount of images (~1 million) ...
1
vote
2answers
30 views

Clusterize item set with items as vectors of features

I have to clusterize this dataset in which I have houses and water consumption in this form: $$ House1 = (x_{1},x_{2}... x_{n});\\ House2 = (y_{1},y_{2}... y_{n});\\ House3 = (z_{1},z_{2}... z_{n});\\ ...
1
vote
0answers
203 views

What exactly negative/positive value of Captum's Integrated Gradient mean?

I use Captum's Integrated Gradient to interprete my PyTorch's neural network. I know that from github and original paper mentioned that ... Positive attribution score means that the input in that ...

1 2
3
4 5
15