Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
633
questions
0
votes
0answers
8 views
Data Selection according to Feature Values
I am given a dataset consisting of 10 million molecules. Each row contains:
The average value predicted by an ensemble of regression models trained to predict a
certain property about chemicals (...
1
vote
0answers
8 views
1
vote
3answers
16k views
Is there a way to replace existing values with NaN
I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column ...
0
votes
0answers
23 views
Convert a dataset to a binomial distribution in R [closed]
I have an excel file that has COVID-19 data for Indiana. Which can be found here: https://hub.mph.in.gov/dataset/covid-19-county-statistics. The data has the following columns:
ZIP_CD (zipcode)
...
-1
votes
1answer
75 views
What are the differences between IBM BlueMix and IBM Data Science Experience?
This may seem like a silly question, but as I am going through the documentation for both services it is difficult to disentangle what each does, specifically. From what I've gathered
BlueMix is ...
0
votes
1answer
17 views
Why are ordered lists considered one dimensional?
At Uni I am currently taking data visualization as a course.
While reading thru the script I stumbled upon two examples for one dimensional data.
Example one:
Bus stops with their name and order (...
0
votes
0answers
11 views
How to organize data in workflows with user input between steps?
I am building a multistep workflow.
User provides input files input.csv, parameters_global.yml, ...
0
votes
0answers
18 views
Data visualization multiple choices multiple condition conditions
Say I have a element that base on a certain condition (1, 2, 3 or 4), it chooses a decision (A,B or C). What diagrams can be used to present this data if I have more than one thousand elements so that:...
0
votes
1answer
20 views
What kind of data analysis problem is this?
This might be a dumb question, but I am an absolute beginner.
What kind of data analysis problem is the following?
Is it an Exploratory Data Analysis problem/Data Mining problem/data analytics problem?...
1
vote
1answer
24 views
Are we allowed to transform the continuous target variable by creating a log transformation in order to have a normal distribution?
The following code gives the target variable Item_Outlet_Sales before transformation and Item_Outlet_Sales_log which is transformed
...
0
votes
0answers
11 views
Manually labeled data quality check methods?
In this article, the author says that manual data labeling is a "not-as-bad-as-it-sounds" method, but to err is human and in this article by Shopify, I learned that high-quality annotation ...
1
vote
0answers
143 views
Association between Categorical Variables and regression
We perform data analysis and build models. Say, for example, I built a regression model that has more than one predictor (multiple regression). We then check many things: normality, multicollinearity, ...
0
votes
0answers
22 views
Load data from SQL for Machine Learning
I have an SQL DB which has millions of records(10 years of data) and wanted to implement some ML Models. Most of the courses/tutorials explains using a local file or downloaded CSV. But in my case I ...
1
vote
1answer
42 views
Best practice - memory management when data wrangling
I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below:
...
0
votes
0answers
21 views
What should be the architecture design for the dataset to be used for a machine learning API
I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
0
votes
1answer
37 views
Image classification with CNN Python
I'm working on image classification using CNN, my dataset contains more than 50 classes (50 folders) which represent the types of car parts, and in each folder we have vehicle brands, each vehicle ...
2
votes
1answer
47 views
Python is reading my data with NANS and Infs, but they don't have any
I'm having an issue in Python where it says that the dataframe I have loaded through pandas.read_csv() cannot be scaled using ...
0
votes
2answers
9k views
How do I read a dat file, for which I don't know its structure?
Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it.
How do I figure out the contents in my dat file. ...
1
vote
1answer
34 views
Hi, im currently working for a company that has some inventory control problems
First, I was asked by the manager to make a plot showing produced vs received items, its a multistage process so we are only in charge of one of the steps which is designing, I made the plot comparing ...
1
vote
2answers
2k views
Orange3 summarizing data, grouping data values
Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values?
For example, instead of plotting a scatter with lots of data points, I would like to ...
0
votes
1answer
25 views
Exploratory Data Analysis on dataset divided by winners and losers
I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players:
...
2
votes
1answer
57 views
Balance data using different criteria
I have a dataset of audio and text files that I want to balance using different criteria to train a neural network. The text and matching audio file are grouped under one ID.
For each ID, I have a ...
0
votes
1answer
58 views
What to do if a specific label of a category appears only a few times?
Let's say I am trying to predict whether a car will be auctioned or not (not what I'm actually trying to do, but it represents it pretty well) using tabular data. I have the year the car was made, its ...
1
vote
2answers
31 views
Tools for nutritional value tracking and visualization
I want to track the nutritional value of different flours over time. What I got so far is a
data table like this:
...
0
votes
2answers
49 views
Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?
I have several marketing files that I need to download on a weekly basis.
Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, ...
1
vote
0answers
6 views
Available data sets on basic income pilot studies? [closed]
I'm interested in exploring data relating to any UBI/basic income pilot studies that have been conducted but have been unable to find any so far?
thanks
0
votes
0answers
6 views
Question on What I would use for a Piece of Data
I have 3 categories that we can call A, B, and C. In almost every case we B/A = C. B and A are given in the Dataset and C is what I am find an trying to use elsewhere. The higher number C is the ...
13
votes
7answers
6k views
Interactive labeling/annotating of time series data
I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use ...
0
votes
1answer
38 views
Which data science framework should I start learning? [closed]
I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of ...
0
votes
1answer
18 views
Technical term to describe/differentiate the structure of an instance by the number of samples per instance ('tuple' vs 'time series')
Several terms are used to describe the structure of instances of a dataset. For example, an instance can be 'univariate' if it depends on only one variable or 'multivariate' if it depends on 2 or more ...
0
votes
1answer
139 views
PCA in visual Analytics
I m studying visual analytics and i have a theoretical question about this topic.
My professor introduced this schema in him slide
For connect data to visualisation.
Some topic is very easy to ...
2
votes
3answers
1k views
Modality of data
Can anyone please explain in clear words what is generally meant by "modality of data"?
I know what modality means with respect to distributions.
1
vote
1answer
51 views
Why does an imbalanced data set badly effect distance measures like Mahalanobis?
I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data ...
0
votes
1answer
30 views
How to specify different columns stacked vertically within CSV using pandas?
I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the ...
1
vote
2answers
175 views
How to cluster categorical and numerical data in the same dataset?
I have a dataset in which it contains both numerical and categorical data. This can be done using supervised learning algorithms, but I am eager to see how this data can be clustered using some ...
8
votes
6answers
2k views
Generate timeseries data
Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
26
votes
4answers
20k views
Is pandas now faster than data.table?
Here is the GitHub link to the most recent data.table benchmark.
The data.table benchmarks has not been updated since 2014. I ...
13
votes
3answers
27k views
How to create US state choropleth map
I have a value associated with each US state (let's pretend it's the average temperature in January for each state). I want to display this data as a heat map of the United States. To be clear, it ...
2
votes
0answers
27 views
What is the difference between Pachyderm and Git?
I learned that tools like Pachyderm version-control data, but I cannot see any difference between that tool with Git. I learned from this post that:
It holds all your data in a central accessible ...
0
votes
0answers
8 views
How to Combine Oracle and Snowflake/Redshift data on the fly
Have a Schema.Table_Name on Oracle. Exactly similar table is on Snowflake with same metadata(Schema.Table_Name). How can I combine data from two on the fly without moving Oracle data to Redshift.
Most ...
2
votes
1answer
993 views
How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library
Sorry for a long story, but it is a long story. :)
I am using h2o library for python to build a decision tree and to extract a decision rules out of it.
I use some data for training where labels get ...
0
votes
1answer
313 views
R - How do I remove a varying number of digits from a date-vector
I want to remove a varying number of digits from a date vector. My date vectors looks like this:
I want to convert this vector into a date vector, but first I have to get rid of the number in front ...
1
vote
1answer
19 views
Where can I find a dataset that contains criminal case sentencing data? [closed]
I would like to study a dataset where each record represents a criminals case in the US and contains attributes such as:
Type of crime
Defendant Age/Sex/Race
Plea
Verdict
Sentence
Is there a dataset ...
0
votes
1answer
15 views
How to Impute Top-Censored Data
What are the main imputation methods that can be used to deal with top censored panel income data if a big proportion (about 40%) is censored?
0
votes
0answers
22 views
Can I generate synthetic data after getting approximate distribution parameters using Kolmogorov-Smirnov test?
I have a very limited amount of data. I need to generate more synthetic data . My data shape is (x,21) . Right now , I have used KS (Kolmogorov-Smirnov) test to get the closest matching distribution ...
2
votes
1answer
8k views
What is “noise” in observed data?
I am reading pattern Recognition and machine learning by Bishop and in the chapter about probability, "noise in the observed data" is mentioned many times. I have read on the internet that noise ...
0
votes
1answer
20 views
Problem with binning
I am trying to change continuous data points to categorical by using binning. I know two techniques, i) equal width bins ii) bins with equal number of elements.
My questions are:
Which type of ...
0
votes
0answers
17 views
1
vote
0answers
17 views
MCMC algorithm — understanding some paremeters
I am trying to understand an MCMC program. I manage to run it, but I am trying to understand the meaning of the some parameters in the analysis.
The code is something like this
...
2
votes
2answers
45 views
Is it possible to detect drift with real time predictions?
I have been reading up on detecting data drift and concept drift, I have found this library but it seems all the methods here detect concept drift and take input as if the prediction was correct or ...
