Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
573
questions
-1
votes
0answers
6 views
Dataset for plant identification [migrated]
Im looking for a plant identification dataset to train plant identification that is free or not too expensive to use and has plant images, specifically garden plants. Just wondering if anyone came ...
1
vote
0answers
26 views
Association Rules
I'm using association rules for a project and noticed that there is a dearth of papers in the last 10-15 years on the topic although it seemed really popular 15-25 years ago. Is there any reason why?
0
votes
0answers
11 views
Why is scipy.optimize.curve_fit so complicated to use? [closed]
I would like to do some nonlinear curve-fitting on 2D data using scipy.optimize.curve_fit following this tutorial and find the steps required in order to "feed&...
-2
votes
0answers
12 views
data on world press freedom index by RSF [closed]
I have not been able to find the data used by RSF to create their index, which have been obtained through the survey that is disposable on their website. As far as you know, are they somewhere?
0
votes
0answers
14 views
Feature-Target correlation graphs [closed]
I have a set of features and labels with the following dimensions:
where axis=0 corresponds to the number of pixels and axis=1 corresponds to the features and time, for example, 17=117 where 17 is the ...
0
votes
0answers
7 views
Is there any GitHub pull-request dataset with labels information? [closed]
I also tried to get the labels for a pull-request using GitHub API, but that did not work as labels are not linked with the pull-request. Any idea? How can I get the PR comments labels for GitHub ...
0
votes
1answer
16 views
How to use SimpleImputer class to multiple numeric columns in a datasets? [closed]
impter=imputer.fit(df[["Age","Salary"]])
Error
Input contains NaN, infinity or a value too large for dtype('float64').
0
votes
1answer
18 views
Combining Two CSV's in Jupyter Notebook
I want to combine both CSV files based on Column1, also when combined each element of Column1 of both csv should match and also each row or Please suggest how to reorder Column1 according to another ...
0
votes
0answers
7 views
Are there labeled multivariate time series data sets where only a subset/partial number of time series is relevant for each class?
I am looking for a data set of multivariate time series data which contains classes that only require a subset of time series for their identification. Assume a human activity recognition data set ...
0
votes
1answer
16 views
How would you encode missing pixels in image data?
I am working through an example on the MNIST dataset, and was just curious, if your image input data were missing some pixels, how would you encode it. Since the values are always positive, and ...
1
vote
2answers
84 views
How to Make Meaningful Conclusions here?
I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question.
The question-
Suppose 7.5% of the ...
0
votes
0answers
21 views
Is there a way to calculate the degree of classification of data points in a set?
I'm trying to write a program in python that takes in list of all stocks on different exchanges and filters them out according to multiple criteria (e.g. marketcap has to be larger than 500m, PE ratio ...
-1
votes
1answer
22 views
Matrix of pairwise cosine similarities from matrix of vectors [closed]
I have a matrix of 200d vectors stored as follows:
$ X = \begin{pmatrix} \text{id}_1 & 0.5 & -2 & \dots & 10
\\ \text{id}_2 & -4 & 6 & \dots & -0.3
...
2
votes
2answers
47 views
X, Y names in data-science [closed]
I am from CS background and moving towards Data Sciences, I have came to know ML is highly influenced by Statistical Inference/signal processing. The X we use in data science is called input, feature ...
0
votes
0answers
9 views
Free exchange rate data API for all countries at at least daily frequency since at least 2000?
I thought it would be a very common dataset: world exchange rates at daily frequency for every currency pair (or each currency's rate vs USD so triangulation can fill in for every other rate)
But ...
1
vote
0answers
35 views
cor() function giving unexpected result for numeric data frame(Updated) [closed]
I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the ...
1
vote
0answers
21 views
Show overall average on plots of single data
I have data from a survey from multiple locations (1 data set per location per year, each containing about 20-25 numerical scores from 0 to 5). This data is aggregated over topics (e.g. first 5 ...
0
votes
0answers
33 views
User-Item based Recommendation system with data containing binary data
I have a data set which contains about 400,000 unique items present on a platform. The users on this platform can like and save this in their own list. Now, I have about 4000 users with their like ...
0
votes
1answer
16 views
How should I choose n_features in FeatureHasher in sklearn?
How should I choose n_features in FeatureHasher in sklearn ? assume that I have 1000 categories in feature "case" and I want to hash them
1
vote
0answers
16 views
Correlation with Multi-Dimensional Clustering
I have a dataframe with multiple features, where I'm selecting 3 features to cluster on. Ex.
...
5
votes
1answer
316 views
One Hot Encoding for any kind of dataset
How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how ...
0
votes
1answer
23 views
Exploratory Data Analysis on dataset divided by winners and losers
I have a dataset where I have features from winning tennis players and the other half are from a losing tennis players:
...
0
votes
0answers
12 views
How can a CNN account for spectro-temporal constraints in neural data?
What are there the best ways to leverage the unique "geometrical" constraints of spectro-temporal signal representations (architecture, filter shapes, data augmentation, etc.)? For example, ...
2
votes
1answer
155 views
How can I count the number of matching zero elements between two numpy arrays?
I have a function that returns the predicted accuracy of a time-series model. I have two equally-sized numpy arrays, one for the actual direction and one for the predicted direction. I'm classifying ...
1
vote
2answers
29 views
Modality of data
Can anyone please explain in clear words what is generally meant by "modality of data"?
I know what modality means with respect to distributions.
2
votes
1answer
79 views
SMOTE oversampling for class imbalanced dataset introduces bias in final distribution
I have a problem statement where percentage of goods (denoted by 0) is 95%, and for bads (denoted by 1) it is 5% only. One way is to do under sampling of goods so that model understands the patterns ...
0
votes
0answers
14 views
How do I efficiently load data from disk during training of deep learning models in pytorch?
I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this?
It seems like HDF5 is a common method that people ...
2
votes
4answers
36 views
Scikit-learn: train/test split to include have same representation of two different types of values in a column
I have a dataset of online purchase orders that contains two types of customers:
Customers who have an account and thus are known customers with unique customer number.
Customers who do not have an ...
1
vote
0answers
26 views
pandas: reshaping the dataset, moving column name as rows [closed]
I currently have the following table:
...
0
votes
1answer
14 views
Historical exchange rate data via World Bank API?
I can see a massive list of indicators available via the World Bank API here (may take 1-2 minutes to load)
I found exchange rates for various currencies into USD, but cannot see how to get exchange ...
0
votes
0answers
24 views
Compare two columns and visualize third column based on comparison in Power BI
I am new to Power Bi, learning myself and I got a dataset with some questions. Most of the questions are about comparing two columns and visualize the effect on the third column.
For example , lets ...
1
vote
1answer
43 views
Dealing with missing data
I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
1answer
21 views
The OLAP (On-Line Analytical Processing) cube with 4 dimensions [closed]
A typical OLAP cube looks like this:
As I can see, this cube can work with 2 or 3 dimensions, but what if I have 4 dimensions to produce facts?
Should I use star schema instead when having more than ...
0
votes
0answers
15 views
Determining common columns among many datasets without having to manually inspect them
I have a bunch of datasets (10+) and I want to be able to join/ relate them to one another through columns that may contain similar values. For example, date and location are common columns that could ...
-3
votes
1answer
22 views
Accuracy of a forecasting model for prediction of COVID-19 occurence
My goal is to find the best performing forecasting model for the occurrence of COVID-19 in Toronto.
I pre-train the network with data on the occurrence of SARS in ten countries and Toronto. Then I ...
2
votes
0answers
41 views
Regression dataset with categorical features
I have thought of a regression technique that I want to try on several datasets. I would like these datasets to have the following properties:
Be a tabular dataset (no images).
Have at least 20k rows,...
0
votes
1answer
12 views
Ranking tumors from most malignant to most benign
I have a classification problem classifying tumors as benign or malignant. However, I want to go a step further to provide a ranking of these tumors as most malignant to most benign. Are there any ...
1
vote
1answer
22 views
Random Forest in R with only character variables
I am new to using random forest in R an my goal is to identify the independent variables which have the highest impact on the dependent variables. I am looking at sales data, and sales is my dependent ...
-1
votes
2answers
28 views
Data Selection & Visualization Tool
I am looking for a tool that will show neatly all the rows from a large table (100GB+) for complex selection conditions from an sql/text/csv table.
DB may be on linux or windows.
Looking for easy ...
0
votes
0answers
20 views
Why use raw.githubusercontent.com link in a script?
In the book "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow"
The author provides the link:
DOWNLOAD_ROOT = "https://raw.githubusercontent.com/ageron/handson-ml2/master/"
My ...
0
votes
1answer
23 views
Higher variance in PCA can mean, that data structure is less informative?
I have the possibility to describe data with two different data structures, both data structures are some sort of approximation to the true data. I would like to compare the two data structures with ...
0
votes
0answers
11 views
Normalization around the nominal value while avoiding to get too big results
Lets say we measured a signal N times. Each measurement contains T time instance. For example for N=4 measurements and T=5 time instances the data matrix is as follows:
...
0
votes
1answer
22 views
How to adjust/smooth a certain number using constants or rules
Hi, I am handling a dataset with a customer purchase history.
The field ord_cnt represents the purchase without coupon usage, and cpn_ord_cnt represents the purchase with coupon usage.
There are two ...
2
votes
2answers
46 views
Data Imbalance in Regression Tasks [closed]
Having read a lot about class imbalance in classification tasks, I'd like to know what is the methodology for data imbalance in regression tasks.
Particularly,
- What is the procedure to check for ...
3
votes
2answers
203 views
Is it acceptable not to transform() test data after train data is being fit_transform()-ed
We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
0
votes
0answers
18 views
What's the best way to store and then call 3 Million+ records for training?
Good afternoon all,
I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline).
Essentially I've built a function that can convert raw data into ...
2
votes
3answers
56 views
Continuous VS Categorical variable
In the dataset I have a continuous variable AGE and categorical variable AGE_CATEGORY as well. They are both highly correlated.
...
1
vote
2answers
21 views
Tools for nutritional value tracking and visualization
I want to track the nutritional value of different flours over time. What I got so far is a
data table like this:
...
0
votes
0answers
9 views
Resources for building data pipelines
I am finding it difficult to find sources which walk you through building a data pipeline from scratch, for example from:
- SQL Server (where I want to do my cleaning/ETL)
- R/Python (where I want to ...
0
votes
1answer
119 views
Data augmentation for multiple output heads in Keras
I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a ...
