Questions tagged [sparse]
The sparse tag has no usage guidance.
25
questions
0
votes
0
answers
17
views
How to calculate dataset and feature sparsity/density
I have a dataset with 8 features and 30,000 samples but which is probably a sparse sampling. I would like to quantify how sparse or dense the dataset and individual features are, as described in the ...
- 101
1
vote
1
answer
32
views
Techniques for classification AI with sparse labels
I want to create an AI to classify images with a large set of labels (1000+ labels). However, the labels in the data set are correct but each image is not fully labelled. This means that each image's ...
- 11
0
votes
0
answers
4
views
Regression with high dimensional sparse boolean predictors
Suppose I have a continuous y response variable and a very large matrix of boolean sparse predictor variables X. What would be the best regression method to use?
0
votes
0
answers
11
views
What exactly is activity sparsity and why is it beneficial?
I have been reading about weight sparsity and activity sparsity with regard to convolutional neural networks. Weight sparsity I understood as having more trainable weights being exactly zero, which ...
- 101
1
vote
0
answers
16
views
Suggestions for binary time-series-classification model for small dataset
Hopefully I´m at the right place for my question:
I´m looking for suggestions for models to use to classify multivariate time series. I´m trying to find a way of classifying the behaviour of motors ...
- 11
2
votes
1
answer
490
views
Best metric and hyperparameters in dimension reduction with UMAP for binary sparse data
I am playing with a dimensionality reduction step prior to clustering for a pretty large sparse binary matrix of almost 3000 columns and 50k rows.
My idea is to embed the 3000 dimensions into a two-...
- 121
1
vote
0
answers
36
views
Linear regression on sparse matrix?
I have a matrix with sparse data. A small extract from it is seen below. The columns represent years and the rows represent different race tracks. The feature values are velocities on that specific ...
- 11
0
votes
1
answer
94
views
Dictionary learning for image classification
I'm wondering if the approach I'm thinking of could even work.
I want to use dictionary learning for image classification. The first step would be to learn the dictionary from a set of similar yet ...
- 101
1
vote
1
answer
298
views
Predicting high frequency sparse time series data in python
I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0's, since there is no continuous flow of cars coming to charge but rather ...
- 11
1
vote
3
answers
644
views
How to create a big data frame in Python
I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error.
I tried
...
user109341
1
vote
1
answer
96
views
Predicting sparse time series data
I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0s, since there is no continuous flow of cars coming to charge, but rather ...
- 11
0
votes
1
answer
61
views
How to work with input which is a combination of metadata+ vectorized text data + image pixel data to build a Regression Model (predict views)?
There are 4 datasets (all in csv format), each has a uniqueID column by which each record can be identified. Image and text datasets are dense datasets.(need to be converted to ndarray).
Can someone ...
- 31
2
votes
1
answer
409
views
Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?
I'm following a guide here to implement image segmentation in Keras.
One thing I'm confused about are these lines:
...
- 115
1
vote
1
answer
85
views
Anomaly detection on sparse categorical data
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ...
- 19
0
votes
1
answer
63
views
Correlation/distance between sparse vectors
I am looking for a metric for comparing gene count tables. These are long columns of data (a few millions genes by a few dozen samples), with all non-negative entries, about 90% of which are zeros. ...
- 101
1
vote
1
answer
489
views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
- 11
0
votes
1
answer
203
views
Softmax regression cost function code [closed]
I really do not understand what does this code do
M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray()
The code is related to calculating the ...
1
vote
0
answers
17
views
Autoencoder for Extremely Sparse Data
I am attempting to train an autoencoder on data that is extremely sparse. Each datapoint is only zeros and ones and contains ~3% 1s. Being that the data is mostly zero the autoencoder learns to ...
- 113
0
votes
0
answers
32
views
Classifying sparse binary data for int value
I'm very new to data science and still trying to get the grips. The problem I'm trying to tackle is, we have a pool of footballers from a league and data objects representing a group of 11 footballers ...
- 109
1
vote
0
answers
20
views
Sparse Covariance Selection
I was reading this article https://www.di.ens.fr/~aspremon/PDF/CovSelSIMAX.pdf, whose goal is to estimate the covariance matrix from a the sample covariance matrix drawn from a distribution $X$.
' ...
- 11
1
vote
0
answers
46
views
Custom Loss Function for Mixing Sparse and Dense Features for a Prediction Problem
I have a largely uncorrelated feature space of about 40 dichotomous features, using which I'm trying to predict a continuous target variable.
Now, some of these features are very sparse (Active less ...
- 11
3
votes
0
answers
2k
views
Converting pandas dataframe to scipy sparse arrays
Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning.
Now, if my pandas' data frame ...
- 339
0
votes
1
answer
316
views
What are the sparse and dense vector ? I cant undestand ,can you explain to me?please.Why do we use for?
I am new to neural networks, embeddings, etc. I am struggling understanding things like sparse representation, embeddings, and especially sparse vectors. Could you explain these to me? Why do we need ...
2
votes
1
answer
1k
views
Convert Pandas Dataframe with mixed datatypes to LibSVM format
I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
- 339
1
vote
0
answers
214
views
Scipy Sparse vstack memory error
I have a bunch of scipy matrices (of the same #columns) loaded from disk.
I want to combine them into one scipy sparse matrix.
I am using scipy sparse vstack method.
I am able to load the ...
- 3,495