Questions tagged [sparse]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
30 views

Feature selection in high-dimensional datasets with sparse features

What are the most effective techniques for feature selection in high-dimensional datasets with sparse features in the field of natural language processing?
Todd Takala's user avatar
1 vote
1 answer
74 views

Is it valid to use Spark's StandardScaler on sparse input?

While I know it's possible to use StandardScaler on a SparseVector column, I wonder now if this is a valid transformation. My reason is that the output (most likely) will not be sparse. For example, ...
user12138762's user avatar
0 votes
0 answers
7 views

Is there a specific gain from using a dataset of sparse images in CNN training instead of regular images?

By sparse images, I mean images where each R, G, B value is either 0 or 1. Does this contribute in faster training or any other process of NN training? My guess would be that having multiple nodes ...
Amadeo Amadei's user avatar
0 votes
0 answers
12 views

What is the disadvantage of sparse NLP models?

I was reading this paper -- in preparation for a job interview -- and I was trying to determine what the advantages and disadvantages of the approach were. So I feel that the advantages were stated by ...
yishairasowsky's user avatar
0 votes
0 answers
11 views

how to effectively select "locally" important variables in regression (like an imbalanced variable with a lot of zeros and few 1)

so I have situations where there are some variables which I know are relevant from a business perspective which might decrease variance on a part of the dataset. example: while estimating cars values, ...
Asher11's user avatar
  • 159
0 votes
0 answers
28 views

Which algorithm to use for predictors which are sparse for classification problem

I have a classification problem with target has 85% to 15 % ratio (0,1) and around 35 predictor which all are 0 or 1 , I tried building logistic regression however the auc is around 0.53 , I am not ...
Dexter1611's user avatar
0 votes
1 answer
115 views

Impact of many zeros in LightGBM Regressor training set [duplicate]

I have a LightGBM Regressor model with 15 features. 5 of these features have 98.7% NA for the training set. All five of the features are NA for each row. I impute the missing values with zero before I ...
ashton's user avatar
  • 1
1 vote
1 answer
823 views

Clustering for Sparse Data Matrix of high dimension

I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
Is land's user avatar
  • 11
0 votes
1 answer
40 views

ML/DL model needed to perform binary classification on binary input image dataset

I desperately need help regarding ML/NN models that would be appropriate for binary input data.. So, I have an image dataset in which [R,G,B] values can only take ...
Amadeo Amadei's user avatar
1 vote
0 answers
78 views

How to implement simple VAE with sparse tensor in Tensorflow

thanks for reading. I have been attempting to train a simple VAE on very sparse 2D and 3D data. So far I have been training using dense tensors which - I think - is resulting in horrible training due ...
Zephrom's user avatar
  • 11
1 vote
1 answer
202 views

Techniques for classification AI with sparse labels

I want to create an AI to classify images with a large set of labels (1000+ labels). However, the labels in the data set are correct but each image is not fully labelled. This means that each image's ...
Liam's user avatar
  • 11
1 vote
0 answers
29 views

Suggestions for binary time-series-classification model for small dataset

Hopefully I´m at the right place for my question: I´m looking for suggestions for models to use to classify multivariate time series. I´m trying to find a way of classifying the behaviour of motors ...
sensation96's user avatar
2 votes
1 answer
2k views

Best metric and hyperparameters in dimension reduction with UMAP for binary sparse data

I am playing with a dimensionality reduction step prior to clustering for a pretty large sparse binary matrix of almost 3000 columns and 50k rows. My idea is to embed the 3000 dimensions into a two-...
linello's user avatar
  • 121
1 vote
0 answers
89 views

Linear regression on sparse matrix?

I have a matrix with sparse data. A small extract from it is seen below. The columns represent years and the rows represent different race tracks. The feature values are velocities on that specific ...
Hanson's user avatar
  • 11
0 votes
1 answer
257 views

Dictionary learning for image classification

I'm wondering if the approach I'm thinking of could even work. I want to use dictionary learning for image classification. The first step would be to learn the dictionary from a set of similar yet ...
Jakub Małecki's user avatar
1 vote
2 answers
664 views

Predicting high frequency sparse time series data in python

I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0's, since there is no continuous flow of cars coming to charge but rather ...
Ruben Kl's user avatar
1 vote
3 answers
892 views

How to create a big data frame in Python

I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error. I tried ...
user avatar
1 vote
1 answer
425 views

Predicting sparse time series data

I have a dataset of a couple of EV charging stations (10 min frequency) over 1 year. This data consists of lots of 0s, since there is no continuous flow of cars coming to charge, but rather ...
Ruben Kl's user avatar
0 votes
1 answer
67 views

How to work with input which is a combination of metadata+ vectorized text data + image pixel data to build a Regression Model (predict views)?

There are 4 datasets (all in csv format), each has a uniqueID column by which each record can be identified. Image and text datasets are dense datasets.(need to be converted to ndarray). Can someone ...
Mathew's user avatar
  • 31
2 votes
1 answer
860 views

Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

I'm following a guide here to implement image segmentation in Keras. One thing I'm confused about are these lines: ...
TomSelleck's user avatar
1 vote
1 answer
106 views

Anomaly detection on sparse categorical data

I have a big dataset with a column "clientid" and a categorical column "choice". I want to find out what are the clients that have strange combinations of choices (less frequent ...
DataLover's user avatar
0 votes
1 answer
146 views

Correlation/distance between sparse vectors

I am looking for a metric for comparing gene count tables. These are long columns of data (a few millions genes by a few dozen samples), with all non-negative entries, about 90% of which are zeros. ...
Roger Vadim's user avatar
1 vote
1 answer
813 views

unsupervised anomaly detection on sparse data

Given that I have a very sparse data matrix with continuous features, like this dataframe for example ...
greghouse1's user avatar
0 votes
1 answer
357 views

Softmax regression cost function code [closed]

I really do not understand what does this code do M = sparse.coo_matrix(([1]*n, (Y, range(n))), shape=(k,n)).toarray() The code is related to calculating the ...
Mostafa Atallah's user avatar
1 vote
0 answers
20 views

Autoencoder for Extremely Sparse Data

I am attempting to train an autoencoder on data that is extremely sparse. Each datapoint is only zeros and ones and contains ~3% 1s. Being that the data is mostly zero the autoencoder learns to ...
PDPDPDPD's user avatar
  • 113
0 votes
0 answers
39 views

Classifying sparse binary data for int value

I'm very new to data science and still trying to get the grips. The problem I'm trying to tackle is, we have a pool of footballers from a league and data objects representing a group of 11 footballers ...
abdus_salam's user avatar
1 vote
0 answers
26 views

Sparse Covariance Selection

I was reading this article https://www.di.ens.fr/~aspremon/PDF/CovSelSIMAX.pdf, whose goal is to estimate the covariance matrix from a the sample covariance matrix drawn from a distribution $X$. ' ...
mbz0's user avatar
  • 11
1 vote
0 answers
46 views

Custom Loss Function for Mixing Sparse and Dense Features for a Prediction Problem

I have a largely uncorrelated feature space of about 40 dichotomous features, using which I'm trying to predict a continuous target variable. Now, some of these features are very sparse (Active less ...
Recyclops's user avatar
3 votes
0 answers
2k views

Converting pandas dataframe to scipy sparse arrays

Converting pandas data frame with mixed column types -- numerical, ordinal as well as categorical -- to Scipy sparse arrays is a central problem in machine learning. Now, if my pandas' data frame ...
learner's user avatar
  • 359
1 vote
1 answer
588 views

What are the sparse and dense vector ? I cant undestand ,can you explain to me?please.Why do we use for?

I am new to neural networks, embeddings, etc. I am struggling understanding things like sparse representation, embeddings, and especially sparse vectors. Could you explain these to me? Why do we need ...
Tugba Ozkan's user avatar
3 votes
1 answer
2k views

Convert Pandas Dataframe with mixed datatypes to LibSVM format

I have a pandas data frame with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, ...
learner's user avatar
  • 359
1 vote
0 answers
294 views

Scipy Sparse vstack memory error

I have a bunch of scipy matrices (of the same #columns) loaded from disk. I want to combine them into one scipy sparse matrix. I am using scipy sparse vstack method. I am able to load the ...
SHASHANK GUPTA's user avatar