All Questions
34,489
questions
0
votes
0
answers
2
views
LSTM Feature engineering: using different Knowledge Graph data types
For a research project, I'm planning to use an LSTM to learn from sequences of KG entities. However, I have little experience using LSTMs or RNNs in general. During planning, a few questions ...
- 1
0
votes
0
answers
3
views
As a newbie in Data Science, should I learn the main python language first or should I learn it's libraries
I'm currently learning as a beginner and I need guidance. Thank you
0
votes
0
answers
3
views
How to find significance for Gini coefficient changes?
I'm using the Gini coefficient to evaluate the performance of a model.
Making some changes (feature selection, hyperparameter tuning, etc.) I created variant models with different Gini coefficients.
...
- 594
0
votes
0
answers
8
views
How does batch normalization make a model less sensitive to hyperparameter tuning?
Question 22 of 100+ Data Science Interview Questions and Answers for 2022 asks What is the benefit of batch normalization?
The first bullet of the answers to this is The model is less sensitive to ...
- 162
3
votes
1
answer
43
views
Time series data for prophet model
I have a time series data that has some missing dates. For example, in the dataset below '2017-08-06' is missing.
To enter this data in the Prophet model, do I have to create the data frame with all ...
- 133
1
vote
0
answers
7
views
Algorithm suggestion for correlated models
I'm looking for suggestions on how to proceed with predicting on separate but correlated models.
The example I will use is housing data. I have three inputs:
Latitude
Longitude
1-Google Street View ...
- 495
2
votes
1
answer
283
views
URGENT: who knows how many matrix combinations this is
Input Matrix:
A B C D E
A 1 - - - -
B - 1 - - -
C - - 1 - -
D - - - 1 -
E - - - - 1
...
- 185
1
vote
1
answer
11
views
Building a Time Series Model
I am working on building a time series model, but the dataset I have only has date features for the year; the month and date are not available. What would be a suitable model to use and is it even ...
- 11
0
votes
0
answers
5
views
How can SHAP feature importance be greater than 1 for a binary classification problem?
Let's say I build a binary classification model to predict survival on the Titanic. I then use SHAP to get feature importance for each feature. I see that the SHAP importance for the top feature, <...
- 101
0
votes
0
answers
17
views
Pre-trained BERT model running slow on CPU
I'm running a inference model using a pre-trained BERT model (BERTikal). The model works but is not fast enought running on CPU. It's taking about 5 minuts to classify a batch of 300 phrases.
The the ...
0
votes
0
answers
10
views
Input size vs hidden state in RNNs
Im using PyTorch to implement RNNs on univariate time series data. This is the documentation for the RNN class: link
I think I'm understanding the math behind an RNN cell. But I have an specific ...
- 101
0
votes
1
answer
10
views
Preserve relations between data points when preprocessing
I am tasked with a project that aims to predict the probability of a product being returned before the product is even ordered.
I have an excel containing a bunch of orders. In order to make ...
0
votes
1
answer
13
views
Proof of perpendicular distance of an observation from the Maximal Margin Hyperplane
I was reading about Maximal Margin Classifiers in "Introduction to Statistical Learning" and could not understand how is the perpendicular distance of an observation (which is a vector) from ...
0
votes
1
answer
25
views
Advantages of different tokenizers for NLP (specifically text generation)
What are the advantages of using different tokenizers? For example, let's take the sentence:
"In Düsseldorf I took my hat off. But I can't put it back on."
The treebank tokenizer yields: &...
0
votes
0
answers
11
views
Incremental semantic similarity with sentence embedding using sentence_transformers
I'm trying to find similar sentences to a given query sentence from a corpus. Also, I want to incrementally add new sentences to that corpus for future prediction without retraining the whole corpus ...
- 1
0
votes
1
answer
29
views
Draw a decision tree with depth 2 that is consistent with the data
I am trying to come up with a solution to this for an exam preparation but cant come up with anything, dont know how to tackle it... if i use information gain the depth increases beyond 2.
What would ...
0
votes
0
answers
10
views
Distance between deterministic policies that are not probability distributions
This question asks if there is a way to measure distance between policies that are in fact probability distributions.
In the case of continuous control with deterministic policies where they take a ...
- 117
0
votes
0
answers
11
views
Predicted images are quite good with loss=0.20 while are black with loss=0.02
I'm trying to train a U-net with VGG16 as a backbone in order to recognize 4 classes: sky, rocks, trees and background in a dataset of about 10000 images.
I'm using categorical crossentropy as a loss ...
- 111
0
votes
0
answers
8
views
Time-Series Similarity & Clustering
I need to investigate Python-based tools for time series clustering and / or similarity matching on specific dataset and evaluate different approaches. Could you please suggest approaches that could ...
0
votes
0
answers
14
views
VC- dimension calculate
Let X = {1, 2, 3, ... , 100}. Let H be the class of all subsets of X that contain at least 20 and at most 80 elements. What is the VC-dimension of H?
0
votes
0
answers
13
views
- Models to rank sentences
I am working with tasks made by some occupations and am trying to find out the importance of these tasks within the occupation. My solution was to use tf-idf and then text rank and use word2vec and ...
0
votes
1
answer
13
views
Training XGBoost on time series features of varying sample length
I have some time series data that contain features that that go back anywhere from 5 to 50 years. I've considered imputation (e.g. taking the mean), but I'm not sure it's feasible to impute such large ...
0
votes
0
answers
14
views
Multiclass classification (gradient boosted trees) predictions distribution using softmax()
Let's consider a multiclass problem where the target is composed by 20% class 'A', 50% of class 'B' and 30% of class 'C'.
The model is trained and then the class predictions are obtained via the ...
- 133
3
votes
1
answer
30
views
Logistic Regression using Logisticregression() class
In the documentation of Logisticregression() offered by sklearn library, it states the following note:
The underlying C implementation uses a random number ...
- 199
0
votes
1
answer
18
views
Kernel ridge regression (KRR), accuracy scale?
What does a good range for the accuracy score look like for the KRR model?
For example, RMSE produces a value between 0 and 1, where values closer to 0 represent better fitting models. What's the ...
- 101
0
votes
1
answer
19
views
Adding another 'hue' to a pairplot
I have plotted a pairplot in Seaborn with a hue, similar to the one shown below. I would like to add another hue by changing the shape of the markers based on another categorical feature. E.g., the ...
- 1
0
votes
1
answer
14
views
Extract the embedding from a specific layer of MarianModel
I am using using MarianModel from the hub of HuggingFace for a translation task. Now I want to extract the embedding from the output of the last MarianEncoderLayer ...
- 231
0
votes
0
answers
9
views
How to calculate total observation time per focal individual across entire observation period?
I have a large amount of data for a set focal individuals that were under observation on a daily basis over three years. I am trying to calculate total times visible doing an activity and not oos (out ...
0
votes
1
answer
21
views
Transform dataset to regression problem by sorting?
I have a raw unlabeled dataset, and I want to design a model to perform a regression. In my dataset, it does not make sense to give each observation a value, but it does make sense to sort them. Can I ...
0
votes
0
answers
3
views
using xmgrace in batch mode
I am using xmgrace to plot 2D bar plot from the following input data:
...
1
vote
1
answer
24
views
Combine datasets of different domains to ehance generalizibility
so I try to implement an Emotion Classifier, which should detect several emotions from a text. There are several datasets for this (ISear, GoEmotions, etc.). However, a lot of them come from different ...
- 13
1
vote
1
answer
24
views
Nearest Neighbor Recommendation System w/ categorical variables
I would like to build a recommendation system:
no ratings are available at the time of recommendation, therefore only a purely context-based recommendation system is needed
as input features answers ...
- 31
0
votes
1
answer
16
views
Likert Scale Target Variable
I have a case study where the target variable (a single factor) gauged through multiple items. the items are measured using 5-Likert scale (Never, Seldom, Sometimes, Often, Very often, Always)
since ...
0
votes
0
answers
33
views
clustering customer base purchase behavior
I have a set of data and I want to know that whether they are necessary to add in the clustering analysis. Like ONEOFF_PURCHASES_FREQUENCY, I am not sure it is wether helpful in doing cluster analysis....
- 1
0
votes
0
answers
20
views
Why am I seeing these spikes in model loss curve
I am training an image classifier on 1152 images in 4 classes , I have used data augmentation too .
...
0
votes
1
answer
24
views
Feature scaling in Linear Regression
I always use Linearregression() class in sklearn library for creating a linear regression model. According to my understanding, we need feature scaling in linear ...
- 23
0
votes
0
answers
21
views
How can create deliberately biased models?
I deal with an image classification problem with 3-class. I want to create a model which takes side to one specific class. I mean, while the model predicts a sample, if it is hesitant between class-1 ...
- 101
3
votes
3
answers
437
views
How to train deep learning model on high dimensional dataset with limited memory and disk
For large datasets in terms of rows, usually it is handled by splitting data into pieces and feeding them into the model one at a time using tf.datasets or custom generator.
However what if number of ...
- 385
0
votes
0
answers
13
views
pix2pix TensorFlow Tutorials
I am trying to follow this tutorial, but I want to use my own dataset. The problem I am having is that in this tutorial they merge the realimage and the label image together in to one image( which ...
- 1
0
votes
0
answers
16
views
is it possible to turn a list of sentences into paragraph?
I have a problem and seeks advise, I have a couple of sentences like:
...
- 1
0
votes
0
answers
6
views
What is the space and runtime complexity of sklearn MiniBatchNMF
I am trying to scale my virtual machine instance to be the necessary size but not waste extra space. What is the space and runtime complexity for sklearn MiniBatchNMF?
- 1
2
votes
1
answer
29
views
Why is it an advantage "that Markov chains are never needed" to obtain gradients?
In the original GAN (Generative Adversarial Network) paper, Generative adversarial networks by I. Goodfellow, J. Pouget-Abadie, M. Mirza et. al. they state an advantage of the GAN is "that Markov ...
- 21
0
votes
0
answers
18
views
Border values at which timeseries decrease and increase
I have a timeseries data of signals in stock market ([-1, 1]) and I want to find mean values at which I have down trend and upwards trend.
I already used Moving ...
- 31
0
votes
1
answer
22
views
How do I know that my weights optimizer have found the best weights?
I am new to deep learning and my understanding of how optimizers work might be slightly off. Also, sorry for a third-grader quality of images.
For example if we have simple task our loss to weight ...
- 3
1
vote
1
answer
23
views
How do you train an ML algorithm to achieve a desirable clustering?
Most clustering examples on the net are unsupervised learning.
There is a given vectorization into a 2D space and the algorithm discovers clusters.
However, what if the input data that I want it to ...
- 117
0
votes
1
answer
19
views
Testing the impact of events on time series
Context
I am working with product data for a retail company. I have the daily impressions (number of times it was viewed online) for all products over a 30 day period (can get more data).
Here is the ...
- 3
0
votes
0
answers
29
views
What is the best practice for combining cross-validation with hyperparameter tuning and comparing preprocessing methods
The Goal Compare several preprocessing methods and models - while tuning hyperparameters for each model - without leaking information into the final generalization estimate, applying cross-validation (...
- 1
0
votes
1
answer
49
views
Binary classification performance difference between 0 and 1 class
I have trained a binary Random Forest classifier on a dataset containing 7M rows. I also set aside a holdout validation set of 1M rows that the training pipeline never sees. The dataset consists of ...
- 13
0
votes
0
answers
9
views
Embeded Google Maps map has delay on Oracle Analytics Cloud
I've been using Oracle Analytics Cloud for a while and wondered how to embed the Google Maps map (Dynamic Javascript Map) instead of using the default map. I finally made it work (using an API key ...
1
vote
1
answer
17
views
Newbie questions: real-time clustering of messages
I'm very much a newbie in NLP, so please accept my apologies if this is an obvious question, the wrong place to ask it or any other error I could be making.
I am considering using NLP for some subset ...
- 111