All Questions

Filter by
Sorted by
Tagged with
11
votes
5answers
25k views

How to scrape imdb webpage?

I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage. I am using BeautifulSoup module. Following is the code I am ...
0
votes
2answers
16 views

How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python)

I have some data which contains anomalies as well. I want to model data using Gaussian distribution assuming variables are dependent in Python. How can I model this? Should I use the PDF formula as ...
0
votes
0answers
2 views

Understanding Learning Curves

I would like to clarify my understanding of learning curves with two example plots below. I am experimenting with small data sets here between 500 and 1500 samples to clarify my understanding. My ...
1
vote
1answer
11 views

Keras 1D CNN always predicts the same result even if accuracy is high on training set

The validation accuracy of my 1D CNN is stuck on 0.5 and that's because I'm always getting the same prediction out of a balanced data set. At the same time my training accuracy keeps increasing and ...
0
votes
1answer
22 views

How to pad a batch of documents?

Hello PyTorch experts: Sentences and documents both can be variable length. Lets say, we have following 2 docs: ...
2
votes
1answer
39 views

Contextual Spell Correction

I want to create a spell checker that corrects the spelling mistakes contextually. For example, Erroneous sentence: I want to apply for credit cart Corrected sentence: I want to apply for credit ...
1
vote
0answers
2 views

Region Proposal Network - How to subsampling 256 fg/bg anchors

I am trying to understand training process of RPN. I have problem with creating mini batches of 256 anchors. If features map has shape 18x25=450 and every position has 9 anchors it is 4050 potential ...
9
votes
3answers
122k views

Dataframe has no column names. How to add a header?

I am using a dataset to practice for building a decision tree classifier. Here is my code: ...
1
vote
2answers
30 views

Which correlation test can be used on low count contigency table?

I need to find the correlation between the ratings given by two parties. Let's say we have 2 coaches, namely A and B. Coach A will rate the skills of the 12 players, either "Strong", "Moderate" or "...
0
votes
1answer
16 views

why do transformers mask at every layer instead of just at the input layer?

working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one ...
3
votes
2answers
1k views

Multidimensional scaling producing different results for different seeds

I took the data from here and wanted to play around with multidimensional scaling with this data. The data looks like this: In particular, I want to plot the cities in a 2D space, and see how much it ...
0
votes
0answers
6 views

Detect photos of boats which have the Europe's flag installed on it

My task is a binary classification: is there a boat in the picture that has the Europe flag installed on it? In other words, my classifier should first detect if there is a boat in the picture or not, ...
1
vote
0answers
3 views

What is the best way to deal with image sizes in object detection using YOLO v. 4?

I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios. Examples of ...
3
votes
0answers
11 views

At what stage are ROC curves used when building machine learning model?

When developing a machine learning model, at what stage are ROC curve with AUC used? Typically I have three data sets train - ...
2
votes
0answers
43 views

How is bayesian risk computed to prune decision trees?

I've been trying to follow this paper on Bayesian Risk Pruning. I'm not very familiar with this type of pruning, but I'm wondering a few things: (1) The paper describes risk-rates to be defined per ...
2
votes
0answers
8 views

Cannot understand feature extraction

I'm following an AI course and we've just entered the deep learning chapter. Speaking about the difference between classic machine learning models and deep learning, it turns out one of the most ...
0
votes
0answers
6 views

Remove correlated features before or after splitting test and training set?

I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it ...
2
votes
1answer
75 views

How can I “export” a model from Orange once I have prototyped a solution?

Using Orange, I would like to use the underlying python model within the control system of my device. The device is a simple machine which is not connected to the internet. My intent is to use the ...
0
votes
0answers
4 views

VAE: first hidden layer of encoder having more neurons than the input layer

Many blog posts explain the architecture of variational autoencoders (VAEs) with symmetrical encoder-decoder architecture where the number of neurons in the encoder's hidden layers decreases, starting ...
4
votes
2answers
112 views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
0
votes
2answers
42 views

What pandas function is most similar to the transpose procedure in SAS?

I am trying to transpose Amt by Txn_ID across Item_Desc in Python using pandas. Which pandas ...
1
vote
1answer
634 views

Out of Memory Error when Selecting Data from Redshift Table

I am selecting data from Amazon Redshift Table with 500 millions rows. I have 64bit python installed. code ...
0
votes
2answers
20 views

Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model

I am currently using transaction amount as a feature in an XGBoost classification model designed to identify fraudulent transactions. Furthermore, transaction amount is bounded for this problem ...
0
votes
0answers
8 views
0
votes
2answers
62 views

Metrics values are equal while training and testing a model

I'm working on a neural network model with python using Keras with TensorFlow backend. Dataset contains two sequences with a result which can be 1 or 0 and positives to negatives ratio in dataset is 1 ...
0
votes
0answers
10 views

Rolling window on uneven time series classification

I have a univariate time series data that I would like to take about 60 seconds of, extract features using tsfresh and classify into multiclass. So I might end up with a dataframe like: ...
-2
votes
0answers
14 views

Homework Assignment [closed]

Can you recommend any good books to read?
0
votes
0answers
14 views

kmeans cluster results for use as xgboost feature

I am curious if kmeans clusters can be used as xgboost features, along with the original features? Specifically for features X and labels Y can we: Split into X_train, y_train & X_test, y_test ...
0
votes
1answer
28 views

the library 'transformers' works also with older version of Tensorflow?

i am working with Tensorflow version 1.14 and i would like to use the bert embedding. In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/) but i ...
-1
votes
0answers
16 views

Import module in python script

I am using Orange visual programming on a Mac OS and would like to use a python script to import additional modules (like keras, tensorflow ...). However, I am receiving the error message: ...
0
votes
1answer
7 views

Spectral Networks and Deep Locally Connected Networks on Graphs

I’m reading the paper Spectral Networks and Deep Locally Connected Networks on Graphs and I’m having a hard time understanding the notation shown in the picture below (the scribbles are mine): ...
2
votes
1answer
35 views

libraries for multiple machine NN training?

As detailed here, the way to go to break NN training over multiple machines/threads, is decompose training data set on multiple chunks and send to each node, then sum results back in main node. There ...
0
votes
1answer
8 views

Negative Feedback to GradCam method

I use EfficientNetB0 for performing image classification with one of the class as "stone countertop table", the other class is "not stone countertop table". I use the reddest part ...
2
votes
1answer
186 views

Discriminator of a Conditional GAN with continuous labels

OK, let's say we have well-labeled images with non-discrete labels such as brightness or size or something and we want to generate images based on it. If it were done with a discrete label it could ...
0
votes
0answers
8 views

One-Class classification

I am currently using an one class classification svm and I am trying to boost the classification results by employing more than one svm-occ with varying gamma parameters and combine these decisions ...
1
vote
1answer
26 views

Splitting Temporal Data

I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in ...
2
votes
1answer
47 views

reinforcement learning: Decompose a policy gradient

I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f Couldn't figure out how the first equation becomes the second ...
1
vote
1answer
17 views

Creating features from raw accelerometer data

I have a dataset containing raw 3-axis accelerometer data collected from a users lower leg and I want to create a classification model (as simple as possible) that detects if the user is sat down or ...
0
votes
0answers
10 views

Prevent model from predicting the same examples for different inputs

I have a ANN model, that predicts a fixed length curve. The problem is, those curves are really similar to each other, for example these two: To compare those curves, I use RMSD between their points....
1
vote
0answers
13 views

Advice on using Recurrent Neural Networks for non-time series dataset

I'm testing different machine learning algorithms for predicting week-to-week fantasy football scores for individual players. For those who don't know, fantasy football is a game in which players pick ...
0
votes
0answers
24 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
1
vote
1answer
23 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
0
votes
0answers
10 views

What is explicit fitness sharing in NEAT?

I cannot understand what is explicit fitness sharing. Could someone explain it?
0
votes
1answer
13 views

Import image into Tensorflow model [closed]

I trained a simple Handwriting model with Tensorflow and MNIST: ...
0
votes
0answers
6 views

Modelling probability of switching of label from 0 to 1 in next time period for multivariate time series data

I have a data set consisting of unbalanced panel data, i.e. longitudinal multivariate measurements on multiple individuals. I want to estimate a probability that the individual will become 1 in next ...
0
votes
1answer
39 views

Merging three different customer segmentation systems into one

I have been given a task where I have three existing customer segmentation systems (rule based e.g. if customer spends X in Y amount of time AND whatever then put in top spender segment is one segment,...
0
votes
0answers
27 views

How to make spiky data more continuous

Plotted below is the column of a time series feature of my dataset. As you can see, there are spiky jumps that are not appropriate and result in poor performance for my deep learning model I am ...
1
vote
1answer
16 views

Merging 2 pandas dataframes on key with duplicates but do not want it to result in two lines

I am trying to merge a "roster" where there are supposed to be several positions with one person per office. However, I have come across two instances where there are two people in one ...
2
votes
2answers
39 views

Find repeating patterns in data

I have database of sequential events for multiple animals. The events are represented by integers so it looks something like: ...
6
votes
2answers
485 views

15 30 50 per page
1
2 3 4 5
538