Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).
5,248
questions
0
votes
1answer
264 views
Problems with class embedding in keras
I am doing a toy example with mushroom dataset to learn class embedding with keras:
I am trying to embed a single feature:
...
0
votes
1answer
14 views
Is there a way to perform edit distance (Levenshtein) character by character between two string columns? [closed]
I have two datasets: dataset1 & dataset2, which have a common column called SAX which is a string object.
...
1
vote
0answers
17 views
Efficient Decision Tree Pruning
Is there an efficient way to handle pruning in Decision Tree with Python ?
Currently I'm doing that:
...
-2
votes
0answers
10 views
Improve Accuracy in deep neural network with Keras [closed]
Below is the code of what I'm trying to do, but my accuracy is always under 50% so I'm wondering how should I fix this?
The data I used here is provided in this link
https://drive.google.com/file/d/...
1
vote
3answers
615 views
Custom Named Entity Recognition using deep learning
I have a dataset with two columns. First column has some text (news article) and the second column contains names of people (not exactly English names) present in those news articles (first column). I'...
0
votes
1answer
17 views
Shaping columns (and column headers) into multi-index rows using pivot/pivot_table
Problem
I am trying to shape a dataset which includes stacked dates/data as rows and half-hourly data as columns (Trading periods, eg. TP1, TP2), into a chronological half-hourly order with the ...
0
votes
1answer
2k views
Dimension Mismatch Error during dot product in Python
I have two matrices user_vecs and item_vecs
I am trying to take the dot product of the two to build a recommendation engine:
...
1
vote
2answers
73 views
Train MLP Neural Network on time series data?
Newbie question here but I was curious to ask if an MLP Neural type network can be trained on time series data?
The dataset that I have is an electricity type data set from a building power meter and ...
0
votes
1answer
43 views
Model Dump Parser (like XGBFI) for LightGBM and CatBoost
Currently my employer has multiple GLM in a live environment. I am interested in identifying new features and interactions to enhance the accuracy of these GLM; for now I am limited to the GLM ...
0
votes
0answers
8 views
Python TimeSeries Error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value
I'm creating a time series model in Python. I converted a int field that counted the days (1,2,3,4) to a datetime field using the following code:
...
0
votes
0answers
20 views
Using variance Threshold removes all my features (clustering)
I have 100+ features for clustering. I am unfamiliar with unsupervised tehcniques for clustering. I have heard of variance threshold. However the features i have used have low variance.. e.g. some ...
-1
votes
0answers
8 views
Conversion of a YAML file to XLSX using Python [migrated]
I want to convert a set of yaml files in a folder into an xlsx file. I thought I'd start with trying to convert one yaml file into an xlsx file. The yaml files in the folder are all in the format ...
-1
votes
0answers
14 views
How to predict multiple related values with one value [closed]
first of all i have to say that i am a complete beginner in machine learning. For an architecture project i would like to predict multiple values with only one Value. For example: I would like to ...
0
votes
0answers
14 views
BinaryEncoder on test set causes different feature count than on training set
I am using a BinaryEncoder from the category_encoders package. I used it to encode a feature in my training set: 30k samples, 1170 unique categories for the feature in question, that resulted in 12 ...
1
vote
1answer
508 views
What are the possible values of a filter in a CNN?
I am a trying to write a CNN from scratch in python but I am bit new to CNNs specifically the convolution layers as I am comfortable with the dense layers. I was reading Do filters have different ...
0
votes
1answer
161 views
How fit_transform, transform and TfidfVectorizer works
I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' :
When I have to ...
0
votes
1answer
281 views
How to refit GridSearchCV on Multiclass problem
I'm trying to use GridSearchCV for my Multiclass problem. For starters, wanted to test it on KNeighborsClassifier.
First, here's the code where I define the function which uses GridSearchCV:
...
6
votes
1answer
902 views
Why Scikit and statsmodel provide different Coefficient of determination?
First of all, I know there is a similar question, however, I didn't find it so much helpful.
My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can ...
-1
votes
1answer
10 views
How can I resolve the error “ValueError: shapes (300,50) and (31,) not aligned: 50 (dim 1) != 31 (dim 0)” that I get when I predict the test data?
When I try to predict the test data for submission I get the following value error:
...
-1
votes
1answer
11 views
How can I convert a dataframe column of fraction float that is stored as a string into float? [closed]
The df is as follows:
...
2
votes
1answer
55 views
Can I forecast with discontinued data using ARIMA?
I have data for sales on monthly basis, but a few months' information is not in the CSV file or data file. Can I forecast or fill that missing month with other calculated values from present records?
...
0
votes
2answers
35 views
Don't know how to preprocess my dataset for image classification
I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy.
I believe the issue might be with my data ...
1
vote
3answers
16k views
Is there a way to replace existing values with NaN
I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column ...
0
votes
2answers
62 views
Keras fit_generator() for long signals
I want to make an LSTM network and I have quite a long signal that I want to use as my training data
My X_train is a CSV-file which contains 12 signals with a length of 54 837 488
My y_train is an ...
2
votes
2answers
51 views
How to make a neural network generalizes better?
I designed a neural network model with large number of output predicted by softmax function. However, I want categorize all the outputs into 5 outputs without modifying the architecture of other ...
4
votes
1answer
317 views
How can I detect partially obscured objects using Python?
I'm building a computer vision application using Python (OpenCV, keras-retinanet, tensorflow) which requires detecting an object and then counting how many objects are behind that front object. So, ...
30
votes
5answers
41k views
Is it necessary to standardize your data before clustering?
Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line:
...
2
votes
2answers
73 views
Many separation line using RBF kernel in SVM
Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1.
In case the number is ...
1
vote
1answer
12 views
Naives Bayes Text Classifier Confidence Score
I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any ...
2
votes
2answers
930 views
How can I find anomalies in each row of data?
I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility.
...
2
votes
2answers
4k views
Custom loss function which is included gradient in Keras
I want to make a custom loss function.
Concretely, I use a 2D Convolutional neural network in Keras.
So far, I've made various custom loss function by adding to losses.py.
However, in this case, I ...
2
votes
1answer
104 views
How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline
Let us say we have a dataset with a feature such as Surname.
arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith']
And I want to encode this categorical info ...
0
votes
0answers
15 views
How can i decide on which features to use for clustering?
I am clustering on a dataset where each row is a customer and each column is a feature. I have 200 features, this seems like alot for clustering. I plan to experiment with a variety of clustering ...
0
votes
0answers
15 views
How to classify all words in a sentence with a context?
I have the names of the companies (in Russian). The name can contain abbreviations, words with capital letters, words with lowercase letters, and mixed words. The model is trained according to the ...
0
votes
1answer
44 views
my model is predicted new images wrong?
I used a CNN network unet for a segmentation task
this is the architecture I used
...
1
vote
2answers
22 views
Perform clustering from a similarity matrix
I have a list of songs for each of which I have extracted a feature vector. I calculated a similarity score between each vector and stored this in a similarity matrix. I would like to cluster the ...
0
votes
0answers
15 views
How to filter data samples which do not improve classifier?
I have a text dataset with noisy labels and an unbalanced shape.
There are various ways to find features which do not drive improvement in some metric, and help to prune those from the pipeline.
I ...
0
votes
1answer
55 views
How to extract sub sentences from sentence mentioning a particular subject?
I am trying to solve an NLP problem. For a given sentence like :
"The Pasta was delicious, the Pizza was average"
I want to extract the sentiment attached to food items. Having built my own NER ...
0
votes
0answers
27 views
Interpretation of Loss and validation Loss in Keras
I am building a model to predict one label by taking one feature as an input. The two variables seems to be strongly correlated. I wanted to build a (sequential) Neural Network model with Keras in ...
0
votes
1answer
13 views
I'm trying to do a time series model without a datetime field in python. Is this possible?
I have a dataset with data like this:
Day Revenue
1 1.2
2 1.5
3 1.1
4 1.34
I want to do a time series model on it, but am ...
1
vote
0answers
18 views
How to get KNN linearly hybridised by two similarities?
I'm writing a KNN (collaborative filtering) hybrid similarity recommender and I need some advice. It is based on this paper.
I've currently got 2 datasets. The first one is ...
0
votes
1answer
23 views
How to find out what portions of an image is helping CNN to classify it
I am working on an image classification problem using Transfer learning. Right now, I am getting an accuracy of 75% on train data and 67% in test data. Now I want to understand what portions/parts of ...
3
votes
2answers
159 views
How to perform feature selection on dataset with categorical and numerical features?
I am working on a dataset with 30 columns (29 numerical, 1 non-ordinal categorical). I hot-encoded the categorical feature and reached at 35 columns. To improve training efficiency, I want to perform ...
-1
votes
1answer
1k views
F1_score(average='micro') is equal to calculating accuracy for multiclasification
Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case?
I have tried with different values and they gave the same answer but I don't have the analytical ...
0
votes
3answers
74 views
K-Nearest Neighbor Classifier Best K Value
I created a KNeighborsClassifier for my dataset adjusting the k hyper-parameter (the number of neighbors) in a for loop. The k ...
29
votes
3answers
16k views
Difference between OrdinalEncoder and LabelEncoder
I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing:
In the Documentation it is given about ...
0
votes
0answers
7 views
How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?
I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
-1
votes
0answers
14 views
Design of loss function for unsupervised learning neural network
Input is a string: get filename (e.x., get file.txt get hello.txt)
output: predict a number between 1 to 15
Property need to be translated into loss function: define in the loss function that, even if ...
-1
votes
0answers
10 views
Grid Search returns only NaNs. Is it my data? Is it my application? [closed]
I am attempting to optimize parameters using GridSearchCV . However, when I do so the gridsearch appears to run, then returns all NaN values. I read somewhere that this means the model is failing to ...
1
vote
2answers
47 views
Determining threshold in an area with very few samples of positive label
I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...
