Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
1answer
264 views

Problems with class embedding in keras

I am doing a toy example with mushroom dataset to learn class embedding with keras: I am trying to embed a single feature: ...
0
votes
1answer
14 views

Is there a way to perform edit distance (Levenshtein) character by character between two string columns? [closed]

I have two datasets: dataset1 & dataset2, which have a common column called SAX which is a string object. ...
1
vote
0answers
17 views

Efficient Decision Tree Pruning

Is there an efficient way to handle pruning in Decision Tree with Python ? Currently I'm doing that: ...
-2
votes
0answers
10 views

Improve Accuracy in deep neural network with Keras [closed]

Below is the code of what I'm trying to do, but my accuracy is always under 50% so I'm wondering how should I fix this? The data I used here is provided in this link https://drive.google.com/file/d/...
1
vote
3answers
615 views

Custom Named Entity Recognition using deep learning

I have a dataset with two columns. First column has some text (news article) and the second column contains names of people (not exactly English names) present in those news articles (first column). I'...
0
votes
1answer
17 views

Shaping columns (and column headers) into multi-index rows using pivot/pivot_table

Problem I am trying to shape a dataset which includes stacked dates/data as rows and half-hourly data as columns (Trading periods, eg. TP1, TP2), into a chronological half-hourly order with the ...
0
votes
1answer
2k views

Dimension Mismatch Error during dot product in Python

I have two matrices user_vecs and item_vecs I am trying to take the dot product of the two to build a recommendation engine: ...
1
vote
2answers
73 views

Train MLP Neural Network on time series data?

Newbie question here but I was curious to ask if an MLP Neural type network can be trained on time series data? The dataset that I have is an electricity type data set from a building power meter and ...
0
votes
1answer
43 views

Model Dump Parser (like XGBFI) for LightGBM and CatBoost

Currently my employer has multiple GLM in a live environment. I am interested in identifying new features and interactions to enhance the accuracy of these GLM; for now I am limited to the GLM ...
0
votes
0answers
8 views

Python TimeSeries Error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value

I'm creating a time series model in Python. I converted a int field that counted the days (1,2,3,4) to a datetime field using the following code: ...
0
votes
0answers
20 views

Using variance Threshold removes all my features (clustering)

I have 100+ features for clustering. I am unfamiliar with unsupervised tehcniques for clustering. I have heard of variance threshold. However the features i have used have low variance.. e.g. some ...
-1
votes
0answers
8 views

Conversion of a YAML file to XLSX using Python [migrated]

I want to convert a set of yaml files in a folder into an xlsx file. I thought I'd start with trying to convert one yaml file into an xlsx file. The yaml files in the folder are all in the format ...
-1
votes
0answers
14 views

How to predict multiple related values with one value [closed]

first of all i have to say that i am a complete beginner in machine learning. For an architecture project i would like to predict multiple values with only one Value. For example: I would like to ...
0
votes
0answers
14 views

BinaryEncoder on test set causes different feature count than on training set

I am using a BinaryEncoder from the category_encoders package. I used it to encode a feature in my training set: 30k samples, 1170 unique categories for the feature in question, that resulted in 12 ...
1
vote
1answer
508 views

What are the possible values of a filter in a CNN?

I am a trying to write a CNN from scratch in python but I am bit new to CNNs specifically the convolution layers as I am comfortable with the dense layers. I was reading Do filters have different ...
0
votes
1answer
161 views

How fit_transform, transform and TfidfVectorizer works

I'm a machine learning beginner and I tried to use the cosine similarity on fuzzy matching purpose. In the following example I want to compare 'data_dirty' with 'data_clean' : When I have to ...
0
votes
1answer
281 views

How to refit GridSearchCV on Multiclass problem

I'm trying to use GridSearchCV for my Multiclass problem. For starters, wanted to test it on KNeighborsClassifier. First, here's the code where I define the function which uses GridSearchCV: ...
6
votes
1answer
902 views

Why Scikit and statsmodel provide different Coefficient of determination?

First of all, I know there is a similar question, however, I didn't find it so much helpful. My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can ...
-1
votes
1answer
11 views
2
votes
1answer
55 views

Can I forecast with discontinued data using ARIMA?

I have data for sales on monthly basis, but a few months' information is not in the CSV file or data file. Can I forecast or fill that missing month with other calculated values from present records? ...
0
votes
2answers
35 views

Don't know how to preprocess my dataset for image classification

I'm trying to do image classification using CNN. The exact model isn't important but I decided to try use AlexNet and I'm getting abysmal accuracy. I believe the issue might be with my data ...
1
vote
3answers
16k views

Is there a way to replace existing values with NaN

I'm experimenting with the algorithms in iPython Notebooks and would like to know if I can replace the existing values in a dataset with Nan (about 50% or more) at random positions with each column ...
0
votes
2answers
62 views

Keras fit_generator() for long signals

I want to make an LSTM network and I have quite a long signal that I want to use as my training data My X_train is a CSV-file which contains 12 signals with a length of 54 837 488 My y_train is an ...
2
votes
2answers
51 views

How to make a neural network generalizes better?

I designed a neural network model with large number of output predicted by softmax function. However, I want categorize all the outputs into 5 outputs without modifying the architecture of other ...
4
votes
1answer
317 views

How can I detect partially obscured objects using Python?

I'm building a computer vision application using Python (OpenCV, keras-retinanet, tensorflow) which requires detecting an object and then counting how many objects are behind that front object. So, ...
30
votes
5answers
41k views

Is it necessary to standardize your data before clustering?

Is it necessary to standardize your data before cluster? In the example from scikit learn about DBSCAN, here they do this in the line: ...
2
votes
2answers
73 views

Many separation line using RBF kernel in SVM

Below is my code, it take a range of a number, creates a new column label that contains either -1 or 1. In case the number is ...
1
vote
1answer
12 views

Naives Bayes Text Classifier Confidence Score

I am experimenting with building a text classifier using Naive Bayes which has been pretty successful on my test data. One thing i am looking to incorporate is handling text that does not fit into any ...
2
votes
2answers
930 views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
2
votes
2answers
4k views

Custom loss function which is included gradient in Keras

I want to make a custom loss function. Concretely, I use a 2D Convolutional neural network in Keras. So far, I've made various custom loss function by adding to losses.py. However, in this case, I ...
2
votes
1answer
104 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
0
votes
0answers
15 views

How can i decide on which features to use for clustering?

I am clustering on a dataset where each row is a customer and each column is a feature. I have 200 features, this seems like alot for clustering. I plan to experiment with a variety of clustering ...
0
votes
0answers
15 views

How to classify all words in a sentence with a context?

I have the names of the companies (in Russian). The name can contain abbreviations, words with capital letters, words with lowercase letters, and mixed words. The model is trained according to the ...
0
votes
1answer
44 views

my model is predicted new images wrong?

I used a CNN network unet for a segmentation task this is the architecture I used ...
1
vote
2answers
22 views

Perform clustering from a similarity matrix

I have a list of songs for each of which I have extracted a feature vector. I calculated a similarity score between each vector and stored this in a similarity matrix. I would like to cluster the ...
0
votes
0answers
15 views

How to filter data samples which do not improve classifier?

I have a text dataset with noisy labels and an unbalanced shape. There are various ways to find features which do not drive improvement in some metric, and help to prune those from the pipeline. I ...
0
votes
1answer
55 views

How to extract sub sentences from sentence mentioning a particular subject?

I am trying to solve an NLP problem. For a given sentence like : "The Pasta was delicious, the Pizza was average" I want to extract the sentiment attached to food items. Having built my own NER ...
0
votes
0answers
27 views

Interpretation of Loss and validation Loss in Keras

I am building a model to predict one label by taking one feature as an input. The two variables seems to be strongly correlated. I wanted to build a (sequential) Neural Network model with Keras in ...
0
votes
1answer
13 views

I'm trying to do a time series model without a datetime field in python. Is this possible?

I have a dataset with data like this: Day Revenue 1 1.2 2 1.5 3 1.1 4 1.34 I want to do a time series model on it, but am ...
1
vote
0answers
18 views

How to get KNN linearly hybridised by two similarities?

I'm writing a KNN (collaborative filtering) hybrid similarity recommender and I need some advice. It is based on this paper. I've currently got 2 datasets. The first one is ...
0
votes
1answer
23 views

How to find out what portions of an image is helping CNN to classify it

I am working on an image classification problem using Transfer learning. Right now, I am getting an accuracy of 75% on train data and 67% in test data. Now I want to understand what portions/parts of ...
3
votes
2answers
159 views

How to perform feature selection on dataset with categorical and numerical features?

I am working on a dataset with 30 columns (29 numerical, 1 non-ordinal categorical). I hot-encoded the categorical feature and reached at 35 columns. To improve training efficiency, I want to perform ...
-1
votes
1answer
1k views

F1_score(average='micro') is equal to calculating accuracy for multiclasification

Is f1_score(average='micro') always the same as calculating the accuracy. Or it is just in this case? I have tried with different values and they gave the same answer but I don't have the analytical ...
0
votes
3answers
74 views

K-Nearest Neighbor Classifier Best K Value

I created a KNeighborsClassifier for my dataset adjusting the k hyper-parameter (the number of neighbors) in a for loop. The k ...
29
votes
3answers
16k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
0
votes
0answers
7 views

How can interparet shap.summary_plot and its gray color concerning outliers/anomaly?

I inspired by this notebook, and I'm experimenting IsolationForest algorithm using scikit-learn==0.22.2.post1 for anomaly ...
-1
votes
0answers
14 views

Design of loss function for unsupervised learning neural network

Input is a string: get filename (e.x., get file.txt get hello.txt) output: predict a number between 1 to 15 Property need to be translated into loss function: define in the loss function that, even if ...
-1
votes
0answers
10 views

Grid Search returns only NaNs. Is it my data? Is it my application? [closed]

I am attempting to optimize parameters using GridSearchCV . However, when I do so the gridsearch appears to run, then returns all NaN values. I read somewhere that this means the model is failing to ...
1
vote
2answers
47 views

Determining threshold in an area with very few samples of positive label

I have a binary classification task where I want to either keep or discard samples. I have about a million samples, and about 1% should be kept. I want to discard as much as possible, but discarding ...

1
2 3 4 5
105