Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Reasons that LIME and SHAP might not agree with intuition

I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making ...
1
vote
0answers
9 views

Some of the p-values are NaN - logistic regression

I am trying to do logisitc regression, but have this issue - some of the p values are NaN ...
-1
votes
0answers
16 views

Renaming Columns [closed]

I want to create a new dataframe based upon an original dataframe. I am not really sure how to approach this and which is why the consultation. I want to take the name of one element in the string ...
0
votes
0answers
18 views

Learning Process in Machine Learning

I want to use Unsupervised Leaning to detect Anomalies within a huge Csv file (consisting of headers that are named and thousands of rows belonging to them) Here on this link I have read about the ...
1
vote
0answers
18 views

Alignment of time series

I have two time series of different lengths: ...
0
votes
1answer
14 views

Estimating 3D function - f(x,y) according to its minimums

Problem Statement: I was given 3 minimums of Y = f(X1, X2) such that: Local Minimum1: X1 = 0.20; X2 = 0.30; Ymin1 = 0.70 Local Minimum2: X1 = 0.60; X2 = 0.80; Ymin2 = 0.80 Global Minimum:...
1
vote
1answer
25 views

Comparing excel data sets in Pandas

Pretty new to Python, but as an SEO I'm looking at the benefits of using notebooks in my workflow. I've got two excel files which I've cleaned and imported into a new notebook using pandas. I'm ...
1
vote
1answer
27 views

Using TSNE to Visualize Clusters in Python

I'm using TSNE to visualize my clusters but the output seems a bit strange. There are supposed to be 3 clusters but instead, there are 4 lines. Is there something wrong with how I'm visualizing them ...
0
votes
0answers
11 views

How to create POS Embeddings for input?

I am trying to append POS embeddings to an input word embedding (which is from Glove). In order to do that, I need represent POS tags as embeddings. I saw the link in this channel already; however I ...
2
votes
0answers
8 views

Using an unmerged branch from StatsModels for Anaconda

I am interested in using a statistical test from StatsModels, one that is not available in the default Anaconda StatsModels folder on my drive. See this blog, which as some point mentions, almost ...
1
vote
0answers
19 views

Filter and plot values using matplotlib

In order to understand data a bit more, I want to plot one column from my csv file using matplotlib as a dotted graph This is how my my csv file looks like. I want to filter the csv with ...
1
vote
0answers
8 views

Scipy.optimize getting back x0 for optimization of input to recurrent neural network

I need help with this optimization problem which is either not getting solved at all or is taking a copious amount of time. I am trying to find optimized input to an RNN (GRU) model of a process ...
0
votes
0answers
13 views

How to perform TTest on multiple coliumns

My dataframe is below ...
-1
votes
0answers
13 views

Keras - Preparing the Embedding layer with Word2Vec

I have a word2vec model. But I can't implement word2vec with keras embedding layer. In this tutorial, Preparing the Embedding layer section, they used GloVe. I want to use word2vec instead. How can I ...
-1
votes
0answers
20 views

How to update package in my Anaconda Environment? [closed]

I am trying to update numpy from 1.15 to 1.18. Using the command conda install -c anaconda numpy did not help as the only thing that was updated are the 'ca-...
0
votes
0answers
24 views

How to fill missing value based on other column in Pandas dataframe? [closed]

For the below sample dataframe, I would like to map the respective state with zone and fill na's with the respective zone names. How can we code for the below instead manually filling missing values? ...
1
vote
1answer
19 views

Forecasting - 3 independent variables

I have 3 independent input variables: time, temperature and pressure, and one output - usefulness. Example: Number of the sample; Time; Temperature; Pressure; Usefulness 1;1;30;50;9% 2;3;50;50;80% ...
2
votes
1answer
23 views

Trying to return more than just the top result from sklearn NearestNeighbors

I'm trying to compare a list of names (duplicated into a clean file and a messy file). I then compare the files against each other. My problem is that it returns only the top 1 result for each, ...
2
votes
1answer
40 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
2
votes
1answer
31 views

How to validate regex based Resume parser efficiently

I am using rule based logic to extract features from resume. Basically I am trying to find if the candidate switched the company in less than 1 year. So I have the code in place to find it using ...
1
vote
0answers
31 views

Can anyone direct me on how to got to [closed]

I am a newbie to python. I have uploaded about 2000 images into my google drive, assigned a variable to all content in the folder and tried to split the folder's content into training and testing, ...
1
vote
1answer
33 views

Feature extraction from Resume using Python without rule based logic

I am working on Resume parser project. Currently I am using rule based regex to extract features like University, Experience, Large Companies etc. So Basically I have a set of Universities name in a ...
1
vote
0answers
26 views

Need help regarding my thesis project related to Data Analysis [closed]

I am working on a project with a company that manufactures Injection molding machines. I am to perform data analysis on some selected parameters as a part of my thesis to tell the company if from ...
2
votes
1answer
28 views

Why does MAE differ after prediction (Neural Network)?

I'm having trouble understanding what's happening in the following code. I already have defined x_train, y_train, x_val, y_val and x_test which define my training, validation and test sets. I'm using ...
2
votes
1answer
53 views

Is it possible to make F1_Score differentiable and use it directly as a Loss function?

One of the metrics that is widely used in binary classification is the F1 score: $F_1 = 2\cdot \frac{recall \cdot precision}{recall+precision}$ The problem of the F1-score is that it is not ...
5
votes
3answers
78 views
+100

How can we extract fields from images?

I am making an document parser which extracts data fields from the documents and store them in a structured way. Each field in my dataset is horizontal which is easy to extract. But the model fails ...
1
vote
1answer
18 views

multiclass classification

I want to build an ml model, which can when given a text input, can predict predefined tags or labels for the text. I already built one such model, but the problem with that is that it only predicts ...
1
vote
0answers
19 views

How to classify unseen text data?

I am training an text classifier for addresses such that if given sentence is an address or not. ...
1
vote
1answer
18 views

Passing the output of a model as input to an other model , computer vision?

i want to try my hands at a project where i have to detect the brands + the models of sportswears worn by persons participating to a marathon , the project is quite complex , so my first approach ...
3
votes
1answer
30 views

What is a good method for detecting local minims and maxims?

I'm using kernel density estimation in order to compute probability density function for item (triangles in the figure) occurrence. Using this output, i want to find all the local minims and maxims. ...
2
votes
1answer
35 views

Machine learning model with simultaneous function optimization

Consider the following scenario. I am a sculpturer and customers ask me for what price I am willing to provide them with some statues. Their request for sculptures can vary in difficulty, quantity, ...
1
vote
0answers
19 views

Organize the Dataset

I am working on a neural network(Variational Autoencoder) to separate music instruments I don't know how to organize my data I have a folder full of spectrograms of the songs as a mix and the ...
1
vote
0answers
13 views

How to tune the hyperparameters of XGBoost and RF? [closed]

How to tune the hyperparameters of XGBoost and RF in python?
2
votes
0answers
22 views

Parameter optimization and selection in dynamic neural networks

I have used a Bayesian optimization to tune machine learning parameters. The optimized parameters are "Hidden layer size" and "...
1
vote
2answers
38 views

Predict the average temperature for next 30 years

Objective: I want to predict the average temperature for next 30 years. Q1: What type of dataset is suitable for this (what columns should it contain) Q2: What are the independent variables for ...
1
vote
1answer
28 views

Implementing “full convolution” to find gradient w.r.t the convolution layer inputs

I've been trying to implement "full convolution" w.r.t to convolution layer inputs. According to this article, it looks like this: So, I wrote this function: ...
0
votes
0answers
26 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
1
vote
1answer
19 views

(pre-trained) python package for semantic word similarity

I am searching for a python package that calculates the semantic similarity between words. I do not want to train a model (what most packages seem to offer) - the package should have been pre-trained ...
3
votes
0answers
28 views

Sources to learn Anomaly Detection

Project Goal- Create an Anomaly Detection tool. As the first step I worked on some python code that would parse my XML files into a Dataframe in csv file. The next step is to go into machine ...
-2
votes
0answers
9 views

Looping over lists [closed]

I have list values like this ['01000000111011100010001100101001', '01000110111010010010011000101100', '01110100000011101100111000101000', '01110101000010001111011010000111'] I want to count ...
0
votes
0answers
18 views

detect table of content in PDFs

I have PDF text data that I want to use for NLP tasks. Now during preprocessing, I came to the conclusion that it is not worth analyzing the table of content. It merely generates a bunch of ...
1
vote
1answer
28 views

PCA Regression Problem

I have a regression problem whereby my data has 21 features and I wish to apply dimensionality reduction using PCA. As far as I know, all the tutorials I have seen so far use PCA for classification ...
0
votes
0answers
28 views

Power BI or Dash? [closed]

I'd like to know opinions regarding Dash (python) and Power BI. For context, I am currently working on producing several dashboards and I am not sure which one I should work with. I've never used ...
3
votes
0answers
19 views

Improving a simple trig model

I have some data which I know is well approximated as a trig function, and I can fit it with scipy.optimize.curve_fit as follows: ...
3
votes
1answer
54 views

Best anomaly detection algorithm based on two conditions

Choosing the right anomaly detection algorithm seems quite hard at the moment. It might be because I am bombarded with so many alternatives likes clustering, K-Means DBSCAN and so many others. On my ...
3
votes
1answer
28 views

How to properly store a list of pandas dataframes, so that I can import them in a new python script?

I'm trying to make a GUI in PyQt5 to present the results of my research. The results are around 800 pandas dataframes, stored in lists (so that I can call them individually according to the list index)...
2
votes
1answer
25 views

Ridge regression model creation using grid-search and cross validation

I created python code for ridge regression.For that I used cross validation and grid-search technique in together. i got output result. I want check whether my regression model building steps correct ...
1
vote
0answers
24 views

What is the formula to calculate the precision, recall, f-measure with macro, micro, none for multi-label classification in sklearn metrics?

I am working in the problem of multi-label classification tasks. But I would not able to understand the formula for calculating the precision, recall, and f-measure with macro, micro, and none. ...
2
votes
1answer
27 views

Multiclassification Error: NotFittedError: This MultiLabelBinarizer instance is not fitted yet

After picking the model, when I try to use it, I am getting error - "NotFittedError: This MultiLabelBinarizer instance is not fitted yet. Call 'fit' with appropriate arguments before using this ...
3
votes
4answers
75 views

Machine Learning methods for finding outliers

I have a csv files of thousands of lines, the data is put down into a Dataframe of columns. Some of the column have text information while others might have numbers. I want to detect anomalies or ...