Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
0 votes
1 answer
6k views

GridSearchCV() to fine tune outputs ValueError and FitFailedWarning

I would like to fine tune some parameters for my linear SVM. This is the code: ...
1 vote
1 answer
59 views

Why is this binary state format so much more efficient than a non-binary one?

I am teaching myself Deep Q Learning (and machine learning in general) using python and tensorflow to play Snake. My initial learning came mostly from this article. I have rewritten it in my own ...
0 votes
1 answer
2k views

Jupyter Notebook not importing pandas module

I am running Jupyter on a server on a virtual environment. I then tunnel my connection so I can access Jupyter on my browser. When I SSH into the server, I can use the Panda module in both Ipython ...
0 votes
1 answer
110 views

Text Classification misclassifying?

I am trying to solve a binary classification problem. My labels are abusive (1) and non-abusive (0). My dataset was imbalanced (more 1 than 0s) and I used oversampling of the minority label (i.e. 1) ...
0 votes
1 answer
19 views

Python Tensorflow - Predict human vs horses images always same value

I'm trying to follow a tutorial about Tensorflow in Python and computer vision. In this exercise I'm using a pre-trained model (InceptionV3) and some image augmentation about a datasets of humans vs ...
0 votes
2 answers
131 views

Test data missing values

I've been given a test and train data set, and I have null values in my test data set. What do I do with it?
0 votes
4 answers
95 views

How to order a python dataframe by adding the row values?

I have the following dataframe: M1 M2 M4 M5 N1 45 46 54 57 N2 32 36 29 56 N3 56 44 40 55 N4 57 43 42 54 How is it possible to order ...
0 votes
2 answers
414 views

Ways to increase recall in SVM

I am training an SVM on UCI's Bank Marketing Data Set, the bank additional-full.csv. As the data is skewed I am also interested in recall. I am getting accuracy of about 87.95% but my recall is around ...
0 votes
0 answers
21 views

Common cross-validation code: why does it work?

The following Python code is common practice when creating a folds column for multi-label stratified k-fold cross-validation: ...
1 vote
1 answer
847 views

Clustering on binary data

I am working on clustering on binary data which has 25 features, sample Feature 1 Feature 2 Feature 3 ...... Feature 25 1 1 0 0 011101 1 2 0 1 0 010011 0 3 1 0 1 101001 1 and I have used the ...
0 votes
1 answer
1k views

How to automatically save images from colab to gdrive in seperate folders whose name are as same as the labels of the respective images?

I have a total of 30k images. They are distributed among 32 classes. I want to save these images from google colab to my drive with folder names as their label name. Can it be possible by python code? ...
1 vote
1 answer
72 views

Recommendations based on other products seen

I am trying to develop a basic book recommender system to get in touch with the field and start learning methods and how to prepare the data. The Dataframe I am using is pretty plain, it has the ...
1 vote
2 answers
75 views

How to get different results running sklearn's MeanShift in a single program? (Python3)

I ran into a quirk with sklearn's MeanShift that I don't know how to get around. MeanShift doesn't predictably give the same results on every run, so I wanted to run it multiple times within one ...
0 votes
1 answer
343 views

How to add Error Bar to Matplotlib line plot?

I have the following dataset which I use to plot a line plot. The plot is obtained as the mean of values obtained from the data. I want to add error bars to this ...
1 vote
1 answer
20 views

Difficulty in Coding research papers

I've done a couple of beginner-level competitions on Kaggle, and I thought I should implement research papers in computer vision (YOLOv1) but I'm finding it difficult to code it. How should I proceed? ...
0 votes
0 answers
16 views

What is the steps to generate data from a plot and use them to have a predication using python?

I have a plot that represent a BH curve for magnatic material. The material have a behavior for each H value for two different temperature 25 C and 100 C. Figure 1. I need to extract the data for each ...
0 votes
2 answers
134 views

Can I do the below problem with ML?

My dataset has 3 fields ColumnX, ColumnA, Column B as below. Column X, Column_A , Column_B AA, 1 , KEYWORD1 KEYWORD2 KEYWORD3 AA, 2 , KEYWORD4 KEYWORD5 KEYWORD6 XX, 3 , ...
0 votes
0 answers
19 views

Regression with time series data

I want to predict temperature when time (datetime type, hourly data for five months) and humidity is given. Before starting in python, I created a regression model in excel. But instead of predicting ...
1 vote
1 answer
456 views

Generating synthetic data based off existing real data (in Python)

I am looking for an approach to generate synthetic data for anomaly detection. We have real data, but want to inject anomalies to battle-test the model (the real data is too limited for likely future ...
1 vote
1 answer
102 views

Predicting the next occurrence based on binary

I have no experience in statistics or machine learning. I have a True/False binary array describing occupation of open public spaces ...
0 votes
0 answers
30 views

How can I improve the inefficiency so the function actually completes?

How can I improve the efficiency of the following line: X_df=pd.concat([X_df,pd.concat([pd.concat([dyn_df,stat],axis=1),demo],axis=1)],axis=0) Which can be found ...
0 votes
1 answer
217 views

Tensorflow 2.11 error: AttributeError: module 'tensorflow._api.v2.compat.v2.__internal__' has no attribute 'register_load_context_function'

I had to update Tensorflow to the currently latest version 2.11. when importing i get "AttributeError: module 'tensorflow._api.v2.compat.v2.internal' has no attribute '...
0 votes
0 answers
8 views

Facing Problems in Dilation and Erosion of Segmented Images of Knee X-Rays

I am working on a project to grade the severity of Knee Osteoarthritis using X-Ray Images. Before feeding the images into the machine learning model and enhancing the features of Knee osteoarthritis i....
0 votes
1 answer
373 views

How do I get my Neural network to ignore certain values?

I was wondering if there was a way that I can get my CNN encoder-decoder neural network to completely ignore certain values in my data (2d images). There are some pixel values of 0 that never change ...
3 votes
1 answer
184 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
1 vote
2 answers
79 views

Probabilities in KERAS

I'm struggle with a problem OF class probabilities (binary, 0 and 1). Dont know why but after 100 epoches the probabilities became 0 or 1 (like the class to predict). Maybe something is not working ...
0 votes
0 answers
9 views

GridSearchCV using best paramters

i have got a question, i want to use GridSearch on a MLP and my question is after finding out best parameters do i have to train my model with the data again with the best parameters ?
0 votes
1 answer
50 views

Can I cluster an aggregated data-set (grouped by) and apply dimensionality reduction?

I have data of sales, however it is in the millions, about 500M rows. I aggregate this data by factors such as location, shoptype, country_of_shop, cardtype, and then the aggregated statistic is: ...
1 vote
1 answer
239 views

Predicting churn - deal with missing dates in time series and improve modelling result

This is the follow up question for General approach on time series for customer retention/churn in retail. I have a time series of data in the following form: ...
0 votes
0 answers
11 views

How to split a LGBM.DataSet into trainning/validating/testing sets?

It looks like that LightGBM.train() can accept two arguments i.e. train_set and valid_sets, then it can do validating operations ...
3 votes
3 answers
2k views

How to prepare data for LSTM time series prediction

I have a binary classification task for time series data. Every 14 rows in my CSV is relevant to one time slot. How should I prepare this data to be used in LSTM? In other word how to feed the model ...
1 vote
2 answers
58 views

How to vectorize and speed-up double for-loop for pandas dataframe when doing text similarity scoring

I have the following dataframe: ...
0 votes
2 answers
160 views

How to use function from other Colab file?

I’m coming from a math background and learning some data science. I am very new to some details of this stuff. Working with colab, I’m trying to organize my sample simulating functions in such a way ...
1 vote
1 answer
83 views

How to Proceed with Tokenized text content which is given in number?

I have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed? As I am encountering such data set the first time, ...
0 votes
0 answers
15 views

Curve Fitting for multiple scatter plot from different csv dataset

I have successfully plotted multiple scatter plot from different dataset into a plot with the figure below. The next agenda is curve fitting for this scatter plots and then proceeding to analyze using ...
0 votes
1 answer
241 views

Sparse matrix after vectorization giving size = 1

I am working on a NLP problem https://www.kaggle.com/c/nlp-getting-started. I want to perform vectorization after train_test_split but when I do that, the resulting ...
1 vote
2 answers
178 views

Problem importing CNTK in Azure jupyter notebook

Vineeth Sai indicated in this that with the following code: pip install cntk the problem is solved. However, I am getting the error shown in attached image:
0 votes
1 answer
70 views

How to Manipulate data for multiple visits per person?

I have a query to solve. I have data regarding customers and number of visits done to them. These are in two tables. So I want to join two table and create different features so that I can find better/...
0 votes
1 answer
93 views

Using text classification for system calls

I'm working on a project in which I should classify System calls sequences, my dataset is represented as sequences of integers (from 1 to 340). To do the classification I have inspired from Text ...
0 votes
1 answer
641 views

How do feature selection on a sparse matrix?

Say I want to do features selection on a sparse matrix, i.e., 10,000 rows x 1500 features, but the matrix is mostly sparse. Let's say the features are all numeric and the target is binary and discrete....
0 votes
1 answer
55 views

How to calculate the evaluation metrics (i.e., F1 score) in leave one subject out cv when a subject belongs to single class only

I have dataset of 10 subjects. the dataset has 4 classess. 0,1,2 and 3. The distribution of classes are not same. For example subject 1 does not have 1,2 and 3. It belongs to zeros class. currently ...
0 votes
2 answers
80 views

Predicting a relative distribution between the available instances

I am working on a model that is supposed to predict how a given order volume distributes over the available articles in a retail scenario. For simplicity's sake, let's say I'm a retailer that buys ...
0 votes
0 answers
4 views

Is there a way for CTC to output different types of blanks?

I am using a CTC loss for math handwriting recognition in Tensorflow/Keras. The output consists of a sequence of symbol ids, with a spatial relationship between every pair of consecutive symbols. For ...
1 vote
1 answer
192 views

Python equivalent of Wolfram Language Query for JSON?

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level ...
2 votes
1 answer
39 views

Reference implementation of q-learning in Python

I'm a machine learning newbie, trying to learn Q-learning. I read a few texts and I get the general gist, but what I'd really love to see is a simple example of a Q-learning algorithm in Python that I ...
0 votes
3 answers
234 views

Perform fuzzy lookup to partially match the address column

I have two separate datasets (eg. dataset1.xlsx and dataset2.xlsx). Dataset1 has 2 columns, serial number and service address. Similarly dataset2 has 2 columns service address and customer number. The ...
3 votes
1 answer
1k views

Keras bug NasNetlarge no top

I am trying to use NasNetlarge in Keras without the top but I cant get rid of the top: ...
6 votes
1 answer
8k views

Keras - Implementation of custom loss function with multiple outputs

I am trying to replicate (a way smaller version) the AlphaGo Zero system. However, in the network model, I am having a problem. The loss function I am supposed to implement is the following: $$l = (z -...
2 votes
1 answer
387 views

Custom connections between layers Keras

I would like to manually define connections in neural network between layers using TensorFlow and keras with Python. By default connections are beween all pairs of neurons. I need to make connections ...
0 votes
1 answer
537 views

Set column names to column value in pandas forloop, add prefix to column names

I have a dataframe with repeat column names, and I am trying to add a prefix label to column names with an appropriate identifier that comes from a column within the dataframe. I have this data frame: ...

1
2 3 4 5
132