Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

How to Re-Train TextBlob?

I am conducting a sentiment analysis of historical newspapers. For this, I have been using TextBlob's sentiment.polarity. The results are okay, but I am curious how they would differ if I re-trained ...
user avatar
  • 1
0 votes
1 answer
16 views

Pandas Dataframe grouping and summarizing

I have the following pandas dataframe: df_1: ...
user avatar
1 vote
2 answers
19 views

How to improve Regression RMSE with LightGBM

I have the following dataset: https://raw.githubusercontent.com/Joffreybvn/real-estate-data-analysis/master/data/clean/belgium_real_estate.csv I want to predict the price column, based on the other ...
user avatar
0 votes
1 answer
16 views

What's more important - accuracy on training or accuracy on cross validation?

I optimized a knn algorithm in sklearn with a grid search. However, my accuracy on the training data decreased 1% while my cross validation accuracy increased 0.7%. Is the model better after the grid ...
user avatar
0 votes
0 answers
8 views

ValueError: Input 0 of layer "lstm_6" is incompatible with the layer

I am trying to create a hybrid model which is consists of EfficientNetB7 and LSTM. ...
user avatar
  • 27
0 votes
0 answers
11 views

LightGBM Price Prediction always INF but RMSE 0.19

I have a real state dataset from Belgium: https://raw.githubusercontent.com/Joffreybvn/real-estate-data-analysis/master/data/clean/belgium_real_estate.csv And I want to use LightGBM for price ...
user avatar
0 votes
0 answers
7 views

Which time series model to choose?

I am new to time series forecasting. I have two very large dataset consisting of about 530k values obtained from a scaled dataset. The nature of both of these datasets are different. One of them has ...
user avatar
1 vote
2 answers
15 views

What is the role of epoch in this Geron's code?

I am reading Hands-On ML 2nd Edition. In page 142 there is the following code as an example of Early Stopping: ...
user avatar
0 votes
1 answer
19 views

Is there any faster way to webscrap tables using python pandas.read_html?

I am using pandas.read_html to webscrapping the tables from a website having over 9000 pages. My code (dummy) is : ...
user avatar
-1 votes
2 answers
21 views

Why I am having trouble plotting the AUC?

I am trying to plot the roc_auc curve however I am not getting any results. Any explanation here? Are there any problems with the number of data? Here is my example : ...
user avatar
0 votes
0 answers
10 views

What are the best practices in operationalizing a Python Keras model?

What are the advantages and disadvantages of, Spending the additional effort and operationalize in Python Expose the model via an API which takes the input vector and outputs an output vector and ...
user avatar
0 votes
0 answers
19 views

how can I convert set Threshold function in R to python

Hello all, I am trying to convert this R function into python using rpy2 but I am getting an error msg - ...
user avatar
  • 1
0 votes
0 answers
16 views

How to mathematically calculate individual pixel values after each image rotation?

If I have an image that is being rotated using OpenCV in python, about its centre, How do I mathematically calculate the pixel values after each rotation? A python code for this operation would be ...
user avatar
0 votes
0 answers
12 views

how to determine significance with permutation t-distribution values

suppose I have the following array of t-distribution values from a permutation test (e.g., linear model tested with permutation test). For display, let's say I did 10 rounds. ...
user avatar
0 votes
1 answer
12 views

**tokens when tokens is a dictionary

Trying to understand the code from https://www.analyticsvidhya.com/blog/2021/05/measuring-text-similarity-using-bert/ I am looking at understanding the syntax on these two lines: ...
user avatar
0 votes
1 answer
14 views

Str.contains and isin function do not return all correct rows of dataframe

Given a list of strings L1 L1 = ['a', 'b', 'c'], I need to extract the rows which contain the values given in list L1. I used the isin function: ...
user avatar
  • 23
0 votes
0 answers
7 views

checking correctness of predicted values of a CNN

I have trained a normal CNN to recognize patients with a disease or not. I print the predicted values for the test set, and get the probability of the various images of belonging to a class rather ...
user avatar
0 votes
0 answers
9 views

Is 500 epochs too much for a CNN project?

I am working on a project where I need to train a model with a data set of 250 images. My epochs count is 500. Is that too much? Will it overfit? I did this because ...
user avatar
0 votes
0 answers
8 views

Find the lowest number of variables that characterize a discrete variable

I have a dataset where one of the variables is discrete (for example purposes I took the wisconsin example dataset), In this dataset we have the variable "target" with a 1 or a 0 depending ...
user avatar
1 vote
1 answer
35 views

How to automatically build a high-quality dynamic PDF report?

I want to create a pdf data report programmatically so that it leads to a report in the style of this (it is in German but it shows the design I want to achieve): https://www.destatis.de/DE/Service/...
user avatar
  • 131
0 votes
0 answers
14 views

Neural Network to learn evolution equations in Tensorflow

I am trying to create a NN function that is able to learn how the evolution equations progress, ie. given a time series $\boldsymbol{x}(t)$, recover $\frac{d\boldsymbol{x}(t)}{dt}$. I would expect my ...
user avatar
0 votes
0 answers
10 views

Network Visualisation (Python/Excel)

I have a large data set similar to that in the screenshot below: and I want to visualise the whole data set like the diagram below (made with a lot of effort in PowerPoint!) Is there any way to do ...
user avatar
0 votes
1 answer
17 views

Python-SQL-Excel : When to use them?

I am currently learning data science and learned already a lot of Python not just for data science, also for script writing, web scraping etc. Meanwhile i started to look for how people actually earn ...
user avatar
1 vote
1 answer
13 views

Pulling out Training Data from another Dataframe

I'm trying to do a Random Forest Regression on a geographical dataset. I'm hoping I'm doing things right, and if anyone can see an issue with this please let me know! Problem: I have an area, with a ...
user avatar
0 votes
1 answer
15 views

Time Series Data Imputation

I am studying time series analysis to apply on a new project. Well, I am confronting a dilemma that I need some help. When I read an old version of ggplot2 book (https://ggplot2-book.org/), I guess ...
user avatar
0 votes
1 answer
33 views

Lost human names after 'Lemmatization' for topic modeling in python

I'm using gensim in Python for topic modeling. Currently, I have one problem. If I don't lemmatize, human names will appear as 'Most Relevant Terms for Topic,' but after lemmatization, the human names ...
user avatar
0 votes
1 answer
45 views

TypeError: __init__() takes 1 positional argument but 5 were given

I got this positional argument error. Here is the code that I copy from other sources. ...
user avatar
  • 1
0 votes
1 answer
21 views

How to build correct dataset for ann classifier?

I have x signals each with 5000 rows. Each signal x has its own one output in range from 1 to 6 (categories). So for example signal x1 has output 2, signal x2 has output 1. How can I build X and Y ...
user avatar
0 votes
0 answers
11 views

Need help thinking an inprovement on my company's data flow

So I work at a small startup as a data everything (I ETL the data, create reports, deploy services, all of it) with little to no money to spend in this area. Our DB is around 100MB. Currently this is ...
user avatar
0 votes
0 answers
11 views

How to do a batch trainning of Pytorch model without using Dataloader?

I am doing a time series data training. I have to pad 0s to the data so the sequences have the same length. Because of 0s are padded, I have to mask them during the training, for Keras, it is simply ...
user avatar
  • 1
0 votes
1 answer
18 views

How to find best range of independent variables in Random forest classification

I am running a binary random forest classification model and I need to know what the best possible/optimal range of each of the independent variables used in the model that drives best possible class ...
user avatar
  • 1
0 votes
0 answers
17 views

Feature Engineering Encoding for multiple category with huge category range

I need to encode a column "Tags" that has a total of 144 different types and at the same time a row can contain multiple tags. What's the best encoding method in this situation? One-hot ...
user avatar
  • 1
0 votes
1 answer
25 views

Explaining Probabilities in Python with Naive Bayes

I have run some data on the possibility of churn in a telecom company based on 6 variables How now do I interpret the output below for the probabilities:
user avatar
0 votes
1 answer
20 views

Extracting a value from text based on training data?

I have a large structured dataset with 2 columns containing: A paragraph of text with various text and integers A integer that's found in the text within the first column What I want to do is train ...
user avatar
  • 1
3 votes
3 answers
268 views

Am I building a good or bad model for prediction built using Gradient Boosting Classifier Algorithm?

I am building a binary classification model using GB Classifier for imbalanced data with event rate 0.11% having sample size of 350000 records (split into 70% training & 30% testing). I have ...
user avatar
0 votes
1 answer
30 views

LSTM model accuracy checking

Is this a good result? How do you print the model accuracy (in %) from this graph?
user avatar
  • 1
0 votes
0 answers
9 views

Using a supervised model (BiGRU-CRF) with lack of labelled data. How to get labels with no human labeled data?

I'm working with a project supervisor on a deep learning project. The project involves extracting keywords or catchphrases from legal documents so that they can then be used for semantic search of ...
user avatar
1 vote
2 answers
25 views

Interpolate a point in time for two given geolocations and their times in python

Let's say I have two geolocations at given times. How can I interpolate these two geolocation and find any location for a given time time3 in python? ...
user avatar
  • 111
0 votes
0 answers
21 views

Which model is best for the generating accurate answers of the Boolean questions?

I am trying to generate the question using T5 transformer answer of the questions but I am getting the error like below. here is the code. ...
user avatar
0 votes
1 answer
12 views

Compute similarity with given weights for each different feature

I would like to find similar products based on the features. I have: 3 categorical features (X1,X2,X3) 1 numerical (continuous) feature (X4) 1 date feature (X5) Therefore, I want to give a pre-...
user avatar
0 votes
0 answers
14 views

Improving f1 score in an image dataset

I'm having troubles trying to improve the f1 score of my model. Here is the code: ...
user avatar
0 votes
0 answers
14 views

What model to use to forecast on small dataset with many parameters?

I have a dataset of 20 columns/parameters (x's) and not many rows (10-20 historical values for each x) I need to use to predict a y column for the future 5 years. Each row represents one year. What ...
user avatar
  • 1
0 votes
0 answers
14 views

Extracting meaningful information from time series data in python

The following graph shows the data (Top: Time domain ; Bottom: Frequency domain). The meaningful data in my context is the one marked by red lines. How can I extract only the meaningful data from the ...
user avatar
  • 103
0 votes
0 answers
32 views

Conversion from Tensorflow 2 to a Tensorflow 1 / keras models

I train my deep learning model in tensorflow 2 (Tf2) environment because my latest GPU/Driver doesn't support tensorflow 1 (Tf1). However, the program I need to use post-training does not work with ...
user avatar
0 votes
0 answers
19 views

How can I build and train mode for Arabic word embedding from scratch using BERT and share the model on hugging face?

my project is (building an Arabic word embedding model). I want to build my own model on hugging face like (aubmindlab/AraBERT model) for Arabic language using Bert for word embedding. How can I start ...
user avatar
0 votes
0 answers
25 views

How to improve validation score

I am working on time series classification. My data has 4 classes. I used this paper's architecture on my data (1611.06455). However, my results look like this : . Here is a link to my notebook I ...
user avatar
0 votes
0 answers
4 views

Partial fit with tslearn clustering methods

Is there a way to use partial_fit with tslearn clustering methods like TimeSeriesKMeans? I ...
user avatar
1 vote
1 answer
32 views

How to prepare the Dataset for Keras LSTM Model?

I want to forecast the future sales for products. My Questions: How I have to set up the dataset for time series forecasting? What I have to change, to get int values from output? Model and ...
user avatar
0 votes
1 answer
28 views

The best Python library to build decision tree on binary inputs

Please, could you advise me the best Python library for the following problem. I have 60 binary input variables and a binary output variable. There are 10 000 – 20 000 training examples. I want to ...
user avatar
0 votes
0 answers
22 views

Dataframe to a dictionary as some columns in a list as keys and one as value [migrated]

I have a pandas dataframe df that looks like this: col1 col2 col3 A X 1 B Y 2 C Z 3 ...
user avatar

1
2 3 4 5
126