Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

How to prepare data for the encoder part of the model in chatbot using Tensorflow?

I hope you are doing great. I am trying to build a chatbot using Tensorflow. The dataset I am using is cornell movie dialogs. I have a very basic doubt. I am almost done with the preprocessing part of ...
1
vote
0answers
9 views

px.choropleth not showing any map, but just the legend!

I am unable to figure out why below code doesn't give proper o/p as maps, but just legends. *Summary_Table is my data_frame that contains 'id' and 'Total Deaths' and geojson_data contains the location ...
1
vote
0answers
10 views

Fail to decompose and make stationary time series

I am looking for some suggestions for my time series. I am dealing with the column "Temperature (C)" from this dataset. I am trying to make it stationary in order to do some forecasting on ...
2
votes
2answers
43 views

SKlearn PolynomialFeatures R^2 score

I'm trying to create a linear regression model with use of PolynomialFeatures. But when I evaluate it, I get really strange scores. I know that R^2 can be applied to this model and I think I've trying ...
1
vote
1answer
29 views

Clustering text data based on sentiment?

I am scraping reviews off Amazon with the intent to perform sentiment analysis to classify them into positve, negative and neutral. Now the data I would get would be text and unlabeled. My approach to ...
0
votes
0answers
13 views

Group related items by their description and tag each group. [Pen, Eraser] : Stationary

So have a list of data similar to the table below. It will be captured by a chatbot so I expect natural language but in the form of a structured command: Add {Qty} {item description} to {location} ID ...
0
votes
0answers
13 views

How to standardize a tensor

When I have an image , I can standardize the image channel-wise as follows : ...
1
vote
1answer
44 views

Should I merge multiple target bins into one for better results?

I have a multiclass classification task where the target has 11 different classes. The target to classify is the Length of Stay in a hospital and the target classes are in different bins, for example, ...
0
votes
1answer
11 views

Cost Efficient Machine Learning Development Using Cloud GPUs

Let's say I have a Juypter Notebook I am working on where I am analyzing, visualizing, testing, etc. various Machine Learning Models with different hyperparameters on some arbitrary data set or I am ...
0
votes
1answer
25 views

How to get the expected output shape from a unet model?

I have an image segmentation task where my input image shape is (140, 85, 95, 4) and the output label shape is (140, 85, 95). Below is my model: ...
1
vote
0answers
12 views

What is an efficient implementation of a custom map-style dataset for a hdf5 file with irregular structure?

I have a hdf5 file that contains picture of a certain number of people, from a certain number of source cameras, for many seconds. So it is like this: ...
0
votes
1answer
20 views

Applied and view jobs ratio

I have the following data set where the column "kind" can be V(view) or A(apply), how can I do the following things given a particular job id how many applicant Apply (A) to that particular ...
0
votes
1answer
55 views

Error: `raise ValueError( ValueError: Missing column provided to 'parse_dates': 'Date' [closed]

I am using a .csv with two columns. The first has dates and the second has temperatures. I would like to plot it with dates on the x-axis and temperatures on the y-...
0
votes
0answers
26 views

Split dataframe to a train and test sets with a cross validation of x%

I am working on a dataframe and need to split it into a training set and test set, with 90% for Cross-Validation training, and 10% for a final test set. The problem is that I do not know where to ...
0
votes
0answers
14 views

Time Series Hyperparameter Tuning

My question is about the intuition for hyperparameter tuning of time series. In other models, like Linear or Logistic Regression there is labeled data and according to accuracy or precision, the ...
1
vote
0answers
7 views

Audio Classification with Counter

I'm trying to create a model that can identify one particular sound, and every time it hears that sound, it increases a counter by 1. So for example, if it hears a specific bird chirping ten times, ...
1
vote
0answers
30 views

Price Predition for Irregular spaced historic data of non independent Prices

I am a little unsure how to proceed. I am not an expert but on a decent intermediate level when it comes to regular Timeseries. Now i am faced with a problem that first seemed related, but is an ...
1
vote
0answers
60 views

Keras model prediction always has unwanted offset

I am trying to predict next 10 days by looking into the last 60 days. So tried to implement an LSTM layer. Before jumping into the question, I want to clarify a few points. Firstly, this is a Multiple ...
2
votes
0answers
5 views

Shifting The Result of Regression Model By N-Value

I am doing a multivariable regression problem, which predicts the frequency of a failure mode in a production system. In this problem, I used XGBRF for Regression as my ML model. These are the results ...
5
votes
1answer
137 views

Visualizing effect of regularization for linear regression problem

I wanted to put together an example notebook to demonstrate how regularization makes an impact for such a simple model as a simple linear regression. When executing the below script though, I notice ...
0
votes
0answers
18 views

320GB `YYYY/MM/DD/HH/*.json.gz` -> `YYYY/MM/tenant_id=x/data.parquet`?

I have 1mil gzipped files which contain in total 350mil \n separated json objects. 26GB compressed, ~320GB uncompressed, representing 7 years of data for a multi-...
1
vote
0answers
14 views

Tensorflow model structure is strangely deformed in model.fit()

I was able to confirm (by checking model.summary()) that a model with the correct structure was successfully created. However, when ...
0
votes
0answers
13 views

Two variables polynomial fit with Python

I have two numpy arrays (the first is 2D, the second 1D) in the form: $X = [[x_1,y_1],[x_2,y_2],[x_3,y_3],...]$ $Z = [z_1,z_2,z_3,...]$ I would like to fit them as I expect they respect a polynomial ...
0
votes
1answer
31 views

Data Preparation for dates

I have two columns, a birth year and the first year of being a customer. There is a large number of samples that have a birth year less than the first year of being a customer which does not make ...
1
vote
1answer
16 views

Why does the 1st derivative appear to lag the slope of the fit in Scipy's Savitzky-Golay filter?

I have a simple script that performs the Savitzky-Golay filter on a toy dataset of forex prices from yahoo finance: ...
2
votes
0answers
36 views

Deploying models in the real world

I've been thinking about this for a while. I really would like to know other opinions so please answer, even if it sounds stupid! Just as any application, ML/DL models can be deployed in a very wide ...
0
votes
1answer
33 views

Custom loss function for regression

I am trying to write a custom loss function for a machine learning regression task. What I want to accomplish is following: Reward higher preds, higher targets Punish higher preds, lower targets ...
0
votes
1answer
18 views

How to use TfidfVectorizer on dataframe

I have the dataframe which has 3 colums(Positive Reviews, Negative and Score): ...
0
votes
0answers
17 views

What are my options for classifying multiple multivariate time series and making live predictions?

I'm looking for some ideas on how to handle a multivariate time series classification problem. Specifically, I'm dealing with many hundreds of time series of varying length (but within the same ...
-1
votes
2answers
26 views

gridsearchcv best coefficients do not match well with the perfect line

I wrote a program to find the best combination of coefficients to describe a variable. However, the coefficients from the gridsearchcv do not match well with the expected line. This is a sample of my ...
0
votes
2answers
47 views

How to remove rows from a data frame that have special character (any character except alphabet and numbers)

How to remove rows from a data frame that have special character (any character except alphabet and numbers) I have some unwanted labels which seems to be useless but I want to remove them, this are ...
0
votes
0answers
13 views

Label PCA "clusters" with unique values from a dataframe in a graph

Context: I'd like to label each "cluster" accordingly to some labels. I have about 20k rows with > 50 columns (dimensions) and one of the columns are the labels (for a total of 19 levels ...
1
vote
1answer
35 views

Get Hidden Layers in PyTorch TransformerEncoder

I am trying to access the hidden layers when using TransformerEncoder and TransformerEncoderLayer. I could not find anything ...
2
votes
3answers
592 views

What do x and y mean when working with data in the machine learning domain?

We need train & test data, so what do x and y mean? Does it mean that it divides 15% to the 'x_train', 'x_val', 'y_train', 'y_val' ? ...
0
votes
1answer
65 views

Which Keras metric for multiclass classification

I have a multiclass classification data where the target has 11 classes. I am trying to build a Neural Net using Keras. I am using softmax as activation function ...
0
votes
0answers
19 views

what python package, or algorithm can be used to find a pattern like "buy A, then not buy B", like the opposite of association rule?

we know association rule mining can find the patterns like "bread --> milk". It means if you buy bread, then you might buy milk with high probability. But, is there a python package, or ...
2
votes
1answer
36 views

Correlating & Combining categorical and numerical features for classification problem

I just started with my first Machine Learning project (Jupyter Notebook, Python, Scikit-learn, pandas) and I am working on a Palmer Penguin Dataset. I have been doing some data analysis and I got ...
1
vote
0answers
23 views

What model to train to restore MNIST test dataset

I came across this problem, and not sure where to start. What model would work best for this problem and why? Imagine the digits in the test set of the MNIST dataset (http://yann.lecun.com/exdb/mnist/)...
0
votes
1answer
16 views

Tree based Classifiers with Label Encoder and One Hot Encoder [duplicate]

I m working with Tree-based classifiers in scikit-learn - Decision Trees and Random Forest, for a data classification use case, and the feature set is a mix of both categorical (majority) and ...
1
vote
0answers
20 views

Interpreting XGBoost's results (when they span between [0,0.5])

I would like to classify sentences into one of two different categories. I trained a XGBoost model over a search grid with k-folds cross-validation. My data represents sentences, and features ...
1
vote
1answer
50 views

ValueError: Data cardinality is ambiguous: (Jupyter Notebook)

I'm building an OCR to read text off of water meters. I'm running into the error mentioned above when I try to fit the machine learning model. I am using the segmentation_models python library. ...
0
votes
0answers
16 views

Automatize autocorrelation in python

I'm trying to automatize my autocorrelation study in Python. My question is: is it possible? Let me explain. I have a time series and I just learnt how to interpret the autocorrelation plot. My ...
1
vote
1answer
34 views

Anomaly detection and replacing it with past values in time series

I am trying to use anomaly detection to find the anomalies in my time series, and if I find it, I will replace it with my past values. I'm trying to do this because I want to create an upper and lower ...
1
vote
1answer
31 views

I want to do sub groups in pandas data frame [closed]

I have a dataframe as below I want to do first group by trip_number and then subgroup by mode. ...
0
votes
1answer
20 views

How to handle multi time series data for 10K + items

There are 50 shops and each shop have 30000 items. Goal is to forecast the sale of item based on shop. Forecase the item_cnt_day, for this i dont see this as multi ...
0
votes
1answer
23 views

Not sure how I can get class labels from Neural Network predictions. Would this be an acceptable alternative to the old method

I am currently working with a dataset that has over 300 variables and a target variable with 10 different classes. My goal is to use said variables and produce a prediction for the target variable. ...
1
vote
1answer
49 views

SMOTE for multi-class balance changes the shape of my dataset

So I have a dataset of shape (430,17), that consists of 13 classes (imbalanced) and 17 features. The end goal is to create a NN which btw works when I import the imblanced dataset, however when i try ...
0
votes
1answer
14 views

How to drop the previous rows of a database based on a matching value in a column?

So I am currently trying to sort through a data frame containing attribute classes and values of teams. However, my data has multiple rows of different classes and values of the same Team ID/Attribute ...
1
vote
0answers
17 views

Causal Inference where the treatment assignment is randomized [closed]

I have mostly worked with Observational data where the treatment assignment was not randomized. In the past, I have used PSM, IPTW to balance and then calculate ATE. My problem is: Now I am working on ...
0
votes
0answers
6 views

Implement NestedCV into Neural Networks

I have a regression task for which I am using ML models. My input features are 64. I implement NestedCV to get best ML models and hyperparameters. I have recently learned Neural Networks and want to ...