Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

How to programmatically differentiate between extractive and abstractive summarization in NLP?

We are using different pre-trained models in python transformers library, to generate summaries(both extractive and abstractive). So is there a programmatic way, based on the output summary, we can ...
1
vote
0answers
35 views

Normalizing and joining of independent logistic regression model's prediction

I need to train several Logistic regression models on a different set of data (with a different set of labels): ...
1
vote
0answers
22 views

Calculate the Convolutional Autoencoder sizes - Conv1D

I'm approaching the Conv1D for the first time and I do not understand how to calculate the parameters in each layer. I have an input of (3000, 10, 30), but I ...
0
votes
1answer
12 views

How to aggregate features to a group level as a feature in machine learning model?

I am building a model to predict some behavior at a household level. I could roll up income or number of cars etc so that I can take everyone into consideration. But how can I roll up something like ...
1
vote
0answers
32 views

How to maximize a log linear regression equation satisfying a constraint?

I have a log linear equation of the form $y = w_1(\log{X1}) + w_2(\log{X2}) + ... + w_n(\log{Xn})$. How can I find the value of X's that maximize the value of y subject to a constraint $(X_1+X_2+...+...
0
votes
1answer
13 views

Why does keras.Model subclass behave differently to the sequential API?

I have implemented WGAN based on this blog. I tried to implement define_critic() as a subclass of keras.Model with the goal of having the same result. ...
1
vote
0answers
26 views

Create sequence for a Conv1D layer

Im studying the following tutorial on the Keras website and I'm trying to understand how to create a sequence for a Conv1D layer. This is their method: ...
0
votes
1answer
15 views

Set seed for a Class that calls Keras Models

I have class that I use to optimize parameters of a Keras LSTM model. It is known that to set seed for keras, one must input the follow on its code. But what I'm not understanding is where to put it ...
0
votes
1answer
7 views

Is there a way to make the window in df.rolling dynamic depending on which row it is calculating for?

I have a dataset of stock prices, and I want to add a column of 52 week lows for each day, however for the rows which dont have 365 days above them I just want the column to have the rolling min using ...
0
votes
1answer
38 views

Which data science framework should I start learning? [closed]

I have learned Python but am confused about which data science framework should I start off with? Matplotlib or Pandas or Numpy? Also, can you recommend which tutorials I should look for each one of ...
0
votes
0answers
11 views

How could I find the number of instances per fold to plot a histogram, in my dataset?

I am currently trying to learn machine learning with Python, and I have been given an .npz file containing the data. I have explored the dataset, and after exploration noted that it has 5 datatypes: <...
0
votes
1answer
28 views

How to retrieve tweets from all cities using Twint package in python?

Using Twint, I'm able to get tweets from each city in the list, but while creating a dataframe I only get tweet from the last city in the list, is it possible to append the tweets based on cities? <...
0
votes
0answers
21 views

Predicting % of demand going to each product

I work within an industry with products that expire, therefore we would like to be able to choose which specific marketing keywords we should switch on to drive demand to the products that are over-...
0
votes
0answers
10 views

Which checkpoint file to be used as starting checkpoint for retraining the model in Machine Learning?

I kept a Tensorflow Object Detection model on training to identify diamonds in an image. While training, checkpoint files are saved in Trainings folder. Checkpoints ...
0
votes
0answers
5 views

Using forecasting values of wind speed at different hours in the future to predict power output at different hours in the future for a wind turbine

I need to design a neural network model for a wind turbine which takes as input the forcasting values of wind speed as follows : windspeedPlus001hr , windspeedPlus002hr , windspeedPlus003hr , ...
-1
votes
1answer
22 views

Having an error when converting to data format in python [closed]

I have gone through a lot of forums and discussions. However, can't get my problem solved. Note: 'Released' column has same format for all entries and there are no null values or unwanted spaces in it....
0
votes
0answers
5 views

Automatic Video Editing via Neural Networks; Identify the best start and end frame to trim a video?

I have an abundance of videos from high speed cameras of golf swings. With this data, my ultimate goal is to apply to to these videos an already defined human pose estimation model. However, a clear ...
-1
votes
1answer
45 views

train_test_split() Error: cannot reshape array of size 900000 into shape (100,100,3) [closed]

I am fairly new to python and I have a program for data classification using the k-nearest neighbor method. But I encountered an error when running the program. Here my source code: ...
0
votes
0answers
24 views

How to select correlation between object type data for modelling in Data Analysis?

I am analyzing the data with Python. I am at the stage of finding correlation of columns with the target column (price in this example) after data cleaning, to use the columns which have correlations ...
1
vote
2answers
21 views

How to deal with the difference in the range of Dependent variable in Train and Test data

I am training a XGBoost model for predicting number of applications and the minimum number of applications in training data is 40 and the maximum number of applications is 2000, while in test set ...
0
votes
1answer
19 views

Precision, recall and accuracy metrics significantly different between training/validation and actual predictions

I have two sequential models built with Keras that train on data from a CSV file. This is how they are built ...
0
votes
1answer
26 views

Understanding time series anomaly detection using Autoencoder

I'm studying how to detect anomalies in the time series using an Autoeconder. In particular, I'm following the guide posted in the Keras website, but I don't understand why they are creating and how ...
0
votes
1answer
16 views

keras mnist dataset

I am learning Neural Network. I was running following source code ...
2
votes
0answers
58 views

What are available Python libraries for Interpretable ML?

I recently become familiar with Interpretable ML and I found some libraries like LIME. I would be thankful if you can suggest to me some libraries and what are the advantages of each library.
0
votes
0answers
9 views

How to deal with time features/seasonality in NYC taxi dataset

I'm using Catboost (and will also construct an LSTM) to predict pickup demand for Queens NYC taxi zones 1 step ahead using data spanning a few months (using resamples of an hour per taxi zone). I ...
0
votes
0answers
13 views

Dynamic Time Warping with fast lookup?

My understanding of Dynamic Time Warping is that the algorithm always requires calculation with each comparison/training series and that there is no way to extract the "essence" from a given ...
0
votes
1answer
37 views

How does class_weight work in Decision Tree?

I am interested in Cost-Sensitive learning. And I am trying to understand how class_weight in DecisionTree works in terms of math. I read a lot of articles that ...
3
votes
1answer
49 views

Is there an appropriate use of adjusting class weights for a balanced dataset?

I ask this because I am currently working with a CNN model built for diagnosis of pneumonia. Originally, I followed a notebook on kaggle to build the model and thereby learn what each bit of code is ...
0
votes
0answers
37 views

How to find the optimal number of clusters for K-modes? : Python

I have a categorical dataset (survey data on a likert-scale: never to extremely often). So, I need to cluster them. Therefore, I am planning to use K-modes. But, I am stuck at selecting the optimal ...
1
vote
0answers
18 views

Simple constraints on Logistic Regression

I am doing with a binary classification problem. I have three features (w1x1 + w2x3 + w3x3 + w4), and I want to get a rule so that there is for sure w1 > 0, w2 > 0, w3 < 0 and any constant. ...
1
vote
0answers
9 views

TensorFlow Poisson error

I am trying to assume that my y variable has a Poisson distribution and I am using TensorFlow to do so. The model I have is as follows: ...
0
votes
1answer
48 views

How to work with input which is a combination of metadata+ vectorized text data + image pixel data to build a Regression Model (predict views)?

There are 4 datasets (all in csv format), each has a uniqueID column by which each record can be identified. Image and text datasets are dense datasets.(need to be converted to ndarray). Can someone ...
1
vote
1answer
35 views

Create a single time series plot of multiple devices [closed]

I have a dataset where there are stored the measurements of 30 devices. Each device has about 4000 values and it is structured as well: ...
0
votes
1answer
34 views

Extracting Products Name from Unstructured text

I have unstructured text like this ...
0
votes
1answer
30 views

Plot overlapping time series [closed]

I'm trying to plot my test set and test set predictions to check the differences and see how my autoencoder reconstructed the data, but since I have a test set 30x10 I have a huge visualization ...
0
votes
0answers
9 views

SGD Classifier partial fit throws ValueError on second iteration

I am currently training a nlp model with labelled data to find area in an address. Using SGDClassifier's partial_fit function, I am trying to split training data into smaller batches due to ...
0
votes
0answers
18 views

Torchtext data loading with vector labels

I was wondering if anyone knows how to modify the following code if my target for each sentence is a vector? For example, the first line of my data set could be ("hello world", [0 1 0 0 0 1])...
0
votes
0answers
17 views

How to create training dataset based on sampled or original data?

I am trying to used SMOTE and Feature Selection by following this paper http://jad.shahroodut.ac.ir/article_825_679b8f128dec2874a8fbc314fc922127.pdf In this paper, the authors have mentioned about 4 ...
0
votes
0answers
11 views

Putting a model [sklearn] from Git into production

Currently the DS team provides Python (sklearn) models to put into production. This is considered a devops task. The workaround in lieu of expensive (platform /infrastructure) we went for is using ...
0
votes
0answers
11 views

estimating single or multiple model for Multiple Time Series Forecasting

I am a newbie in the ML field. So please, neglect or better correct, if I am wrong somewhere. I am working on a requirement where details of loading time for each page/component will be given. Now I ...
6
votes
1answer
97 views

What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are: AWS instance with massive RAM and train CPU (slow, but ...
1
vote
1answer
42 views

lookup and fill some value from one dataframe to another

I have 2 dataframes, df1,and df2 as below. df1 and df2 I would like to lookup "result" from df1 and fill into df2 by "Mode" as below format. Note "Mode" has become my ...
1
vote
0answers
16 views

How to generate data for future price predictions with Linear regression and Python

I have written the following Python code that makes predictions using linear regression model. However, I'll appreciate you pointing out what went wrong with how I generated data['X_forecast']. I'm ...
0
votes
0answers
15 views

Curve fitting (or other techniques) for revenue and price optimization

My question is as follows: with historical data we can model the relationship between discounts and sales (number of sold units). So we can address the question "how are sales boosted by ...
0
votes
0answers
8 views

outdoor/indoor recognition in text

For a project I should train a model that predicts whether a certain text given as input (e.g the chapter of a book) is set indoor or outdoor. I do not have any labelled data. Could you point to ...
0
votes
0answers
12 views

Robust Gaussian Fit

I have tried to find some literature on robust gaussian fits, all I could find was good old EM gaussian mixtures. The question is : given a mixture of gaussians, find the dominant one around a given ...
0
votes
1answer
42 views

NameError 'np' is not defined after importing np_utils

I am running a MNIST example in a Jupyter notebook running in an Anaconda virtual environment. I have tried to run the code below (not yet finished, I was testing it) when it comes up with an error (...
0
votes
2answers
59 views

sklearn package with AttributeError: 'MissingValues' object has no attribute 'to_list'

I am currently trying to reproduce this tutorial on building a CNN based time series classifier for human activity recognition. My setup is: Windows 10, Pycharm IDE with a new project for this ...
0
votes
0answers
18 views

Can I get a global attribution when using Shapley values?

I'm using a shap KernelExplainer to interprete my model and I'd like to display a global attribution for each feature. The idea is to represent, in a numerical way, the plot. What would be the best ...
0
votes
0answers
13 views

XGBoost hardware requirements. CPU vs. memory

I am working on predictive models with ML using very roughly 10-50 million records (currently testing with less) and around 10 explanatory variables per model. When outlining hardware requirements for ...