Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
1answer
18 views

Cannot fig out error in my gradient function implementation in python

Im trying to implement following gradient descent function in Python for logistic regression: $∇θ(−logL)=−X^T 􏰀(y−e^{Xθ}􏰁)$ This is my python implementation: ...
1
vote
0answers
29 views

Understanding reduced dimension embedding from tabular data

Background I am working on building a collaborative filtering recommender system in Keras for a school project, following an approach from this article. The approach is to take tabular user, item ...
1
vote
0answers
23 views

Pre-training model with existing seasonal data for new dataset with maximum limit

I apologise if this is a simple one, I'm not sure if this is not possible or if I'm just not using the right keywords to find the answer; Say I've got a pizza store, I've been able to successfully ...
0
votes
0answers
7 views

How to plot the LSTM model

I am using the following simple LSTM model that is implemented using Keras. ...
0
votes
0answers
44 views

Unsupervised learning, Python, text clustering

I want to do unsupervised learning. As I understand with such learning we don't know the clusters before, right? I read about k-means alghoritm, followed mainly these two articles: The first article. ...
3
votes
1answer
50 views

The output of Model from Decision Tree and Random Forests are different?

I have been made a model using both Decision Tree and Random Forest. But, when I tried to test the model on the same DataFrame the output is different. How is this possible? The data file from my ...
1
vote
1answer
40 views

How do I force specified coefficients in a Linear Regression model to be positive?

Looking for a way to do this in Python. scipy.optimize.nnls forces all coefficients to be positive. Some additional context: I have a data frame with a some explanatory variables and a response ...
1
vote
0answers
8 views

Handling a large dataset consisting of npy files

I have a high number of npy files (448 files) each consisting of around 12k frames (150x150 RGB images) which together make the input to my neural network (X). However, since it is impossible to load ...
1
vote
0answers
21 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
2answers
25 views

How many dimensions should matrix be reduced to with PCA?

I want to reduce the dimensions of my dataset to decrease the complexity without losing too much information. But I am not sure how many dimensions I should reduce to, so that I don't lose too much ...
0
votes
0answers
22 views

Recurrent Python Neural Networks - Reshape () Error [duplicate]

The script below creates the arrays for data prediction using recurring neural networks, If I set the period to 4, the script runs, but I have a 5 value input, how to fix my reshape? DATA SET ...
2
votes
2answers
37 views

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as ...
0
votes
1answer
18 views

Getting extra characters while loading data

While loading data using read_sas from sas dataset in Python getting b' along with every string. How can I fix this?
0
votes
0answers
4 views

Training Algorithm for pointcloud data (3d point data)

I have a "PointNet" neural network which theoretically can work with any number of points as input. I have trained the model using an equal number of points from each object. That is fairly simple and ...
0
votes
0answers
6 views

Static and Dynamic neural networks process

Neural networks can be classified into static (convention feedforward networks) and dynamic categories (RNN, LSTMs). ...
1
vote
3answers
97 views

How to anonymize (de-identify) data in Python?

I have tried a simple algorithm to anonymize the data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values of strings ...
0
votes
0answers
6 views

Learning using DDPG with pyhton solely using historical data

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
0
votes
1answer
61 views

pd.qcut bins error!

Hi Data Science community! I'm working on some RFM analysis, but while setting up the bins, I'm getting an error. Here's my code and below the exact error. ...
0
votes
0answers
11 views

How can I use MIT CBCL dataset in python

I have downloaded the MIT CBCL face dataset, but I can't find any code online on how to use it in python. the MIT data set is in two formats .greynorm and .pgm format which I don't know how to use any ...
1
vote
1answer
46 views

How can I increase my accuracy avoiding overfitting? CNN-Keras-VGG16

As I asked in this question: Why are my predictions bad, if my accuracy in train is roughly 100% (Keras CNN) , my problem was Overfitting, so, I reduce the number of layers, and now I have this model: ...
0
votes
0answers
30 views

token_pattern in CountVectorizer?

I have a question about this piece of a program: ...
0
votes
0answers
16 views

CNN model with transfer learning not performing, training loss is still high, test accuracy is very low

Hi I'm trying to train a cnn model with transfer learning, and I am not able to get a good test accuracy (14%) - I don't know why it doesn't work for me. ...
0
votes
0answers
35 views

Python, Logistic regression - How to calculate Nagelkerke pseudo r squared

I am doing logistic regression in sklearn and I would like to calculate (Nagelkerke) pseudo r squared, which makes more sense for logistic regression analysis. I don't see it is available in sklearn ...
0
votes
0answers
8 views

Gensim Corpora: why the document reference is missing?

I'm trying to understand how the gensim corpora model works. At this page, the example at a certain point create the corpus but in inspecting the content I can see ...
0
votes
0answers
7 views

How to write custom de-identification algorithm in Python?

I have tried a simple algorithm to anonymize my data using the de-identification technique. But the code doesn't work for me. I want to anonymize the data by slightly changing the values. The data ...
0
votes
1answer
19 views

Frequency of occurrence - dummy variables

I am thinking about it not the first time, namely if I have a variable that I want to convert later to the variable dummy (cities in this case), should I delete lines that occur less often than N ...
0
votes
0answers
16 views

Detecting specific points in (noisy) dataset

In my recent work I've came across a problem where I need to find certain points in quickly oscillating data. Let's work with syntethic data instead of real measurements and ignore the noise for the ...
0
votes
1answer
47 views

How to save API data into CSV format?

Is there an easy way to convert API data with python commands and output the results in CSV format? Here's what I'm doing now: ...
0
votes
1answer
37 views

Doing predictive modeling on predicted value

It's a project that I'm working on. Here are the steps I took: I want to make a recommendation service based on the customer data. I first used a collaborative filtering method to get the recommended ...
1
vote
1answer
28 views

List of Correlations to Correlation Matrix [closed]

I am using python to do some data analysis and I need to represent the following table as a correlation matrix. The correlation value is a value between -1 and 1. ...
1
vote
1answer
38 views

Anomaly detection thresholds issue

I'm working on an anomaly detection development in Python. More in details, I need to analysed timeseries in order to check if anomalies are present. An anomalous value is typically a peak, so a ...
0
votes
1answer
37 views

NLP techniques to label unlabeled data in a dataset [closed]

So I have a .xls file with negative and neutral reviews of a medicine. However, this dataset does not have labels. I converted this .xls into a dataframe and I am using the Spacy Lib. I have to use ...
0
votes
1answer
32 views

How to fine tuning VGG16 with my own layers

I want to maintain the first 4 layers of vgg 16 and add the last layer. I have this example: ...
1
vote
0answers
30 views

Low ROC AUC with good Information Value

I'm trying to build my first application scorecard with Logistic Regression and getting low ROC AUC score (about 0.7). Dataset ...
0
votes
2answers
23 views

How to Keep Missing Values in Ordinal Logistic Regression

I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars). One of my predictor variables is also ordinal but ...
0
votes
0answers
22 views

How to improve model confidence for each category in binary classification problem?

I am new to machine learning and expanding upon an introductory tutorial I followed with some artificial housing data I created. I asked this question on stackoverflow and was referenced to the data ...
4
votes
2answers
138 views

Why are my predictions bad, if my accuracy in train is roughly 100% (Keras CNN)

In my CNN i have to handle 2 classes in a binary system, I have 700 images each class to train, and others to validation. This is my train.py: ...
0
votes
2answers
34 views
-1
votes
2answers
155 views

AttributeError: 'DataFrame' object has no attribute 'c'

project=pd.read_csv("train.csv",sep=',') this dataset train.csv has 200column, named as var_0,var_1 ....... var_199 ...
0
votes
0answers
22 views

Getting error when trying retrieve polarity from russian sentence Polyglot(Python)

I trying to retrieve polarity from Russian sentence, using this code: from polyglot.text import Text as T print(T("ты не понимаешь").polarity) But I get the ...
1
vote
1answer
57 views

Selecting features for malware analysis

I am trying to build a classifier that detects if I have a malaware by predicting the provenance compiler. To do so I have a dataset composed of assembly code in json format : In particular, I want ...
2
votes
2answers
49 views

How to Extract Information from the Image(PNG)

I'm trying to extract some particular information from the image(png). I tried to extract the text using the below code ...
0
votes
0answers
18 views

My LSTM regression predicts always the same value

I tried two different models for a regression model. The first one is a basic dense neural network. The performance is, for now, not great at all, but at least, it predicts always different values. ...
0
votes
0answers
7 views

How to proceed after tuning hyperparameters?

As I am still on the journey to understand what when and how to use, I am now at the point how to proceed after finding the best hyperparameters: Define Model (NN) Split Data into ...
1
vote
1answer
40 views

Finding similarity between two datasets

I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state. ...
0
votes
0answers
31 views

Does the add_regressor method on Facebook Prophet also work with categorical variables?

I went through the documentation of Facebook Prophet and was able to build a similar model for my time series dataset. The additional regressors I used were numeric. I achieved a reasonable MAPE score....
1
vote
1answer
41 views

MemoryError when concatenating a large data-frame

I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns. My issue is that these CSV files are quite large and when ...
0
votes
0answers
18 views

Fastest way to generate mini-batches from structured data

To generate mini-batches for training, I use a for loop to extract slices of a 2-D numpy array. Here is some example code I wrote to demonstrate this. Basically I loop through a 2-D numpy array ...
0
votes
0answers
37 views

Connecting to IEX with Pandas Datareader

My Problem is regarding Algorythmic trading. I hope this is the right site for this kind of Question. In specific I try to connect to the "iex" API via the pandas Data reader to retrieve some ...
0
votes
0answers
9 views

From stationary value to original value

...