Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Precision score (classification_report) 1/0 not predicting for 0 in logistic modeling using Sklearn

I have a binary outcome for an imported CSV file I am working with. I used the sklearn library to use the logistic model to gather predictions for my target value and also generate probabilities. ...
0
votes
0answers
14 views

How to understand logistic trend parameters proposed by Prophet (Facebook)?

I read the paper and I saw that the logistic trend is defined as below : $$ g(t) = \frac{C(t)}{1 + \exp{ -k(t) (t - m(t)) }} $$ Where $k$ and $m$ are respectively a growth rate and an offset parameter....
0
votes
1answer
15 views

Why does downsampling leads classification to only predict one class?

I have a multi-class classification problem. It performs quite well but on the least represented classes it doesn't. Indeed, here is the distribution : And here are the classification results of my ...
1
vote
3answers
49 views

(Python Basic) more elegant way of creating a dictionary

Is there a more elegant way to write a code like this? ...
2
votes
0answers
22 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
2
votes
1answer
17 views

Why result of CountVectorizer * TfidfVectorizer.idf_ is different from TfidfVectorizer.fit_transform()?

I have dataframe: df = pd.DataFrame({'docs': ['gamma alfa beta beta epsilon', 'beta gamma eta',], 'labels': ['alfa alfa beta', 'gamma fi']}) I do count ...
0
votes
2answers
19 views

How to read specific time for specific value row by row using python

Here I have a dataset with date,time and one input column. This input column is included with values 3 and 4. Here I want to read the datetime value which is having 4 row by row. Other rows which is ...
2
votes
0answers
29 views

Tools for reading data from large, irregular csv files (aka excel file hell)

I have a csv file with 1000 columns and 50 rows that was collected over a few months. Embedded randomly in the file are ~1000 small datasets with semi-standard formats (some columns differ, so does ...
0
votes
1answer
48 views

What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm. ...
2
votes
1answer
22 views

How to deal with memory exceptions while processing large amount of data?

Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps: Take backup of the file Extract new records of previous day ...
1
vote
1answer
16 views

Could the Input shape of the LSTM layer not be a constant?

I am trying a vanilla LSTM on my dataset. According to this course; the LSTM layer should be built as follows: ...
0
votes
0answers
14 views

How to prepare data

I have 3 tables Product table: product I'd ( numerical), product ingredients ( words each in one column: ingredient1, ingredient2, etc ), product brand( words), product url, product name Customer ...
0
votes
0answers
16 views

LSTM forcasting: time series as input and a unique value for each as output

I am sorry for the mistakes I will make in this post, I'm new to deep learning ^^' I am trying to build an LSTM model that can help me predict some unique value according to time series indices and ...
1
vote
1answer
33 views

Load large .jsons file into Pandas dataframe

I'm trying to load a large jsons-file (2.5 GB) into a Pandas dataframe. Due to the large size of the file pandas.read_json() will result in a memory error. ...
1
vote
1answer
24 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
0
votes
0answers
6 views

How to assign optimal min_sup value in apriori algorithm

Have processed product sales data after data transformation(map 0&1's respective product and transaction) through Apriori alogorithm.Output associatiob rules have some product mapings between ...
1
vote
0answers
16 views

How to find year in a text using Spacy NER?

Sample Sentence: ...
1
vote
0answers
14 views

Minimizing overfitting when doing hyperparameter Tuning

Generaly when using Sklearn's GridSearchCV (or RandomizedGridSearchCV), we get best model with best test score even if the model overfits a little bit. How can we compute generalization error ...
1
vote
0answers
27 views

Classify the input set into categories based on pre-defined rule set

I'm trying to solve a problem where there are 2 input files given, Input 1: A set of strings in the format. All the strings start with "A". A-B-C-D A-B-C-3 A-X-Y-Z-4 A-X-5 A-X-P-Q-R A-X-P-Q-S A-M-9-...
2
votes
1answer
13 views

Spacy word embeddings for sentence

Spacy offers pre-trained vectors for words. However I have notices that you can get vectors for sentences too: spacy_nlp('hello I').has_vector == True However I ...
1
vote
1answer
37 views

How to install boruta in conda?

I want to install boruta in my anaconda environment, but if I execute conda install boruta It displays, ...
0
votes
1answer
22 views

Getting Validation Accuracy of 99% with MNIST with less than 10000 parameters CNN

Given MNIST dataset in keras,the challenge is to develop a CNN neural net model with less than 10k parameters with 99% validation accuracy. I tried making the model for the same but am getting ...
-1
votes
0answers
23 views

How to select appropriate phrase from the python dictionary depending on the semantic of the text?

I have a python dictionary with multiple choices. I have the text and want to expand text with appropriate meaning from the dictionary depending on the context of the text. For example, in python ...
0
votes
1answer
12 views

Understanding python XGBoost model dump output of a very simple tree

I am trying to understand the model dump output from XGBoost. I would like to step through and see exactly how the model arrived at it's prediction. To simplify I trained a model with 1 tree and 1 max ...
1
vote
3answers
35 views

When to normalize or regularize features in Data Science

How can I know when I need to use normalization and when regularization of features. I know that when I have big difference between min and max values in some feature / column that i need to scale ...
0
votes
0answers
20 views

Training on huge data

I'm trying to find a code with n output bits and k input bits. The criterion for the code is that the minimum hamming distance ...
1
vote
1answer
21 views

Remove rows that are too much alike not to be duplicates

I have a dataset of real estate advertisements. Several of the lines are about the same real estate property so it's full of duplicates that aren't exactly the same. What would be the best methods to ...
1
vote
1answer
78 views

What I'm doing wrong with my CNN Keras?

In my project I have 700 images for each class (pdr and nonPdr) totalizing 1400 images. To validation I've put 28 samples. The problem is that my validation loss and accuracy is unstable. This is my ...
0
votes
0answers
17 views

attaching multiple variables to a single feature [closed]

for the function .fit(x,y), x is 2D array and y is 1D array, x being features and y being labels.Now if we have multiple labels for each feature, what do we do? i.e. What if ...
-3
votes
0answers
27 views

I need a kick to get started with machine learning [closed]

I am a beginner of machine learning, I have a large dataset. I make it short for only the class "Type-01" like below: I am writing a python program to predict next "Type-01" class data for the column ...
0
votes
0answers
3 views

Training EfficientNet always throws CUDA exhausted error

I have been trying a hands on the EfficientNet-B6 architecture but evertime I try to train the same, It simply throws up CUDA resource exhausted error. I have been training it on 16GB Tesla P100
0
votes
0answers
14 views

Using selected variables after dimensionality reduction throws a value error

I am working on a regression problem, namely the Boston House prediction problem hosted on Kaggle. I am currently using Random Forest Classifier to reduce the dimensions of my dataset. But right now, ...
0
votes
0answers
11 views

Performance of standalone R script loading dependencies [migrated]

I am working on a Python application that extracts data, transforms it, and then calls a script that visualises the data. The vis component is handled by an R script. I'm aware you can do data vis ...
-1
votes
0answers
12 views

how to join two columns in excel [closed]

I have two sheets File 1: File 2: how to join two files to get the Zone column from file 2 in file 1 based on supplier and customer column in file 1 I have used vlookup and index-match but i'm ...
4
votes
2answers
42 views

Trained Tensorflow model performs poorly on inference

I trained an image classification model using Keras with Tensorflow backend. The model got good accuracy on validation dataset as well as on the testing data, I save the entire model to ...
1
vote
0answers
14 views

Big dataset for multi-class classification can't be dasked and split, normal one can't be handled

I have a huge dataframe (550MB), the lending club one available here, and I have to predict the class of the grades. The dask dataframe is : ...
1
vote
0answers
21 views

Optimal ML method to detect coupling in tabulated data

Consider a dataset with these features state (s) store_id (s_id) transaction_id (t_id) product_id (p_id) unit_price (p_p) ...
0
votes
2answers
28 views

Data Reshaping for CNN using Keras

I'm a beginner in Keras. I've loaded MNIST dataset in Keras and checked it's dimension. The code is ...
0
votes
0answers
11 views

Comparing two different datasets and updating the first dataset using python [closed]

I have two dataframes. Sample data is attached above. I have all the record in second dataframe , but in first dataframe i have pieces of records. How can i compare the first dataframe with second ...
0
votes
0answers
16 views

Ensure class balanced batches while hyperparameter tuning keras models with grid search

Ensuring class balanced batches while training keras models is possible using fit_generator method. I used imblearn.keras.BalancedBatchGenerator for that and it works fine! But i wanted to do that ...
0
votes
0answers
4 views

Measuring object's surface inside an image

Greeting everyone, I'm working on a project on Jupyter Notebook, python 3 , so I have a set of images of products and I'm trying to measure the surface tooken by each product inside the image which ...
1
vote
0answers
32 views

Data sets that have strings and numerical data all in one column

I m new to the field of Data Science, but I have a great passion for it. Here's a code I wrote for pre processing a data set. It works ...
0
votes
0answers
17 views
1
vote
3answers
17 views

How to drop row where all columns BUT the timestamp are zeros in python

I have a dataset that like any other has zeros and i need to get rid of them. The problem is that I want to delete rows where all the columns values but the timestamp are zeros
0
votes
0answers
82 views
+50

Deep Learning for EMC tests

I have a project that is about deep learning of EMC (Electro Magnetic Compability) traits in televisions. Let me explain it in more detail. Every electronic device in the market is obliged to meet ...
2
votes
1answer
53 views

how to pass parameters over sklearn pipeline's stages?

I'm working on a deep neural model for text classification using Keras. To fine tune some hyperparameters i'm using Keras Wrappers for the Scikit-Learn API. So I builded a Sklearn Pipeline for that: <...
0
votes
2answers
27 views

Pandas Series Sort [closed]

I want to sort digit written in object form as given below: ...
0
votes
1answer
23 views

Error when using np.loadtxt [closed]

I am trying to use np.loadtxt to plot data from a .txt file. Here's the .txt: ...
-2
votes
0answers
16 views

Kaggle Lung Cancer Dataset MISSING (Data Science Bowl 2017)

For a couple of days now, I have been looking for the Kaggle Lung Cancer Dataset (Data Science Bowl 2017) containing 2D, 3D, and labels of CT scans for lung cancer, but I did not succeed yet. It is ...