Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
1answer
36 views

Association Rules with Python (coded dataset)

I have this dataset which I really need to use association rules techniques on. The dataset has like 90 variables, many of which are ordinal. Thing is, the data is already coded using numbers instead ...
1
vote
1answer
993 views

How to calculate NDCG in recommendation system

This is a question about NDCG, which is a recommendation evaluation metric. The following are being used as evaluation indicators for recommendations. $$DCG = r_1 + \sum\limits_{i=2}^{N}\frac{r_i}{...
1
vote
2answers
60 views

How to improve regex while fetching record dynamically

The data is as follows: COL1 COL2 12 :402:agsh,hhjd,:45:hghgh,gruru,:12:fgh,ghgh,:22:hhhh 57 :42:agshhhjd,:57:hghgh,gruru,:120:fghghgh,:12:hhhhhh I ...
0
votes
1answer
15 views

Best Approach to Forecasting Numerical Value Based on time series and categorical data?

Consider a dataset of thousands of car repairs that have been performed. In simplest of terms, the columns to consider are the time of year when it was broken (seasonal changes in demand for car ...
1
vote
2answers
48 views

Bayesian network in Python: both construction and sampling

For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some ...
0
votes
1answer
51 views

generate normally distributed dataframe with 3 columns

I am trying to generate a normally distributed data frame in python with 3 columns. Each columns requires a separate input of mean, standard deviation, lower and upper values. I am trying to generate ...
0
votes
1answer
28 views

How to fix “Expected sequence or array-like”

I am trying to get the accuracy of the model and I am getting this error TypeError: Expected sequence or array-like, got Here's my code sample. ...
1
vote
1answer
24 views

How to get two boxplots on same axis?

I have two dataframes (both of different lengths, etc) but they have exact same columns and I would like to explore the distribution of one of these columns by plotting a boxplot. However, how do I ...
0
votes
1answer
14 views

Adding and multiplying higher values based on different columns of a dataframe

I am trying to calculate scores based on different values in the dataframe. Since these scores are based on different conditions, I am having problems where in the final calculation, which I need in a ...
0
votes
1answer
2k views

The added layer must be an instance of class Layer. Found: <keras.layers.normalization.BatchNormalization object at 0x0000024B0C16B780>

I don't know why BatchNormalization is giving the following error. Googled out but couldn't find any relevant answers My code appears to be this:
1
vote
1answer
620 views

Fasttext error while loading wiki pre-trained data

I am loading the model using gensim package this way: from gensim.models import FastText model = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin') ...
0
votes
1answer
17 views

Multidimensional K-Means wiith sklearn, centroids problem when plotting

I am working with a dataset (X) to predict 12 clusters with K-Means using python SKLEARN library: ...
-2
votes
1answer
19 views

Accuracy of a forecasting model for prediction of COVID-19 occurence

My goal is to find the best performing forecasting model for the occurrence of COVID-19 in Toronto. I pre-train the network with data on the occurrence of SARS in ten countries and Toronto. Then I ...
0
votes
1answer
56 views

Is this attribute numeric or categorical (ordinal)? Help!

So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very ...
1
vote
1answer
22 views

Confusion matrix doesn't display properly

I am trying to plot a confusion matrix using the Logistic Regression for a multi-class ...
4
votes
1answer
96 views

Feature scaling worsens performance?

I noticed that feature scaling can destroy completely neural networks performance in some cases. Below are my results that you can reproduce easily. I use a neural network to approximate function $g$ ...
3
votes
1answer
98 views

Is there a real C4.5 implementation in Python ? (handling missing value)

To my understanding, C4.5 comes with 4 improvements compared to ID3: Handling missing values in both training data and "test" data, Handling continuous data Handling costs on attributes. The pruning ...
1
vote
1answer
66 views

GBM: small change in the trainset causes radical change in predictions

I have build a model using transactions data trying to predict the value of future transactions. The main algorithm is Gradient Boosting Machine. The overall accuracy on the testset is fine and there ...
1
vote
1answer
16 views

How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Say I have a data set for some past period. Now new data appears and for a given variable in the data and we find that the distributions have shifted (for example with "age" it would be that suddenly ...
0
votes
1answer
16 views

Clustering with k-means for text classification based on similarity

I have a column that contains all texts that I would like to cluster in order to find some patterns/similarity among each other. ...
2
votes
1answer
23 views

Detect passive voice in headlines

To detect passive voice in sentences, we can use the spacy module to tag each token in the sentence, then build a classifier to classify it as passive or active based on conventional grammatical rules,...
2
votes
1answer
81 views
0
votes
1answer
19 views

How can I reshape this array?

I have few difficulties in picking values from an array. Here is a simplified version of the concerned part of the code. (If you would like to put the whole code please mention it) ...
0
votes
1answer
18 views

Canny edge detection not working on Gaussian blurred images

I am trying to detect edges on this lane image. First blurred the image using Gaussian filter and applied Canny edge detection but it gives only blank image without detecting edges. I have done like ...
-1
votes
1answer
19 views

How to Select Rows from Pandas DataFrame with condition [closed]

I have a dataframe (csv file) which consists of about 8000 songs (column name "Song")of e.g. 20 singers(column name "Artist"), so each singer has more than one song. I want to ...
1
vote
1answer
20 views

Overfitting results with Random Forest Regression

I have one image that contains for each pixel 4 different values. I have used RF in order to see if I can predict the 4th value based on the other 3 values of each pixel. for that I have used python ...
2
votes
1answer
31 views

Text classification based on n-grams and similarity

I have tried to cluster hundred texts using k-means clustering. I would like to consider other algorithms to group text based on their content and try to spot news not related to other news (topic ...
0
votes
1answer
15 views

Question about reshaping array size for KNN Classifiers

I keep trying to run a new set of data through my KNN Classifier but would recieve the message: ValueError: query data dimension must match training data dimension ...
1
vote
2answers
47 views

Finding roots of a non linear expression when multiplied with a linear expression

Here is a simple polynomial equation: b^2 + 2b + 1 = 0 I could easily solve this as: ...
0
votes
2answers
411 views

How to run a model in an application using gpu (without CUDA)

I am looking for a way to utilize a computer’s gpu without using cuda (or any installable software). The reason for this is I have an application where the neural network runs on a user’s computer and ...
0
votes
2answers
33 views

How can I explain this chart showing 5-days moving average?

I have plotted the frequency of items sold through time, trying to determine the trends by moving average. I considered a 5-days window. I would like to know if this approach makes sense and how I ...
2
votes
1answer
31 views

Data-preprocessing for Machine Learning model

I am confused about how to preprocess range based category such as age, tumor-size & inv-nodes. Should I take an average of the limits, as in - 14.5, 24.5 and so on or do one hot encoding of the ...
12
votes
4answers
16k views

seaborn heatmap not displaying correctly

for some reason, my heatmap is not displaying correctly anymore! It was working just fine even with 6 classes. Since the last time I used it, I've installed many packages ( including plotly), I don't ...
0
votes
2answers
344 views

problems with installing tensorflow using anaconda

I am trying to use anaconda 4.5.12 to install tensorflow on an Ubuntu 18.04.1 LTS system with python 3.7.1 and pycharm 2018.3.3. I issue the following terminal command: ...
0
votes
0answers
12 views

How to calculate final AUC for sequential combinations of binary classification models in Python

I am working on developing a binary classification model using GradientBoostingClassifier on a highly imbalanced dataset (100:1) that I plan to implement in 2 steps. build a model (M1) that will try ...
6
votes
1answer
1k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
1
vote
1answer
23 views

XGBoost - feature importance just depends on the location of the feature in the data

I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data ...
0
votes
0answers
15 views

Time Series Analysis / Modeling with auto_arima

I recently dived into Time Series and was attempting at doing some data analysis and modeling. I'm using the following dataframe ...
0
votes
2answers
759 views

Which one I should choose for random forest?

I fit a dataset with a binary target class by the random forest. In python, I can do it either by randomforestclassifier or randomforestregressor. I can get the classification directly from ...
0
votes
3answers
39 views

How do i get dummies for this dataset

I am using an udemy course for MachineLearning and I am trying to form a dummy for my variable the column is Country I want to change to France Germany Spain France ...
0
votes
1answer
115 views

Metric (other than RMSE, MSE, etc.) to select the best model to better detect peaks in weather forecasting

I have created multiple regression models and wanted to choose the best one. One common metric would be RMSE, as you know. When I looked at the results, second model (RMSE = 0.15) was better able to ...
1
vote
0answers
22 views

Why is Tensorflow LSTM training slower on a machine with far better components?

Training an LSTM using the exact code and dataset on two different machines with different components yields different results in terms of training time. However, for my case, the results were the ...
0
votes
1answer
10 views

How do feature selection on a sparse matrix?

Say I want to do features selection on a sparse matrix, i.e., 10,000 rows x 1500 features, but the matrix is mostly sparse. Let's say the features are all numeric and the target is binary and discrete....
0
votes
0answers
15 views

Transfer learning with Keras for medical image classification

Good afternoon; I'm trying to do Transfer Learning from pre-trained model on imagenet to solve a classification task with Lung CT slices. These slices are stored in ...
1
vote
1answer
171 views

Saving the hidden layers of a trained network

I have used a pre-trained VGG-19 model to build an image classifier as part of a MOOC. I have implemented a classifier like this: ...
2
votes
1answer
70 views

Linear Regression in Python using gradient descent

I am trying to implement a simple multivariate linear regression model without using any inbuilt machine libraries. So far, I have been able to get a root mean squared error for training about $2.93$ ...
0
votes
2answers
32 views

Validation Curve Interpretation

I have been reading about the validation_curve function from scikit learn. When I run this it takes too long. Therefore, I am plotting the results from grid search ...
0
votes
1answer
23 views

Converting a pandas dataframe into a csv with multiple columns

When I convert my pandas dataframe into a csv it all puts it in one column, I would like it to be in multiple columns. this my code ...
1
vote
1answer
36 views

How to use a custom objective function?

The following is what I'm trying to accomplish: I have a charity contact data set. Each contact has features such as sex, age and so on, which we define as X. Now we are doing a solicitation ...

1
3 4
5
6 7
93