Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

1,364 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
14
votes
0answers
843 views

Visualization of multiple Markov models

I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan. Most often single models are visualized with a decision tree or transition ...
6
votes
1answer
3k views

sklearn: SGDClassifier yields lower accuracy than LogisticRegression

I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement ...
5
votes
0answers
1k views

Why is my Keras model not learning image segmentation?

Edit: as is turns out, not even the model's initial creator could successfully fine-tune it. This is most likely a problem of implementation, or possibly related to the non-intuitive way in which the ...
5
votes
1answer
2k views

Recurrent neural network multiple types of input Keras

For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks....
4
votes
2answers
84 views

Trained Tensorflow model performs poorly on inference

I trained an image classification model using Keras with Tensorflow backend. The model got good accuracy on validation dataset as well as on the testing data, I save the entire model to ...
4
votes
1answer
83 views

How to predict value in every 120 minutes using LSTM in python

I want to predict value in every 120 minutes continuous using LSTM model. Here I wrote the code for predction. But I'm not getting proper prediction values . Here from start time I need to predict ...
4
votes
1answer
313 views

Find matching text from a text column

This is my first time to use Data Analytics tool to figure out a solution to a problem. I have a table with following columns ...
4
votes
2answers
143 views

Text topic classification in tensorflow

I want to create a CNN in tensorflow that does the following: Classify a recipe headline and find out the topic. For instance <...
4
votes
0answers
242 views

Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

I've been searching around for an explanation to this, and haven't come across one yet- in scikit-learn, when I compute the auc() of the ...
4
votes
1answer
458 views

HMMLearn Predict Next Observed Event

From my understanding you can use the transition matrix to predict the probability of going from the last predicted hidden state(state t), to the t+1 hidden state. My confusion is how in code format ...
4
votes
1answer
3k views

Memory error on using data generator in keras

I am using the following augmentations on dataset of size 9 GB: ...
4
votes
2answers
10k views

Multivariate Time-Series Clustering

I have a streaming data along with timestamp dataset that looks like this: 1.png Timestamp can be inclusive of "seconds" too, but the data may or may not change every second. it depends on the ...
4
votes
2answers
20k views

Is there any way to read Xlsx file in pyspark?Also want to read strings of column from each columnName

pd is a panda module is one way of reading excel but its not available in my cluster. I want to read excel without pd module. Code1 and Code2 are two implementations i want in pyspark. Code 1: ...
4
votes
1answer
390 views

How to implement patternet in python as it is in matlab?

NOTE: This question was first posted in cross-validated website but I was instructed to move it off that website as it was not a good fit. I am new in implementation of machine learning, neural ...
4
votes
0answers
1k views

How does Seaborn calculate error bars when using estimators other than the arithmetic mean?

If I create a barplot using Seaborn and specify the geometric mean or the median as the estimator, does Seaborn know to use the appropriate standard error formula to create error bars?
3
votes
0answers
57 views

Are there any graph embedding algorithms like this already?

I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples. The question is: Am I reinventing the wheel? ...
3
votes
1answer
50 views

The output of Model from Decision Tree and Random Forests are different?

I have been made a model using both Decision Tree and Random Forest. But, when I tried to test the model on the same DataFrame the output is different. How is this possible? The data file from my ...
3
votes
1answer
112 views

Fast Python implementation of the gradient descent

I'm looking for fast Python implémentations of gradient descent optimization algorithm. I have a convex problem , with no constraint, so for now I'm using the BFGS algorithm implemented in scikit-...
3
votes
1answer
529 views

How do I create a Keras custom loss function for a one-hot-encoded binary classifier?

I have a CNN binary classifier with one-hot-encoded labels that I've written using Keras and it's just not training to the metric I want to encourage. My data is very imbalanced (91% class 0, 9% class ...
3
votes
0answers
217 views

How to apply rolling volume profile function with a pandas TimeDelta

I am setting up a volume profile series over a stock data. I have implemented the market profile code from this github repo. Here the author gets slice the index and apply the main function by ...
3
votes
0answers
60 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
3
votes
1answer
336 views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
3
votes
0answers
71 views

Feature scaling worsens performance?

I noticed that feature scaling can destroy completely neural networks performance in some cases. Below are my results that you can reproduce easily. I use a neural network to approximate function $g$ ...
3
votes
0answers
49 views

Is there an algorithm that imputes missing values based on n nearest columns? (KNN hybrid)

I have a dataset of 70 columns that have missing values. Each column has a few columns (3-5) that it is significantly more correlated than the others but each column's correlated columns are very ...
3
votes
2answers
67 views

Is ARIMA appropriate for time series prediction involving a mix of explanatory and independent variables?

I have a table with the following columns: Date(Month,Year), Sold_Past_Month, Quantity_Available, Quantity_Shipping_In, Missed_Sales, Quantity_Needed Quantity_Needed is the dependent variable that ...
3
votes
1answer
144 views

How can I detect partially obscured objects using Python?

I'm building a computer vision application using Python (OpenCV, keras-retinanet, tensorflow) which requires detecting an object and then counting how many objects are behind that front object. So, ...
3
votes
1answer
82 views

Pandas DataFrame Rollup Error

I am trying to compute Percentage Contributions on a pandas dataframe. I have a data frame which has three columns (Buyer, ...
3
votes
1answer
113 views

ML regression poor performance

I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very ...
3
votes
1answer
641 views

Imbalanced dataset with 3 classes xgboost scale_pos_weight parameter

The xgboost classifier states the use of parameter scale_pos_weight for 2-class problems. I have a highly imbalanced dataset with 3 classes. Classes '1' and '-1' ...
3
votes
0answers
377 views

FREAK feature extraction OpenCV

I want to know how to use FREAK feature extraction in python, I read the documentation but I need some examples. I wrote this code : ...
3
votes
0answers
897 views

ValueError: Numpy arrays that you are passing to your model is not the size the model expected

I am trying to perform concatenation on the Bidirectinal LSTM layer. I have my model defined like this: ...
3
votes
0answers
169 views

What are the rules when extracting SVO triples from preprocessed text?

If you have some already preprocessed text that is tagged, what are the rules to extract SVO triplets if you want a triple like (word, word, word). Can you give the sentence as example and extract all ...
3
votes
0answers
115 views

Genarate one hour time interval array using pandas in python (import from csv) to predict next value

I am trying to generate one hour one hour time interval to predict next value according to my data set imported from csv file. Here according to the time it will give outputs include in x column. This ...
3
votes
0answers
39 views

Fully connected layer output explodes, but weights, gradients, and inputs all have sane values

I'm trying to train a GAN, and the architecture includes a fully connected layer before the output activation function. In my case, by the second training iteration this layer's output always explodes....
3
votes
0answers
44 views
3
votes
0answers
34 views

Tuning Lexicon Sentiment-values Using Machine-learning

I'm constructing a sentiment-analysis model using the lexicon-based approach, and wondering if I can tune the weights of each word-feature in the lexicon using machine-learning. Is this achievable ...
3
votes
0answers
2k views

Isolation Forest Feature Importance

As of scikit-learn version 0.19.1, there is no implementation for calculating feature importance in an Isolation Forest. I'm also having trouble finding any online resources proposing ways to get at ...
3
votes
0answers
345 views

Neural Network Prediction regression task, output is a multiple factor of input with same peaks

When I missed some details please point this out. I made a simple sequential LSTM model for regression. The model loss is 3.2145e-06. The data is scaled between 0 and 1. I tried different variations ...
3
votes
0answers
78 views

Pandas: Calculating the best seller to incorporate number of houses sold

I have been analysing seller data and trying to get insights. I have written a groupby statement to get the average price of selling for every seller ...
3
votes
1answer
520 views

Keras bug NasNetlarge no top

I am trying to use NasNetlarge in Keras without the top but I cant get rid of the top: ...
3
votes
0answers
66 views

Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning. Should I include the prediction matrix (ie. df containing columns of prediction results ...
3
votes
0answers
257 views

Multiple activation functions with TensorFlow estimator DNNClassifier

I just want to know if is it possible to use tf.estimator.DNNClassifier with multiple different activation functions. I mean, could I use a DNNClassifier estimator which use different activation ...
3
votes
3answers
1k views

Are there any good NLP APIs for comparing strings in terms of semantic similarity?

I want to create a chatbot which informs the user about traffic at the streets but not in real-time for the moment. I have created a small database with MySQL which has some data stored regarding ...
3
votes
0answers
135 views

How to do give input to CNN when doing a text processing?

As a signal processing engineering and being new to NLP, I am confused with giving input to CNN network. With my knowledge of CNN, I am trying to build a classifier for ethnicity with inputs as text ...
3
votes
0answers
113 views

Create top 10 index fund based on >100 stocks

Data I have a zip with ca M=160 csv files, with one csv file per stock. The ticker symbol is in the file name. Each csv file contain the same columns: date, txVolume(USD), txCount, marketcap(USD), ...
3
votes
0answers
343 views

Anomaly detection in cooling process data without exact labels

I have a data set where I look at the cooling of a process. The starting temperature may vary between 580 and 180 degrees. I know that at some point the cooling system failed (see examples in the plot)...
3
votes
1answer
2k views

Export Python Code from Orange

Since Orange is just running python code under the hood... is it possible to set up and do my analysis in Orange but then export or output the underlying code for documentation or replication?
3
votes
1answer
237 views

Why is my U-matrix visually not separating the classes?

I am visualizing the U-matrix generated using a Self Organizing Map codebook to (visually) identify regions of similarity in the data. Although I would like to use SOM to identify clusters in ...
3
votes
0answers
807 views

Fine tuning accuracy lower than Raw Transfer Learning Accuracy

I've used transfer learning on Inception V3 with ImageNet weights on Keras with Tensorflow backend on python 2.7 to create an image classifier. I first extracted and saved the bottleneck features from ...
3
votes
0answers
2k views
+50

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...