Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
2
votes
2answers
99 views

Reasons that LIME and SHAP might not agree with intuition

I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making ...
0
votes
0answers
13 views

Keras loss function cost matrix / misclassification penalty

I'll explain my problem through a simplified example. Let's say I have a matrix $X$ of a single variable with the following values ...
1
vote
1answer
38 views

Making outputs of sub-models the inputs of other models LSTM

I'm trying to create a neural network that is composed of different sub-models. Each sub-model have their own inputs, which some of them can be shared between the sub-models. For example, sub-model 1 ...
0
votes
1answer
476 views

How can I extract the residual array from the Scikit Learn PCA routine?

The Nipals PCA algorithm calculates the scores and loadings of a data array iteratively. If only (say) 3 scores and loadings are calculated from a data array with more than 3 variables, there is a ...
6
votes
0answers
115 views

Are there any graph embedding algorithms like this already?

I wrote an algorithm for generating node embeddings based on the graph's topology. Most of the explanation is done in the readme file and the examples. The question is: Am I reinventing the wheel? ...
0
votes
1answer
16 views

RAM crashed for XML to DataFrame conversion function

I have created the following function which converts an XML File to a DataFrame. This function works good for files smaller than 1 GB, for anything greater than that the RAM(13GB Google Colab RAM) ...
0
votes
1answer
12 views

How to run two different models in single frame?

I have mask_detector.model and yolov3 social distancing weights. I want to run them simultaneously with a single webcam stream. how can I run them both i.e. detecting mask and social distancing model ...
2
votes
1answer
59 views

Item-based recommender using K-NN

I'm trying to build an item-based recommender using k-nn. I have a list of items, all of which have some properties (features) in common. ...
14
votes
3answers
4k views

Are there any good out-of-the-box language models for python?

I'm prototyping an application and I need a language model to compute perplexity on some generated sentences. Is there any trained language model in python I can readily use? Something simple like <...
15
votes
4answers
25k views

How to initialize a new word2vec model with pre-trained model weights?

I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset ...
1
vote
1answer
152 views

ValueError trying to use a pickled scikit-learn model

I am new to data science and trying to learn something. I was able to complete the prediction with 98% accuracy and i saved it as pickle model. Now while trying to predict using this model I am ...
1
vote
1answer
21 views

How to find slope of curve at certain points

how to find slope at certain points circled in blue in below curve ? Are these below 2 approaches valid ? though they give different results . How to automatically find the points where the slope ...
0
votes
0answers
10 views

Prediction of Bike Sharing Count Day-wise using Time Series Analysis ARIMA [closed]

My Kaggle Notebook Dataset Link I'm trying to predict the "cnt" variable value on a daily basis. Data is hourly data for 2 years, so I'm taking window size as 24. I'm getting p value as ...
0
votes
2answers
186 views

Remove Outliers from Dataframe using pandas in Python

I would like to remove outliers from my dataset. It looks like this: ...
23
votes
5answers
11k views

Improve the speed of t-sne implementation in python for huge data

I would like to do dimensionality reduction on nearly 1 million vectors each with 200 dimensions(doc2vec). I am using TSNE ...
0
votes
1answer
28 views

How to handle unseen labels in test data?

I get something like TF-IDF of training corpus in python with (something like) TfidfVectorizer. In test data some features (here are the words of test corpus, every word is a feature) are not seen in ...
0
votes
0answers
16 views

Predict how many days until New Purchase

I am trying to introduce myself into predicting with dates and I am having big troubles understanding how to do it. I have a dataset with customer orders from an e-commerce platform with the usual ...
1
vote
0answers
15 views

How can I use the Brier Skill Score in cross-validation for imbalanced data?

I wish to use search for model hyper-parameters (by a grid search) using the Brier score as the scoring method (see code below): ...
-1
votes
0answers
14 views

Understanding the modified version of weighted_categorical_crossentropy() function for keras model?

I had found this piece of code as a solution to solve class imbalance. I am trying to understand it but I still have some ambiguity. Here is the found code: ...
2
votes
1answer
1k views
0
votes
0answers
7 views

Apyori module installation

I am currently trying to import the apyori module in PyCharm. However, I am getting the ModuleNotFound Error. i have installed the module by typing pip install apyori on my terminal but I am still ...
0
votes
0answers
11 views

create and use weights in Python to perform weighted correlation and PCA

I need help with the principal components’ analysis code below in 3 ways: Write Code to create RIM (RAKE) weighting. I’m trying to ensure that using the code below, I get exactly the same results than ...
2
votes
2answers
731 views

How can I find anomalies in each row of data?

I have some reported data I want to spot anomalies on. The columns are a facility name then monthly reports of that given facility. ...
0
votes
1answer
257 views
0
votes
2answers
41 views

Drawing validation set from test set

I am building a 3 neural network models on dataset that is already separated to train and test sets. From my analysis, I found that this dataset has values on test set which don't exist in the train ...
0
votes
1answer
11 views

Collinearity between continuous and categorical variable

I have a medical dataset with features age, bmi, sex, gender, # of children, region, charges, smoker. Here smoker, gender, sex and region are categorical variables and others are numerical features. ...
-2
votes
0answers
24 views

How can I improve my regression problem?

I'm trying to forecast new covid-19 cases in Chile, but I feel that my model is not doing so well. My approach was to use a neural network and the model is as follows: ...
0
votes
0answers
17 views

Why do all DTW-packages use different step-patterns and distance-methods?

I already posted it on StackOverflow. I was not sure if this was the right place. I have several questions about the algorithm DTW. I tried different Python-Packages DTAI-Distance DTW-Python First ...
0
votes
1answer
94 views

No gradients provided for any variable: ['Variable:0', 'Variable:0']

I am using Python 3.6 and Tensorflow 2.0 for the following code for linear regression: ...
0
votes
0answers
50 views

Non-OK-status: GpuLaunchKernel(…) status: Internal: no kernel image is available for execution on the device [closed]

I run my code on tensorflow 2.3.0 with CUDA Toolkit 10.1 CUDNN 7.6.0 (Windows 10) and it returns a issue: ...
26
votes
4answers
12k views

Word2Vec for Named Entity Recognition

I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named ...
0
votes
0answers
12 views

'TabularLIME' is not defined Azure DataBricks

I have just started working on Azure Data bricks. I am facing some error while running a already created Python NoteBook. Here is the code: ...
1
vote
1answer
29 views

Is there a python version of Mitchell's face pose recognition code?

The original code for chapter 4 in Tom Mitchell's book for recognizing face pose from images is here: http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/faces.html But it's in C, not that there ...
1
vote
0answers
29 views

Is it possible to do attribute, value extraction prediction model in Machine Learning?

Previously I asked a question at here, but it doesn't seems to be at the correct place. So, I moved it here with more details of what I've done. Here is the sample data image to be process. Start with ...
2
votes
2answers
988 views

Piece-wise regression by clustering

I was wondering about the possibilities of clustering numerical data (more than 3 dimensions) into different clusters and doing curve fitting on each cluster to get much higher accuracy than using a ...
0
votes
1answer
52 views

How to use historic data (granularity at day level) for ML modeling?

There is a scenario where I have to use historic data which is at the day level for the past 5 years.Actually it is water flow data, what quantity of water was flown on that particular day. I have to ...
0
votes
1answer
44 views

Pandas - Avoid boolean result when using groupby()

I have this script: sectors = df.groupby(['company_sector']).mean()['investment_in_millions'] Output: I wanted to keep the same groupy() but having a result ...
13
votes
4answers
5k views

How do I make an interactive PCA scatterplot in Python?

The matplotlib library is very capable but lacks interactiveness, especially inside Jupyter Notebook. I would like a good offline plotting tool like plot.ly.
1
vote
1answer
49 views

Why is the difference of datetime = zero in the following dataframe?

This issue that I am facing is very simple yet weird and has troubled me to no end. I have a dataframe as follows : ...
3
votes
3answers
8k views

Open-source interactive dashboard in Python

I am trying to find a package to construct a dashboard with interactive graphs (including widgets such as sliders) in python (mainly IPython notebook). I know there is plotly but I would like a fully ...
1
vote
0answers
688 views

Training Word2Vec with names instead of sentences

I have scientific database with articles and coauthors. using this database I am training word2vec model on co-authors. Use use case here is to disambiguate authors. I was wondering my approach here ...
1
vote
1answer
82 views

Python sklearn PCA transform function output does not match

I am computing PCA on some data using 10 components and using 3 out of 10 as: ...
4
votes
1answer
695 views

Parking Prediction based on Mobile application

I am quite new to predictive modelling but have knowledge of GIS, R, python, SQL, etc. I am currently doing a project in work trying to predict when parking spaces will be available based on data ...
-2
votes
2answers
147 views

R + Model + web service

I would like to interface R with some other language (C#/Python/anything easy to use) to provide it with a web service (or other technology) to an other application developed in C#. R => black box ...
7
votes
4answers
3k views

Determinate K in K-Means Clustering

I have salary data of several user (Python list). Now I am using KMeans to cluster them. Given this data, Is there a way to figure out the best value for 'K' automatically through program? I tried ...
15
votes
6answers
15k views

Good “frequent sequence mining” packages in Python?

Has anyone used (and liked) any good "frequent sequence mining" packages in Python other than the FPM in MLLib? I am looking for a stable package, preferable stilled maintained by people. Thank you!
8
votes
4answers
12k views

How do we standardize arrays with NaN?

I used StandardScaler() to standardize data so far, but this doesn't work with NaNs. None of the other methods I know of (MinMaxScaler, RobustScaler, MaxAbsScaler) work with NaNs either. Are there ...
1
vote
1answer
33 views

How to interpret skimage orientation to straighten images?

I have a bunch of images that I am trying to straighten so the images are horizontal (major axis is horizontal) but I don't understand the orientation output from ...

1 2
3
4 5
95