Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (-> stackoverflow).

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Creating table in Databricks using the table from Oracle

I am trying to create a table in Databricks using a object in Oracle Db, but getting an error ...
1
vote
1answer
12 views

Converting binary valued data to boolean or one hot?

I am dealing with a dataset that contains multiple columns (features) that contain binary variables, e.g., a gender feature that contains 'male' and 'female'. I want to apply some supervised learning ...
1
vote
1answer
18 views

Best way to optimize dataframe row by row sum of squared errors calculation?

I am seeing a significant slowdown with the following small snippet of code which computes the sum of squared errors between two dataframes - e.g. it takes approximately 2.5 seconds to run when ...
0
votes
0answers
8 views

Method for predicting future state, based on time spent in previous states

So what I'm looking for is the best approach to predict a future state. Say we have three states: A, B, C. I want to predict if in the next time-interval (f.e. a day or a week) the state will become C....
0
votes
0answers
14 views

Algorithms or statistical methods for detecting trends based on multiple time series

First of all, I have little experience with data science, statistics, and working with time series, so please excuse my perhaps foolish and amateurish question. I have a data set with many different ...
0
votes
0answers
16 views

Regarding pos tagging

I working on a dataset, I did the pos_tagging using nltk. Now I want to know which sequence of grammar is most common in my rows, then I want to define a chunk grammar based on a common grammar ...
0
votes
0answers
20 views

What is the best algorithm for feature selection in categorical variables?

Imagine this question in two parts: Part 1 : My Y variable can be only 1 and 0 Part 2 : My Y variable can be only -1, 0, 1 In both cases my X is continous with float variables. What are the ...
0
votes
0answers
21 views

Where to start with ChatBots?

I want to start my journey into ChatBots and how I can create them. I have read some articles regarding the type of chatbots. Basically there are 2 types of chatbots, one is a rule based and the other ...
0
votes
0answers
21 views

Modeling a probability using scikit

I'm trying to create a model to predict the probability for an object to break in a specific year. I have information for different properties of different of these objects, such the year when they ...
0
votes
1answer
19 views

How to remove outliers properly?

I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a ...
0
votes
0answers
5 views

Can KAMA (Kaufman Adaptive Moving Average) be used for extracting the moving average of numpy arrays?

I'm reading about KAMA algorithm and everywhere I see it's just adopted for stocks and trading kind of things. Since the algorithm looks perfect so I was planning to use this for averaging out a 1D ...
0
votes
0answers
7 views

Ensemble Model for Recommendation Engine

I want to build an ensemble recommendation engine where I can combine Surprise library algorithms like KNN and SVD to achieve the best result. Can anyone know how to ensemble this technique?
3
votes
1answer
35 views

How to merge with smallest Euclidean distance?

Given a main left dataset, how can I merge with right dataset with smallest Euclidean distance (...
0
votes
1answer
54 views

Trying to convert a .txt file into .npy file to split and use for train/validation/test

I have a large data set in .txt form, and was trying to split it into three for testing, validation, and training by transforming the data into a ...
1
vote
0answers
17 views

Bert in imbalanced dataset

I am working in a NLP task for text classiciation with 13 classes. I have different text size in each texts and I was wondering if it's a way to deal with that and generally if it's a problem in Bert ...
0
votes
1answer
23 views

Need help understanding how this Neural Network is working

This is a model I came across, and I need some help understanding how it works It uses South German Credit Prediction data set from Kaggle ...
2
votes
3answers
79 views

What is exactly the difference between Validation data and Testing data

I asked this question on stack overflow and was told that this is a better place for it. I am confused with the terms validation and testing, is validating the model same as testing it? is it possible ...
0
votes
1answer
10 views
0
votes
1answer
12 views

Best approach to find top-k most likely items

I am working with large datasets of papers and authors. I am trying to find top-k authors that are likely to cite a new paper on the unseen dataset (https://www.aminer.org/aminernetwork). My setup is ...
0
votes
1answer
16 views

The Python TSAI library for time series -- how can I make it less verbose?

I tried to RTFM, here is the documentation: https://timeseriesai.github.io/tsai/ I found no mention of how to get rid of the table that is output while training. Here is an example of what I do NOT ...
0
votes
1answer
16 views

How to assess a trend during time?

I wanted to know which analysis method would you suggest for assessing a trend during years. Let us say we have data on antibiotic resistance in a particular bacteria from 2010 until 2021. How can we ...
0
votes
2answers
35 views

Merge two dataframes on multiple columns, only if not NaN

Given two Pandas dataframes, how can I use the second dataframe to fill in missing values, given multiple key columns? ...
1
vote
1answer
18 views

I have two dataframes, df and df1, I want to check if df1 columns values is present in df columns and print the entire row of the df if present

first df: df = pd.DataFrame({'col1':[1,2,3,4], 'col2':[5,6,7,8], 'col3':[9,10,11,12], 'col4':[13,14,15,16]}) second df: <...
0
votes
0answers
3 views

How to specify Search Space in Auto-Sklearn

I know how to specify Feature Selection methods and the list of the Algorithms used in Auto-Sklearn 2.0 ...
0
votes
0answers
10 views

Anomaly Detection with totally Categorical features

I am working on an anomaly detection project that aims to discover which merchant is the point of compromise. The data contains no numerical value and it looks like this; date account merchant fraud ...
0
votes
0answers
10 views

How to stack Transfer Learning models in a Sequential

To make a nice architecture, I wanted to stack Transfer Learning models one over the other. The three models I wanted to stack were : VGG16 InceptionV3 Resnet50 So, I defined the three models as ...
1
vote
1answer
18 views

Feature engineering of timestamp for time series analysis

Following a Tensorflow time series analysis tutorial, I came across a particular way of converting data timestamps into a time-of-day periodic signal, that could help the model interpret the data ...
0
votes
0answers
10 views

What's an appropriate clustering quality estimate / metric for precomputed distance in HDBSCAN?

HBDSCAN supports estimation of clusters from precomputed distances. However, the python implementation of HDBSCAN (scikit-contrib) doesn't create minimum spanning trees in the absence of raw data when ...
1
vote
0answers
39 views

Workflow for stock prediction in machine learning

I'm trying to find the best workflow for a stock prediction problem. My idea goes as follows : I will use a classfication and a regression at the same time Classification (-1 ; 0 ; 1) Regression (...
0
votes
3answers
42 views

p-value of chi squared test is exactly 0.0

I need to do a chi square test of two of my dataset's categorical variables. This two variables have basically the same meaning but comes from two different sources, so my idea is to use a chi square ...
0
votes
0answers
13 views

Backing into a statistical significance level for a proportion

I'm doing some natural language processing which involves assigning labels to a sample of news articles. Its multilabel classification so an article might receive label1, label2, both, or neither. I'...
0
votes
0answers
23 views

Adding a static layer to LSTM text generation

I am using Tensorflow guide to build a LSTM text generation model (uses the keras functional API) (https://www.tensorflow.org/text/tutorials/text_generation) I would like to add an additional input ...
0
votes
0answers
10 views

How to find duplicate rows in a column then find out if two cells in another column sum up to a third cell in an Excel tab in Python?

I need to find all duplicate rows (string values) in "Name" column and then find out if two numerical values in "Amount" column sum up to a third value also in the "Amount&...
0
votes
1answer
24 views

New client's future profits prediction

Problem: I want to evaluate the efficiency of new clients' acquisition. For that reason I want to be able to forecast the profit generated by client (in let's say 12 months from acquisition month) and ...
-1
votes
1answer
22 views

datatypes as input in machine learning algorithms

I have a dataset and I am trying to perform binary classification by using different machine learning algorithms I have seven columns as input where all are int64 except of one that is float64. So my ...
0
votes
0answers
3 views

Plotting Deviance Residuals and Leverage of GLM Model Using H2O

Is it possible to plot the deviance residuals and leverage (e.g. cook's distance) of every observation fitted in a GLM model using H2O? From H2O's documentation, seems it only calculates the sum of ...
1
vote
0answers
18 views

How can I plot the covariance matrix of scikit-learn's Gaussian process kernel?

How can I plot the covariance matrix of a Gaussian process kernel built with scikit-learn? This is my code ...
0
votes
1answer
9 views

Retrieving the ordinal encoding of a variable after it's placed in a pipeline/columntransformer

I am applying ordinal encoding to a dataset through a column transformer - how can I retrieve the ordinal encoding of a feature (e.g. Area)? ...
-1
votes
0answers
9 views

Training and Validation loss are same but not decreasing for LSTM model

I have a timeseries data and I am doing univariate forecasting using stacked LSTM without any activation function, Like following. ...
1
vote
0answers
16 views

Is there a reduced form of the dot product in numpy? [closed]

I am trying to find a numpy operation that gives me the scalar products between all vectors of a 2d array at index i of a 3d array and the vector at index i of a 2d array. Let me give you an example ...
0
votes
0answers
11 views

TensorFlow Speech Emotion Recognition Model gives same prediction for all inputs

Dataset used: RAVDESS (I've only used the audio only files) Here's a sample after I've processed the data: And the code for the label encoding: ...
0
votes
0answers
11 views

How to Find the Next Moves of Cars From Given First Moves?

I want to find next moves of cars from the previous moves, but I could not figure out what should I use as algorithm. Can you help me to find a way to solve this problem? I have a lot of car data like ...
0
votes
1answer
10 views

Pandas and data imputation from a multiindex dataframe

I have the following dataframe: ...
0
votes
1answer
21 views

model.predict gives the same output for all images

I am trying to create a model using resnet50 to classify ct scan images as covid or not. However when using model.predict with a given image its giving the exact ...
0
votes
0answers
6 views

Isomap/LLE python library that allows precomputed distance matrix

I want apply some algorithm to the distance matrix before reducing dimensionality using Isomap, LLE and Laplacian Eigenmaps. Are there Python libraries I can use for this?
0
votes
0answers
6 views

H2O Python H2OModelSelectionEstimator

I want to try H2O's Model Selection function in Python, but cannot load the library for some reason. The following code failed: from h2o.estimators import H2OModelSelectionEstimator Error message: ...
0
votes
0answers
27 views

Xgboost taking some time to run vs hyperopt

Sorry for long post,im triying to run a xgb model but for some reason takes like 20 to 30 min(per run) with a specific set of hyperparams, but when i run hyperopt to get best params, takes like 7 ...
0
votes
1answer
12 views

How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise

I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly. My goal is to write a short text embedding using vector ...

1
2 3 4 5
120