Questions tagged [python]

Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).

Filter by
Sorted by
Tagged with
0 votes
1 answer
16 views

How to implement a customized DataLoader that inherits pytorch's one?

I need to implement a customized DataLoader, that inherits from torch.data.utils.DataLoader. I have searched it for half hour, but there is no example or doc about this. What methods of it should I ...
  • 101
0 votes
0 answers
14 views

Can Shapley values be used with a simple passthrough model to evaluate real-world data as a model, but without an actual model?

I currently run many of the ML models I keep through Shapley Additive Explanation (SHAP) values. I am curious about analyzing a raw dataset using SHAP values. Why shouldn't I be able to construct a ...
  • 1
0 votes
0 answers
15 views

setting width range for peak detection

How to set width range from 750 to 1250 for example that peaks only appear in this area ? My code: ...
1 vote
0 answers
56 views

How do I use a machine learning or AI algorithm on this kind of data?

I have a text data set with keyword, thematic, Sub thematic, intent Now I have to train a keyword then first classify into sub thematic, then into thematic and then into intent. the data looks like ...
  • 11
0 votes
0 answers
29 views

Sklearn Lightgbm 3.3.2 API

I have a model object from a model and am trying to pass in the dataframe for scoring, it is kicking me an error: ...
  • 101
0 votes
0 answers
22 views

Sklearn Bisecting Kmeans prediction issue with processor?

I'm trying to predict a query vector to observe which cluster it belongs to using the SKlearns Bisecting K means algorithm. I get an issue with my console saying Windows fatal exception: int divide by ...
  • 11
0 votes
0 answers
237 views

Unable to load a .h5 keras model

I trained a multi-class Unet model using keras built-in metrics and loss (Accuracy and categorical_crossentropy). After the end of the training, I saved the model with the following command: ...
  • 111
0 votes
1 answer
52 views

XGBoost training on sample of time series data

I am new to XGBoost and would like to use it on a time series dataset. Here is the scenario I'm faced with: The data set contains N samples of length T, with N>>T. I'd like to train an XGBoost ...
2 votes
1 answer
45 views

How to remove noise from signals?

I have sensor which outputs signals (two signals bellow for example). I use 2000 signals as my data, which some of them are clear and some of them are bad signals. All clear signals have peaks, and ...
0 votes
1 answer
49 views

OpenCV isn't processing all the images in a directory

I'm really trying to apply CLAHE to a directory of about 700 images. However, once I manage to get it running, for some reason the code stops before all the images are processed. When I run it on ...
1 vote
0 answers
56 views

How do I Classify text data with multiple sub labels? [closed]

If I have data that was like this How can I train keywords and try to classify them into these labels what was the best algorithm to try this and labels are not limited to 4 might increase and some ...
1 vote
0 answers
29 views

how to reduce the loss and improve the gradient flow - CNN

I am trying to improve this situation, in image classification[3 classes, softmax in the last layer], I constructed the neural network having 7[conv2d+Batchnormalization] layers + 1 linear layer, ...
4 votes
0 answers
51 views

Does ROC AUC different between crossval and test set indicate overfitting or other problem?

I am training a composite model (XGBoost, Linear Regression, and RandomForest) to predict injured people probability. Well, the results of cross-validation with 5 folds. Well, I can see any problem ...
0 votes
1 answer
68 views

How to increase FER2013 dataset validation_accuracy for only 3 classes i.e, happy,sad,neutral?

I am building a face emotion detection model using vgg16. Using FER2013 dataset for 7 classes i am getting= train_accuracy=97%, validation_accuracy=90%. but when i tried with 3 classes i.e, happy,sad,...
0 votes
0 answers
35 views

Self-Attention: Tensorflow to PyTorch Code Conversion

I have a TensorFlow encoder code with a Self-attention layer X = layers.Attention()([X,X]) #Single-head Self-attention layer What is the equivalent PyTorch code? ...
  • 101
0 votes
1 answer
22 views

best clustering algorithm or model for clustering areas on map?

I have a database that has information such as Latitude, longitude, plus other information such as sightseeing locations, restaurants and shopping centers, if it's rural or suburb,... It also has ...
  • 1
5 votes
2 answers
566 views

How to build a personalized recommendation system using real life scenario data?

I am trying to build a recommendation system to recommend items to users. This is the kind of real-life example I am trying to implement. But anywhere I searched about making a recommendation system ...
0 votes
2 answers
53 views

Is R programming a good way to start with Data Science? [closed]

I am a software engineer (currently CTO) specialized on web and mobile applications picking up data science skills. I do this mainly for future projects within my startup that works in digital ...
  • 3
0 votes
0 answers
14 views

Pyspark install failing on Mac OSX

Not able to run pyspark, spark-shell, apache-spark on my mac after attempting install. At ...
0 votes
0 answers
12 views

How to recommend movies for users which are not in trainset with Surprise SVD Matrix Factorisation algo?

I'm trying to implement a Surprise SVD movie recommender system and finally deploy it on a website, where the user would rate, say 10 movies and get back the top 10 recommendations. Since I already ...
0 votes
2 answers
80 views

How to check the impact of predictive Model

Suppose that I build an email marketing campaign where the goal is to predict which members will make purchases or not and based on that change the marketing email campaigns. After we build the model, ...
0 votes
1 answer
93 views

Perform k-means clustering over multiple columns and get the cluster center values?

I read here how to show the number of clusters over $n$ columns. I would like to know how to get in a table, the values of the clusters centers. Could someone help me with this?
1 vote
2 answers
100 views

How can I do mathematical operations to two columns of a CSV file and save the result in a new CSV file?

I have a CSV file (such as test1.csv). There are tabular values like the following. S1 S2 S3 4.6 3.2 2.1 3.2 4.3 5.4 1.4 3.4 6.1 I want ...
  • 131
0 votes
0 answers
16 views

Text mining technique for similarity index (account for spelling mistakes)

I only use Python Context: This is my first time ever doing this so I have no idea where to start. I have a list of 20,000 words. It can contain emotes or different languages. I need to find a way to ...
1 vote
2 answers
41 views

How can I create new dataset maintaining the same trend from an existing dataset?

I am trying to create several new datasets based on an existing dataset. The new datasets should follow the trends (i.e., the correlation within the data) of that existing dataset. As an example, I ...
  • 131
1 vote
2 answers
41 views

Finding max profit from buying and selling items (between an array of prices for a set of specific days) with limited storage capacity

My goal is to find the max profit that I can make from buying and selling items between days given an array of item prices for a set of specific days, i.e. {2.91, 2.775, 2.855, 2.89, 2.595, 2.99, 2....
  • 11
1 vote
1 answer
31 views

My model is predicting values ​for only two labels (instead of 9) how to fix it?

My goal is to predict the count (variable y) based on various features (variable x). My y is most of the time (98.4%) equal to 0, so this data is inflated by 0. Based on this premise, I thought that ...
0 votes
0 answers
46 views

AWS Stepfunctions using SageMaker ScriptProcessor

Hello Data Science/Engineer folks, I need a help in orchestrating a ML model using AWS Step Functions. There are some tutorials on using StepFunction with Python SDKs, but all of them are focused ...
0 votes
1 answer
129 views

Which deep learning model is best in terms of instance segmentation and Object detection both?

I am trying to find most efficient and robust Object detector+Segmentation model. I came to know about Mask-rcnn, Yolov5, Yolact, yolov7. As, YOlov7 is new and i read somewhere that yolov7 surpasses ...
  • 219
0 votes
0 answers
14 views

Binary classification of video segment

Are there models that take a small video stream of 5-10 seconds as input (not a single frame, specifically video), and predict is there was an some kind of anomaly in this video segment or not?
0 votes
1 answer
79 views

Why does my mean average precision metric show as 0.000e+00?

I have an object detection model with my labels and images. I am trying to use the tensorflow ranking metric for MAP, https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/metrics/...
  • 25
1 vote
0 answers
21 views

Algorithm Comparison for Influencers Ranking

I am working on ranking social influencers on Instagram according to their influential power with the metrics collected below. Metrics collected: username categories (the niche the influencer is in) ...
  • 11
0 votes
0 answers
24 views

Why is my LSTM prediction is saturated and have bad prediction?

I am new to deep learning. Currently, I am trying to predict torque based on its past values using an LSTM model. There are two datasets (generated from a scaled test), one with wear and the second ...
0 votes
0 answers
8 views

Is it problematic to set new_state = action in Reinforcement learning PPO and episode length = 1 (1 step per episode)

I am writing my master's thesis. I work for an institute and we tackle a RL optimisation problem. We want to minimise infected animals in an epidemic outbreak. Is it feasible that our state is ...
1 vote
1 answer
19 views

Plot a matrix as a single point in space

I have a dataset of drugs represented as a graph, each of which is described by three non-square matrices: edge index (A), an 2xe matrix, where e are the bonds of the molecule, the first line ...
1 vote
1 answer
120 views

Clustering for Sparse Data Matrix of high dimension

I currently have a dataset of 1000 entries with 512 features that are sparse. I want to cluster them. I have attempted using kmeans, but found that the clustering wasn't very good, and have been ...
  • 11
1 vote
0 answers
37 views

Best Practice to handle large data in HDBSCAN cluserting using tfidf

I am now about to implement HDBSCAN on a very large set of data, a few million sentences, using tfidf. However, as we know ...
  • 411
0 votes
1 answer
26 views

Hello guys, is dimension reduction required for tensorflow? [closed]

I am working on face emotion detection using FER2013 dataset using tensorflow and vgg16 model. I am applying t-sne to my training dataset for dimensionality reduction. My question is that "is ...
0 votes
1 answer
448 views

Found input variables with inconsistent numbers of samples: ValueError

Today I am trying build ensemble model. Where I am working with iris dataset. In my model I am using ...
0 votes
0 answers
15 views

Grouping persons to groups with minimal size based on their selection with making minimal changes of original selection

Problem: I have dataset with each row representing selection of person of several items (number of items is limited). I need to group persons to groups with predefined minimum size (20) with making ...
1 vote
1 answer
74 views

Decision tree implementation in python that correctly handles categorical variables

Is there a Decision tree implementation in python that correctly handles categorical covariates? By "correct" I mean that it is able to send any subset of category levels down one daughter ...
  • 749
0 votes
1 answer
720 views

progress bar for GridSearchCV

I am building CNN algorithm that will output some values. I am using GridSearchCV for parameters tuning and I want to implement progress bar for handling with large datasets but I do not know how to. ...
0 votes
0 answers
59 views

Is it possible to use a python script to open and run a Tableau Prep Flow?

My organization utilizes a tableau dashboard embedded in our webpage, which is manually updated on a regular basis. I am working on automating this process, with the hope of having it updated ...
0 votes
0 answers
208 views

Keras MultiHeadAttention layer: Dimension value must be integer or None or have an __index__ method, got value 'TensorShape([None, 6, 8])'

I am running into a lot of trouble trying to use Keras MultiHeadAttention. My model: ...
0 votes
0 answers
13 views

How can I use the torchtext pad_token tool, or what can I use instead?

I want to test Captum on my nlp model that classifies texts to understand how it classifies them. For that I use the Captum tutorial which uses a pad_token from <...
0 votes
1 answer
18 views

Search max and min value in Date Range cols

I have a pandas df. The index, 'Date', is dt and is continuous daily data. I have two other dt cols, 'startDate' and 'endDate'. For each row I need to find the max and min of 'Value1' between ...
  • 1
0 votes
1 answer
39 views

How to calculate the distance matrix when there is a lot of data And the memory does not support it

I am dealing with a data matrix in which most of the variables are binary or multilevel response. I would like to perform the MDS algorithm and for that, I need to calculate the distance matrix first. ...
  • 302
0 votes
1 answer
226 views

TypeError: object of type 'NoneType' has no len() when implementing neural network

I am building artificial neuron network (ANN) model for predicting values but facing problem: Input: ...
0 votes
0 answers
14 views

Approach to aggregating & creating subtotals for each new record / subgroup rather than doing a full "recalculation" over full dataframe each time

Just wondering if I could pick the brains of the many experts here. I am fairly new to programming in Python so please excuse the novice level question. I am currently working on a machine learning ...
0 votes
0 answers
15 views

Task Distribution

I would like to distribute tasks (going to companies and signing a deal) to sales representatives based on some condition like company location (is it near the representative's place) or profit of the ...
  • 1

1
3 4
5
6 7
129