Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,669
questions
0
votes
0
answers
3
views
Is it bad to average several MAEs calculated from chunks of a big test dataset?
In my regression problem, I am using Mean Absolute Error (MAE) as a metric for my network. My test dataset is too big to fit in memory, so I am reading the test dataset in chunks and then Keras' ...
- 193
0
votes
0
answers
5
views
seasonal adjustment to testset
I would like to understand how to apply a seasonal adjustment to my testset. Let's imagine we have a time series and divide it into trainigset (80% of the samples) and testset (20% of the samples). I ...
- 1
1
vote
1
answer
25
views
Parsing MIT format binary data produced with DATAQ instrument
I'm currently trying to parse in binary data produced from measurements of the membrane potential of a squid giant axon which I found here. My aim is to model the data using a nonlinear VAR (NVAR) ...
- 13
0
votes
0
answers
30
views
Mislabeled problem with hospital data
I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation.
My problem has to do with a ML tool that I am trying to develop in a ...
- 1
0
votes
0
answers
12
views
Saving and Loading PyTorch Models for Inference without Model Definition
I'm working on a PyTorch project where I need to save and then later load a model for inference in an environment where the model definition is not available. Essentially, I want to load the model (...
0
votes
0
answers
20
views
Extracting values from a map image (jpg)
Not sure if this is the right platform but I'm in a bit of a fix here.
I'm trying to convert this image(jpg) to a raster wherein, the colormap attached shows the values. I'm aware I need to use openCV ...
0
votes
0
answers
8
views
Why does TF-IDF work in TfidfVectorizer?
As I understand TF-IDF, the IDF value of the word "art" = log_e(3/1) + 1 because there are 3 documents in the data set and the word "art" appears once. But after I use the print ...
0
votes
0
answers
18
views
How the RecommenderNet model works
I'm newbie to collaborative filtering based recommendation, I have some questions about collaborative filtering when using keras' RecommenderNet model
The RecommenderNet model uses Item-based ...
0
votes
0
answers
7
views
Adapting PyTorch GRU Model for Variable Number of Detected Objects
I'm currently grappling with an issue while working with a PyTorch GRU model in a dynamic scenario where the number of detected objects can vary. The model is specifically designed to process bounding ...
0
votes
0
answers
9
views
Combining Computer Vision and traditional machine learning to predict respondent reactions (from a survey) to seeing a picture. Is it possible?
I have the following task Computer Vision/prediction task which I’m interested in hearing whether you guys think is feasible.
I have a dataset of 1000 respondents coming from a survey, where ...
- 11
0
votes
0
answers
15
views
Pytrends package in python is not working correctly
I would like to download data for the keyword "Investment" for all countries from google trends. I wrote the following code:
...
- 1
0
votes
0
answers
10
views
How can I set two anchors with different widths and the same height using PyTorch’s AnchorGenerator module?
I’m trying to use the AnchorGenerator module in PyTorch, but I’m having trouble setting up two anchors with different widths and the same height. Specifically, I want one anchor to have a height of 43 ...
0
votes
0
answers
10
views
Graph Clustering algorithms when both nodes *and* edges have features (numerical, categorical and potentially even temporal!)
I'm trying to figure out how much complexity I can get away with and am looking for model recommendations.
I have transactional data on hand - the features being customer id, customer balance, ...
0
votes
0
answers
20
views
Python Implementation of ODE for matrix operations
I am trying to develop a model and I am stuck as to how this should be implemented in python using ODE. My entire population is divided into regions and part of the model takes this into consideration ...
0
votes
0
answers
11
views
Open Source Model or dataset for attention map visualization
Helloo (:
I'm trying to develop a python application where users can upload an image e.g. of their website and receive a heatmap or attention map in return. This map should indicate where users are ...
- 1
0
votes
1
answer
108
views
Understand and compute confidence interval and coefficient of variation for regression model
I would like to better understand the concepts of:
coefficient of variation and confidence interval.
Trivially taking the definitions from wikipedia:
confidence interval (CI)
In frequentist ...
- 1
0
votes
0
answers
14
views
NLP approach for classifying webscraped data
I have a challenge in a project of mine where I will be provided with a list of scraped datas from a website. Along with the data i will also be provided some parameters like the tag of scraped ...
- 1
1
vote
0
answers
13
views
Can the k-index in either of the two individual locations be used to predict the estimated kp-index?
The k-index measures the condition of the magnetosphere. It is usually averaged over three hour, so each day has 8 measurements.
The planetary k-index (kp-index) is an average of the measures taken ...
- 173
2
votes
2
answers
63
views
How to deal with high data volumes? (Tools, techniques, concepts, etc.)
I have some doubts about how to deal with high volumes of data. I'm currently working in the data analysis/data science field, so I've had the chance to perform calculations, manipulate data, and ...
- 21
0
votes
0
answers
23
views
How to calculate Pairwise Correlation Difference?
I was trying to implement the pairwise correlation difference (PCD) found in the following paper.
https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-020-00977-1
PCD measures the ...
0
votes
0
answers
28
views
vscode cannot reconnect to existing jupyter notebook server when waking my laptop
I recently switched from browser to vs code for using jupyter notebook and the IDE-like features (e.g. auto-complete, debugging, etc) provided by VS code is really so much better than a browser. But ...
- 161
0
votes
0
answers
8
views
Why is my Histogram Gradient Boosting Classifier model still producing type II error? How can I reduce the type II error?
Type 2 error and how to hypertune or feature engineer a solution for it
I trial and tested different techniques and kept the structure which made the most sense to me. But still my model confusion ...
0
votes
0
answers
6
views
Visualize loss and gain of share to competition
I have a dataset where I capture daily sales across different companies. What can be an intuitive visualization that captures which companies are gaining when Company 1 is losing its market share? For ...
- 101
0
votes
0
answers
13
views
Should offline algorithms (SAC, DDGP, TD3) be used with suprocesses vectorized environements?
I read most of offline algorithms from Stable Baselines 3 should not be subprocessed (see here : https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html - "Which algorithm should I ...
- 1
0
votes
0
answers
6
views
Measuring Product Search effectiveness
I want to measure the effectiveness of my search engine, one of the ways i can do that is by measuring the rate at which a customer reformulates the previous query. Hence, I need to quantify inter-...
- 23
0
votes
0
answers
11
views
Do you use custom classes in your Data Science code?
After a few years in the field I find myself leaning more and more to a functional, side-effect free, deterministic input-output codeing style.
More or less everything in my work is a function:
...
- 101
0
votes
0
answers
37
views
Can't overfit Transformer Encoder
In the below code I am trying to train a very simple Transformer Encoder model to basically do nothing with its input. Giving some arbitrary input vector x, the aim of the model is then to output that ...
0
votes
1
answer
46
views
Can lag features be applied into test data without label?
can lag features be applied into test data without label? I've been wondering. I tried to build random forest model using dataset: training data (with label Y) and testing data (without label Y). The ...
- 141
1
vote
1
answer
24
views
keep track of all versions of variable values corresponding to a version (commit) of code in jupyter notebook
I am a data analyst and use jupyter notebook for various data explorations. Since I often modify code a lot, one issue I constantly have is lost track of which code version a specific variable ...
- 161
0
votes
0
answers
14
views
Best approach to find whether a scientific research paper has human trials/human testing or no
I want to know the best way to know if a paper has human trials/test subjects/testing. Was thinking of searching for some keywords in the paper like "Human", "Trials" etc.
The ...
1
vote
1
answer
27
views
Scale dataset while preserving relative distributions between columns
I have a large dataset with 460 columns. The columns have names such as
'AppOpen_1day', 'AppOpen_2day', ...... 'AppOpen_15day',
'Dig_Pos_1day', 'Dig_Pos_2day', ...... 'DigPos_15day'
etc. Each column ...
- 111
0
votes
1
answer
24
views
How to run our python scripts utilizing our device's GPU?
My laptop has NVIDIA GeForce GTX1650 GPU. I want to utilize this GPU to run my Python script. Any help in the form of code would be really helpful. I mean tried researching this so much but I couldn't ...
- 1
0
votes
2
answers
19
views
changing to gray scale
I want to transfer X-ray data images to grayscale in this code
...
0
votes
0
answers
55
views
Environment doesn't exist error gym_anytrading
I am trying to create the environment using gym_anything and I got an error that the environment doesn't exist.
...
- 1
0
votes
1
answer
79
views
Which python libraries do you recommend for label ranking?
I'm currently looking for python libraries that offer models for the label ranking problem. So provided with a context x and a set of Labels Y, the model should output a ranking of those labels.
I'm ...
- 11
0
votes
1
answer
25
views
Time Series Forecasting for Multiple Store Sales with Simultaneous Timestamps
I have a sales dataset with each store having a unique identifier. The dataset contains daily sales data for each store over a period of around two-years. I'm looking to build a time series ...
0
votes
0
answers
9
views
Loss MAE when estimating the angle of rotation of an object in an image is stuck at about 90
I am dealing with the problem of estimating the angle of rotation of objects in images. The problem is that the network gets stuck when training at a loss level of about 90.
Below is the code for my ...
- 1
0
votes
0
answers
8
views
Pretrained computer vision models that accept as input a segmented image and the original image
My data is a set of segmented images with extra details:
there is 30 object classes
each object is labeled with its state (very old, old-fashion, modern)
and each object is also labeled with a second ...
- 1
0
votes
0
answers
22
views
Cannot load audio files using AudioSegment
I am trying to load an mp3 file using pydub's AudioSegment method with below code.
...
0
votes
0
answers
11
views
0
votes
0
answers
21
views
Why does the AutoKeras NAS require reshaping of data?
Please take a look at the following source codes:
training.py
...
- 101
0
votes
0
answers
10
views
Highly unbalnced text data giving very low matrics
I have an unbalanced multi-class banking text data with around 76 classes. Classes are badly distributed such as one class which is combination of 240 other different categories, represents 50% of ...
- 1
1
vote
1
answer
38
views
Using scipy poisson, is it possible to calculate lambda if given random variable (k) and cumulative density function (p) when k is a large value?
I'm learning the Poisson distribution and am trying to "backward" calculate lambda if given a random value (k) and cumulative density function value (p). My k value is rather large, over 200,...
- 359
0
votes
0
answers
5
views
Inquiry on Point Cloud Processing for Robot Navigation Simulation
I am working on a project aimed at utilizing point clouds for creating a simulated environment for robot navigation. The process I am employing is delineated below, and I would greatly appreciate your ...
0
votes
0
answers
35
views
Avoid killing learnable parameters when transforming input into intervals
I'm trying to make a model using Pytorch which is training and transforming a set of coordinates, and then is downsampled using the model below. However, when I'm making the input coordinates into ...
- 1
0
votes
1
answer
29
views
For some reason getting an odd error when hyper tuning my model
I was hoping to hypertune my decisiontree model , however I keep running into this error:
TypeError: DecisionTreeClassifier() got an unexpected keyword argument 'criterion'
here what I tried:
...
1
vote
2
answers
178
views
Why do we need hyperparameter tuning in Scikit learn? Doesn't sk learn models by default give best model?
When I have the option to build a classifier like this directly
clf = RandomForestClassifier()
why do we perform tuning by restricting the parameters like this
<...
- 13
0
votes
0
answers
12
views
Databricks Extracting features out of Big Files like Step and PDF
So i dont know if this is the right community to ask this question, but lets see.
So we are using databricks in our company and we extract Features out of big binary files like PDF and STEP. We trying ...
2
votes
0
answers
44
views
Detecting upside down numbers in the MNIST dataset
I'm trying to predict numbers using the MNIST dataset. Since I don't know whether the numbers that I'm trying to predict are rotated (most of the time upside down, but can also be 90 or 270 degrees), ...
- 121
0
votes
0
answers
25
views
divide a large group of people into subgroups based on two parameter
note in advance: I'm new to data-analysis and although my major civil engineering taught me about statistics, I did not apply it the way I would have encountered in real life, or in this field of work....
