All Questions
36,638
questions
0
votes
0
answers
3
views
Looking for datasets on automobile parts information for machine learning
I'm embarking on a machine learning project that requires a comprehensive dataset of automobile parts information. The goal is to train a model that can identify and categorize various auto parts, ...
- 1
0
votes
0
answers
4
views
Number of stop words variation in libaries sklearn and nltk
Is there a reason why there is a big variation in the number of stop words? I assumed that there would be a general agreement from English experts how many stop words there could be. And even with ...
- 101
0
votes
1
answer
12
views
Car Make and Model detection
I am trying to develop a deep learning model that given an image of a car, it detects a car's make and model among 50 different brands, each with say another 50 models. What approach is probably the ...
- 217
1
vote
0
answers
14
views
Is it possible to determine the probability of each time sample to belong a certain class using gaussian distribution with Recurrent Neural Networks?
I'm trying to train a deep learning model that predicts the probability of each time sample in a two-component time series . In this case, I want the target tensor (Y) to be a probability value for ...
- 11
0
votes
0
answers
13
views
Why can a model's SHAP values change on a new dataset?
Background
I'm validating a model and as part of the process I've been calculating SHAP values for different validation datasets.
I've calculated SHAP values for every sample in each dataset taken ...
- 617
1
vote
1
answer
17
views
Simple rebasing formula - beginner quesitons!
I have a bunch of risk scores out of 100 (there are a few 1000 of them). The issue is they all fall within 42 and 66 - but for the most part they are normally distributed (non-expert analysis).
I was ...
- 11
0
votes
0
answers
22
views
Multiple Hypothesis Testing in feature selection process
I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My ...
- 47
0
votes
1
answer
29
views
Linear regression shows b_0 negative while it is a positive quantity
In linear regression, x is weight and y is price; none of the x and y can be negative. The linear regression line with b_0=-57.9 shows a negative y for x<=10 approximately. This signifies that more ...
- 153
1
vote
1
answer
55
views
Solve a nonlinear system with the Gauss-Newton algorithm in Python?
I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
- 111
0
votes
0
answers
6
views
Hassle-free platform for a small data science team?
our team of 4 data scientists has been exploring options to automate our ETL, data storage and model updates and are looking for a hassle-free platform that can help with this.
We've had several ...
0
votes
0
answers
6
views
How to get / calculate statistics about voxceleb2?
VoxCeleb2 contains ~1M utterances from ~6000 speakers (multilingual).
I want to get statistics about voxceleb2:
number of utterances & speakers for each language
I tried to find the metadata ...
- 363
0
votes
0
answers
12
views
Seeking datasets for training a Language Model on U.S. mortgage loan processes
I'm in the process of training a Language Model (LLM) and require datasets that encompass various aspects of the U.S. mortgage loan process. The model's aim is to understand and simulate decision-...
- 1
0
votes
1
answer
5
views
Any Interface/Library that can take the Python ML code and run on spark cluster without learning PySpark?
I have been working with Python for machine learning and have a fair amount of code written in Python using libraries such as scikit-learn, pandas, and numpy. Recently, I’ve been faced with larger ...
- 452
1
vote
1
answer
29
views
How does oversampling or undersampling approch is going to help during the testing on real time data?
We have a dataset with class A as 10% only and Class B as 90% . Let say we did undersampling or oversampling on training data and we made 50% of class A and 50% of class B. But in reality the data ...
- 13
1
vote
1
answer
20
views
Is it bad to average several MAEs calculated from chunks of a big test dataset?
In my regression problem, I am using Mean Absolute Error (MAE) as a metric for my network. My test dataset is too big to fit in memory, so I am reading the test dataset in chunks and then Keras' ...
- 203
0
votes
0
answers
13
views
seasonal adjustment to testset
I would like to understand how to apply a seasonal adjustment to my testset. Let's imagine we have a time series and divide it into trainigset (80% of the samples) and testset (20% of the samples). I ...
- 1
0
votes
1
answer
24
views
Test set won't have all the features present in the training set
I have to analyze a cardboard packaging system. To simplify, it consists of three parts, each of which needs to be set up (this depends on the pre-order): the printer, the slotter, and the cutter. The ...
- 1
0
votes
0
answers
7
views
File length in audio classification
I am dealing with audio data with samples as long as 20 min. I was wondering if I can use their spectrograms as input for Resnet18 or is it better to split them to obtain files with smaller size. Does ...
0
votes
0
answers
36
views
Missing values handling in LightGBM
I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific.
In a regression setting with no categorical feature, I have ...
- 101
0
votes
0
answers
7
views
How to selectively train a deep model based on the unavailability of a subset of the feature set
I am creating a deep learning binary classification model. Each sample in the dataset contains two mutually exclusive feature sets X and Y.
Feature set X is present in all samples; however, there are ...
0
votes
0
answers
45
views
Examples of decision processes where action has no effect on the next state, but has on the reward?
Within my studies, I work on a recurrent reinforcement learning project and I struggle to find real-world problems with a property that is important to my solutions. I look for instances of problems ...
- 136
0
votes
0
answers
9
views
Prefix tuning in LLM uses learnable vectors to fine tune the model
I would like to implement a new architecture for Transformer.
Below description is my thought.
Prefix tuning in LLM uses learnable vectors to fine tune the model.
Is there a way to use the output ...
- 25
1
vote
1
answer
35
views
Probability Distributions
I was going thru a course on probability and probability distributions for data science and in that they were describing the various probability distributions with mathematical formulae. I was ...
- 31
0
votes
1
answer
20
views
Timeseries Sales Forecasting
In my current work sales forecasting and budgeting is being done rather classical way:
Take the sales from last year for comparable date and add or decrease X% on top to reflect recent trend.
This ...
- 101
1
vote
1
answer
19
views
Out-of-Range Target Variable in Sequence-based Machine Learning Model
I'm encountering a scaling issue in a machine learning project. I'm predicting a target variable from an input sequence (and doing this for many). However, I've encountered a challenge where the ...
- 11
2
votes
1
answer
1k
views
Why images, audio/video clips, text are regarded as unstructured data?
Tabular data is regarded as structured data, while other data types such as images, audio, video, text are regarded as unstructured data.
I am confused that, taking images as an example, they are just ...
- 23
0
votes
0
answers
27
views
T-sne Inconsistent Results - Solvable?
The algorithm is highly dependent on hyperparameters such as perplexity and can yield dissimilar visualizations as a result. However, I was wandering if in the case of clear clusters in the high ...
- 1
0
votes
0
answers
13
views
How can I tell if my CNN tuning made a difference?
I'm working on a detection CNN, estimates pose for some classes of objects. I am able to compute a bunch of different metrics on performance, things like position error, rotation error, tracking ...
- 101
0
votes
0
answers
10
views
Finding Accuracy, Recall, Precision, and F1 from Matlab Confusion Matrix
I'm working on a project to find the highest accuracy between KNN and a Decision Tree for Classification using Matlab.
How to calculate the Accuracy, Recall, Precision, and F1 from the output below? ...
- 1
0
votes
1
answer
24
views
Is there a measure to compare features on the basis of normality
I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'.
I am supposed to check which if these features is the most similar to a normal distribution, so I ...
0
votes
0
answers
6
views
Decision Metrics for Selecting Promopt Structures: Creating a Prompt-Based System
I'm currently working on developing a system that generates various prompts structures (such as tree of thoughts, chain of thoughts, etc.) in response to user prompts. However, I'm facing a challenge ...
0
votes
1
answer
43
views
Correct order of preprocessing/EDA/feature engineering?
I was wondering if I have the correct order of preprocessing/EDA/feature engineering below?
Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for ...
- 111
0
votes
0
answers
8
views
how many different customer hierarchies exist in a customer data set
looking for some ideas that lead to a starting point on how to define all existing hierarchies in a data set.
e.g. A customer database has many different customer hierarchies. how could one find the ...
0
votes
0
answers
6
views
SVM kernel for detecting if a substring appears in some given string
I'm trying to do the exercise in 16.1 in the book Understanding Machine Learning, Ben-David, et al. formulated as follows:
Consider the task of learning to find a sequence of characters
("...
- 155
0
votes
0
answers
14
views
How to estimate the weights for a weighted average if not every weight is used in every example?
Say that you have a data set that contains $N$ examples. For each example, there are three main sections of important information. The first most important piece of information is the name of the ...
0
votes
0
answers
29
views
Some questions from a noob - Chat bot
I'm a T-SQL and .NET developer but I need to understand some basis about a data science / ml project.
I would like to know the guidelines and be directed to the right topics about the steps that need ...
0
votes
0
answers
18
views
Using LLMs for structured data?
I've been trying to work with structured data in language models, and it's proving to be quite challenging. I'm confident that with Langchain, I should be able to solve the problem, but I'm not ...
- 21
0
votes
0
answers
16
views
Fine-tune zero-shot classification model multi-label
I started a small project where I am trying to fine-tune a zero-shot classification model on a proprietary dataset. I was thinking to use the NLI approach, building contradiction and entailment ...
0
votes
0
answers
19
views
Is there a Tensorflow built-in function to create a matrix from a single-layered feedforward neural network without activation functions?
In Tensorflow, I implemented a simple single-layer feedforward neural network with N inputs and N outputs without activation functions and biases. Simply, it is just a N-by-N matrix. Question: is ...
- 1
1
vote
1
answer
26
views
Parsing MIT format binary data produced with DATAQ instrument
I'm currently trying to parse in binary data produced from measurements of the membrane potential of a squid giant axon which I found here. My aim is to model the data using a nonlinear VAR (NVAR) ...
- 13
0
votes
0
answers
18
views
Training with few samples, dropping training loss but constant validation loss
I am training a resnet50-based model using transfer learning. My dataset has 10 classes and about 10 occurrences per class, so it is very small. The training loss is decreasing steadily to 0.07 for ...
0
votes
0
answers
5
views
Getting Bigquery resources exceeded error in LookerML
we are getting the "Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 100% of the limit. Top memory consumer(s): aggregate functions ...
0
votes
0
answers
31
views
Mislabeled problem with hospital data
I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation.
My problem has to do with a ML tool that I am trying to develop in a ...
- 1
0
votes
0
answers
13
views
Saving and Loading PyTorch Models for Inference without Model Definition
I'm working on a PyTorch project where I need to save and then later load a model for inference in an environment where the model definition is not available. Essentially, I want to load the model (...
0
votes
0
answers
6
views
Are the filtering problem and decoding problem the same thing?
Is there any distinction the filtering problem and the decoding problem?
Wikipedia's definition for a filtering problem is:
The problem of estimating the states or ideally the posterior distribution ...
0
votes
0
answers
21
views
Assign layers and weights in BERT
I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture:
In the following, I can assign query, key and ...
- 101
0
votes
0
answers
12
views
Bayes HyperParameter tuning using wandb
Here in below code, I'm trying to use wandb sweep to find optimal lr, weight-decay using the below code:
...
0
votes
0
answers
22
views
propper feature encoding
I working with the following data set
also here is it's detailed description of "packet_dat" column
I can't understand how I can encode packet_dat column into proper feature so my ...
- 101
1
vote
1
answer
14
views
Question regarding coefficents regarding linear modeling in R
I had a question regarding the composition of coefficents of a linear model in R.
Let's say my model is as follows, with three columns/independent variables: X, Y, and Z.
And my model is: (X * Z) + (Y ...
- 11
0
votes
1
answer
18
views
Is there a standard data science workflow/decision tree?
I'm looking for some kind of reference that essentially shows an example of an entire data analysis workflow beginning with feature engineering, and ending with analyzing the results.
I know the ...
- 101