All Questions

Filter by
Sorted by
Tagged with
0 votes
0 answers
3 views

Looking for datasets on automobile parts information for machine learning

I'm embarking on a machine learning project that requires a comprehensive dataset of automobile parts information. The goal is to train a model that can identify and categorize various auto parts, ...
Anand 's user avatar
0 votes
0 answers
4 views

Number of stop words variation in libaries sklearn and nltk

Is there a reason why there is a big variation in the number of stop words? I assumed that there would be a general agreement from English experts how many stop words there could be. And even with ...
heretoinfinity's user avatar
0 votes
1 answer
12 views

Car Make and Model detection

I am trying to develop a deep learning model that given an image of a car, it detects a car's make and model among 50 different brands, each with say another 50 models. What approach is probably the ...
Tina J's user avatar
  • 217
1 vote
0 answers
14 views

Is it possible to determine the probability of each time sample to belong a certain class using gaussian distribution with Recurrent Neural Networks?

I'm trying to train a deep learning model that predicts the probability of each time sample in a two-component time series . In this case, I want the target tensor (Y) to be a probability value for ...
Kevin Vargas's user avatar
0 votes
0 answers
13 views

Why can a model's SHAP values change on a new dataset?

Background I'm validating a model and as part of the process I've been calculating SHAP values for different validation datasets. I've calculated SHAP values for every sample in each dataset taken ...
Connor's user avatar
  • 617
1 vote
1 answer
17 views

Simple rebasing formula - beginner quesitons!

I have a bunch of risk scores out of 100 (there are a few 1000 of them). The issue is they all fall within 42 and 66 - but for the most part they are normally distributed (non-expert analysis). I was ...
George hill's user avatar
0 votes
0 answers
22 views

Multiple Hypothesis Testing in feature selection process

I am doing feature selection of features which are of binary nature i.e. each feature represents presence or absence of a substructure in a molecule. And I have a target variable of two classes. My ...
As13's user avatar
  • 47
0 votes
1 answer
29 views

Linear regression shows b_0 negative while it is a positive quantity

In linear regression, x is weight and y is price; none of the x and y can be negative. The linear regression line with b_0=-57.9 shows a negative y for x<=10 approximately. This signifies that more ...
PS Nayak's user avatar
  • 153
1 vote
1 answer
55 views

Solve a nonlinear system with the Gauss-Newton algorithm in Python?

I would like to solve a non-linear system (which contains the goals of a football team in previous matches) using the Gauss-Netwon algorithm, in order to find the parameter (of frequency) to use as ...
Horiatiki's user avatar
  • 111
0 votes
0 answers
6 views

Hassle-free platform for a small data science team?

our team of 4 data scientists has been exploring options to automate our ETL, data storage and model updates and are looking for a hassle-free platform that can help with this. We've had several ...
Rotten_Cabbagges's user avatar
0 votes
0 answers
6 views

How to get / calculate statistics about voxceleb2?

VoxCeleb2 contains ~1M utterances from ~6000 speakers (multilingual). I want to get statistics about voxceleb2: number of utterances & speakers for each language I tried to find the metadata ...
user3668129's user avatar
0 votes
0 answers
12 views

Seeking datasets for training a Language Model on U.S. mortgage loan processes

I'm in the process of training a Language Model (LLM) and require datasets that encompass various aspects of the U.S. mortgage loan process. The model's aim is to understand and simulate decision-...
Anand 's user avatar
0 votes
1 answer
5 views

Any Interface/Library that can take the Python ML code and run on spark cluster without learning PySpark?

I have been working with Python for machine learning and have a fair amount of code written in Python using libraries such as scikit-learn, pandas, and numpy. Recently, I’ve been faced with larger ...
Mohith7548's user avatar
1 vote
1 answer
29 views

How does oversampling or undersampling approch is going to help during the testing on real time data?

We have a dataset with class A as 10% only and Class B as 90% . Let say we did undersampling or oversampling on training data and we made 50% of class A and 50% of class B. But in reality the data ...
Professor's user avatar
1 vote
1 answer
20 views

Is it bad to average several MAEs calculated from chunks of a big test dataset?

In my regression problem, I am using Mean Absolute Error (MAE) as a metric for my network. My test dataset is too big to fit in memory, so I am reading the test dataset in chunks and then Keras' ...
ihavenoidea's user avatar
0 votes
0 answers
13 views

seasonal adjustment to testset

I would like to understand how to apply a seasonal adjustment to my testset. Let's imagine we have a time series and divide it into trainigset (80% of the samples) and testset (20% of the samples). I ...
Cata's user avatar
  • 1
0 votes
1 answer
24 views

Test set won't have all the features present in the training set

I have to analyze a cardboard packaging system. To simplify, it consists of three parts, each of which needs to be set up (this depends on the pre-order): the printer, the slotter, and the cutter. The ...
Isaac's user avatar
  • 1
0 votes
0 answers
7 views

File length in audio classification

I am dealing with audio data with samples as long as 20 min. I was wondering if I can use their spectrograms as input for Resnet18 or is it better to split them to obtain files with smaller size. Does ...
ml_nnoobb's user avatar
0 votes
0 answers
36 views

Missing values handling in LightGBM

I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific. In a regression setting with no categorical feature, I have ...
Augustin's user avatar
  • 101
0 votes
0 answers
7 views

How to selectively train a deep model based on the unavailability of a subset of the feature set

I am creating a deep learning binary classification model. Each sample in the dataset contains two mutually exclusive feature sets X and Y. Feature set X is present in all samples; however, there are ...
flamingo_stark's user avatar
0 votes
0 answers
45 views

Examples of decision processes where action has no effect on the next state, but has on the reward?

Within my studies, I work on a recurrent reinforcement learning project and I struggle to find real-world problems with a property that is important to my solutions. I look for instances of problems ...
Tomasz Witkowski's user avatar
0 votes
0 answers
9 views

Prefix tuning in LLM uses learnable vectors to fine tune the model

I would like to implement a new architecture for Transformer. Below description is my thought. Prefix tuning in LLM uses learnable vectors to fine tune the model. Is there a way to use the output ...
jackson's user avatar
  • 25
1 vote
1 answer
35 views

Probability Distributions

I was going thru a course on probability and probability distributions for data science and in that they were describing the various probability distributions with mathematical formulae. I was ...
grkiran2011's user avatar
0 votes
1 answer
20 views

Timeseries Sales Forecasting

In my current work sales forecasting and budgeting is being done rather classical way: Take the sales from last year for comparable date and add or decrease X% on top to reflect recent trend. This ...
miroslaavi's user avatar
1 vote
1 answer
19 views

Out-of-Range Target Variable in Sequence-based Machine Learning Model

I'm encountering a scaling issue in a machine learning project. I'm predicting a target variable from an input sequence (and doing this for many). However, I've encountered a challenge where the ...
Bloggy's user avatar
  • 11
2 votes
1 answer
1k views

Why images, audio/video clips, text are regarded as unstructured data?

Tabular data is regarded as structured data, while other data types such as images, audio, video, text are regarded as unstructured data. I am confused that, taking images as an example, they are just ...
XMB-7's user avatar
  • 23
0 votes
0 answers
27 views

T-sne Inconsistent Results - Solvable?

The algorithm is highly dependent on hyperparameters such as perplexity and can yield dissimilar visualizations as a result. However, I was wandering if in the case of clear clusters in the high ...
codeplay's user avatar
0 votes
0 answers
13 views

How can I tell if my CNN tuning made a difference?

I'm working on a detection CNN, estimates pose for some classes of objects. I am able to compute a bunch of different metrics on performance, things like position error, rotation error, tracking ...
Mr Squid's user avatar
  • 101
0 votes
0 answers
10 views

Finding Accuracy, Recall, Precision, and F1 from Matlab Confusion Matrix

I'm working on a project to find the highest accuracy between KNN and a Decision Tree for Classification using Matlab. How to calculate the Accuracy, Recall, Precision, and F1 from the output below? ...
Istiamel's user avatar
0 votes
1 answer
24 views

Is there a measure to compare features on the basis of normality

I have a dataset of cars and it has many features including 'acceleration’, ‘horsepower’, and ‘mpg'. I am supposed to check which if these features is the most similar to a normal distribution, so I ...
Tareq Ewaida's user avatar
0 votes
0 answers
6 views

Decision Metrics for Selecting Promopt Structures: Creating a Prompt-Based System

I'm currently working on developing a system that generates various prompts structures (such as tree of thoughts, chain of thoughts, etc.) in response to user prompts. However, I'm facing a challenge ...
Jay Padhiyar's user avatar
0 votes
1 answer
43 views

Correct order of preprocessing/EDA/feature engineering?

I was wondering if I have the correct order of preprocessing/EDA/feature engineering below? Yes there are nuances and may vary from problem to problem, but am just looking for a general pipeline for ...
Katsu's user avatar
  • 111
0 votes
0 answers
8 views

how many different customer hierarchies exist in a customer data set

looking for some ideas that lead to a starting point on how to define all existing hierarchies in a data set. e.g. A customer database has many different customer hierarchies. how could one find the ...
Deathstar2008's user avatar
0 votes
0 answers
6 views

SVM kernel for detecting if a substring appears in some given string

I'm trying to do the exercise in 16.1 in the book Understanding Machine Learning, Ben-David, et al. formulated as follows: Consider the task of learning to find a sequence of characters ("...
Tran Khanh's user avatar
0 votes
0 answers
14 views

How to estimate the weights for a weighted average if not every weight is used in every example?

Say that you have a data set that contains $N$ examples. For each example, there are three main sections of important information. The first most important piece of information is the name of the ...
Ignatius the Learner's user avatar
0 votes
0 answers
29 views

Some questions from a noob - Chat bot

I'm a T-SQL and .NET developer but I need to understand some basis about a data science / ml project. I would like to know the guidelines and be directed to the right topics about the steps that need ...
Pietro Ariano's user avatar
0 votes
0 answers
18 views

Using LLMs for structured data?

I've been trying to work with structured data in language models, and it's proving to be quite challenging. I'm confident that with Langchain, I should be able to solve the problem, but I'm not ...
Cosapocha's user avatar
0 votes
0 answers
16 views

Fine-tune zero-shot classification model multi-label

I started a small project where I am trying to fine-tune a zero-shot classification model on a proprietary dataset. I was thinking to use the NLI approach, building contradiction and entailment ...
lucatrovato's user avatar
0 votes
0 answers
19 views

Is there a Tensorflow built-in function to create a matrix from a single-layered feedforward neural network without activation functions?

In Tensorflow, I implemented a simple single-layer feedforward neural network with N inputs and N outputs without activation functions and biases. Simply, it is just a N-by-N matrix. Question: is ...
H.C.'s user avatar
  • 1
1 vote
1 answer
26 views

Parsing MIT format binary data produced with DATAQ instrument

I'm currently trying to parse in binary data produced from measurements of the membrane potential of a squid giant axon which I found here. My aim is to model the data using a nonlinear VAR (NVAR) ...
Ben Andrews's user avatar
0 votes
0 answers
18 views

Training with few samples, dropping training loss but constant validation loss

I am training a resnet50-based model using transfer learning. My dataset has 10 classes and about 10 occurrences per class, so it is very small. The training loss is decreasing steadily to 0.07 for ...
ml_nnoobb's user avatar
0 votes
0 answers
5 views

Getting Bigquery resources exceeded error in LookerML

we are getting the "Resources exceeded during query execution: The query could not be executed in the allotted memory. Peak usage: 100% of the limit. Top memory consumer(s): aggregate functions ...
Dhiraj Bhalerao's user avatar
0 votes
0 answers
31 views

Mislabeled problem with hospital data

I am writing this post to ask or see if someone can help me with this problem in case you may have faced a similar situation. My problem has to do with a ML tool that I am trying to develop in a ...
Daniel's user avatar
  • 1
0 votes
0 answers
13 views

Saving and Loading PyTorch Models for Inference without Model Definition

I'm working on a PyTorch project where I need to save and then later load a model for inference in an environment where the model definition is not available. Essentially, I want to load the model (...
Carpediem's user avatar
0 votes
0 answers
6 views

Are the filtering problem and decoding problem the same thing?

Is there any distinction the filtering problem and the decoding problem? Wikipedia's definition for a filtering problem is: The problem of estimating the states or ideally the posterior distribution ...
Tommaso Bendinelli's user avatar
0 votes
0 answers
21 views

Assign layers and weights in BERT

I print the weight names and shape of the BERT transformer. Now, I want to assign the printed weight to the layers in the transformers architecture: In the following, I can assign query, key and ...
Christian01's user avatar
0 votes
0 answers
12 views

Bayes HyperParameter tuning using wandb

Here in below code, I'm trying to use wandb sweep to find optimal lr, weight-decay using the below code: ...
Sarvagya Porwal's user avatar
0 votes
0 answers
22 views

propper feature encoding

I working with the following data set also here is it's detailed description of "packet_dat" column I can't understand how I can encode packet_dat column into proper feature so my ...
Roma's user avatar
  • 101
1 vote
1 answer
14 views

Question regarding coefficents regarding linear modeling in R

I had a question regarding the composition of coefficents of a linear model in R. Let's say my model is as follows, with three columns/independent variables: X, Y, and Z. And my model is: (X * Z) + (Y ...
Nevermnd's user avatar
0 votes
1 answer
18 views

Is there a standard data science workflow/decision tree?

I'm looking for some kind of reference that essentially shows an example of an entire data analysis workflow beginning with feature engineering, and ending with analyzing the results. I know the ...
David's user avatar
  • 101

15 30 50 per page
1
2 3 4 5
733