Questions tagged [data-science-model]
Questions about the organization of elements of data, and the standardization of their relations.
651
questions
0
votes
0
answers
4
views
Couldn't get the correct values after denormalizing LSTM Model
I couldn't get the correct value of the actual stock price after I denormalized the prediction value. The actual stock price should be around 150+-, however, after denormalizing it only shows around ...
0
votes
0
answers
18
views
General question on the test and train data
Currently I'm working on a Kaggle problem. I have to predict an outcome by the given information. There are few metafiles for training a model with lots of features (>30). However, in the test file ...
- 1
1
vote
1
answer
26
views
Making a netcdf data using xarray
I am very very new to the world of data science as I only started using it in my new job so I would really appreciate help from the community experts (maybe also in simple words :)).
I am trying to ...
- 113
3
votes
2
answers
212
views
What techniques are used to analyze data drift?
I've created a model that has recently started suffering from drift.
I believe the drift is due to changes in the dataset but I don't know how to show that quantitatively.
What techniques are ...
- 511
2
votes
3
answers
136
views
How to forecast a timeseries with geolocation data?
I have created a dataset with my geolocations from the last three months. The data set contains longitude, latitude, and timestamp, with a frequency of every 5 minutes. Based on this data, I want to ...
- 93
-3
votes
1
answer
33
views
What to do if your model's prediciton result wrong because of unlucky?
Have you ever had a situation where your model backtested with very good with historical data, and you also felt that your model was very logical? But when put it into practice case to predict the ...
7
votes
5
answers
438
views
How to handle missing value if imputation doesnt make sense
I have column/feature in my dataset showing years a person has been married "years_married".
Since not every person is married there are NaN fields.
It does not make sense to fillna(0) "...
- 111
1
vote
0
answers
20
views
Object Detection: Setting threshold values as trainable parameters?
I am building my first object detection model (Mobilenet SSD, to detect animals in images) and happy with the current test results.
When I tested it using images without bounding boxes, I noticed some ...
- 11
0
votes
0
answers
15
views
Decent cloud provider to run python algorithmic trading [closed]
I'm searching for stable safe intuitive solutions to deploy my algo trading online, I have no experience in cloud computing, and I want to train my models weekly locally and update cloud.
I't will ...
0
votes
2
answers
38
views
I am getting all scores as 100% on my machine learning models. Is it okay to have this kind of result?
I am getting all scores for my ML model as 100% for the Extra Trees Algorithm. I am applying the necessary pre-processing steps (duplication removal, correlations validating, cardinality validation, ...
- 1
4
votes
2
answers
799
views
Do model training pipeline should run on dev, staging and production environment?
I know it's a best practice to ship our code from dev to staging to production by including different level tests and validations that will help to confidently deploy on the production environment.
...
- 141
0
votes
0
answers
9
views
Can I create my validation set after preprocessing the train set?
I have another question regarding the dataset split.
If I have a train and test set can I perform all the preprocessing steps (scaling, imputation, ...) on the training one and then split it into ...
0
votes
0
answers
17
views
Time series forecasting: Youtube Views
I have some monthly data for 200~ videos from a youtube channel. I can see how many views each video got each month. Videos were released consistently each week and there is no missing data.
Usually a ...
- 1
0
votes
1
answer
17
views
Theory behind time Series Test dataset being the last x%
The standard flow for time-series that i'm aware of, is that you divide your dataset for Training & Validation (60% and 20% respectively for example) and the last 20% is used for Unbiased Testing....
- 101
0
votes
0
answers
6
views
Approaching multiple records for one observation; radiomics of 2D slices of a 3D object
Background
I am trying to create a model that can predict Type 2 diabetes in a patient based on MRI scans of their thigh muscle. Previous literature has shown that fat deposition in the muscle of ...
0
votes
1
answer
21
views
Data Leakage possibility related to replacing rare values on a categorical column
I am exploring the possibility of data leakage for categorical columns, replacing rare values with category 'Other'
Let's say I have a DF with 40 categorical columns. I will check each of them, find ...
- 29
0
votes
1
answer
25
views
ML Predicted Model for 2 values
I have a data set with 96 rows. It contains date, source, spend and number of customers.
I have 4 different sources that generate customers and you can see in the dataset how much I spend and how many ...
0
votes
0
answers
13
views
Machine learning with 2D data table and single outcome to model
I am new to machine learning and am trying to conceptualize how to effectively build a database of sports data for machine learning. I currently have a list of games and outcomes as well as separate ...
- 1
0
votes
1
answer
20
views
Can using the mean of absolute Shapely Values for feature importance give very wrong results?
In a classification problem, suppose a model has 2 variables, A and B, and the null model (the model without any variable) predicts 50% probability for belonging to class 1 for all the instances.
Now ...
- 173
1
vote
1
answer
32
views
How to increase retention?
As you might already know there is a concept of retention. Let's say I have created a game and today hundred people have downloaded my game. Let's say tomorrow 47 out of yesterday's hundred people are ...
- 121
4
votes
1
answer
35
views
Resources for Promotion/Demotion Strategies for ML Item Recommendation Systems?
We are looking to design a system where specific items or categories of items can be boosted/promoted up or relegated/demoted down the recommendation order.
What are the common strategies or standards ...
0
votes
0
answers
13
views
How to Forecast Stock Index against Global Indicators
My goal is predict the benchmark index of the Indian stock market with the help machine learning algorithm using of global market indices as mentioned below. Put simply, forecast whether tomorrow’s ...
0
votes
0
answers
11
views
Assigning Final Scores to Identified Technologies: Considering Users' Reputation, badge counts, post scores, no.of posts, & post date
I am trying to determine the importance of various factors in assigning a score for identified technologies using the user's StackOverflow post tags and content.
The considering factors are users' ...
- 103
-1
votes
1
answer
49
views
Curve val_loss and loss in keras after training a model
I trained a Keras model to diagnose disorders and want to make sure it is good enough to start deploying.
From the below graph, can anyone advise me as to whether my model is overfitting or ...
0
votes
0
answers
5
views
Can Data Mining or traditional EDW also do Process Mining?
Can Data Mining also do Process Mining? Can frameworks and tools used for Data Mining or EDW be used for Process Mining?
Summarize the problem
I'm investigating Process Mining with a view to create ...
- 101
0
votes
0
answers
36
views
Adding Data sequences as unique data on dataset for regression model
I want to predict a force plate using plantar pressure. The shape of the force plate data is a 15000x6 array, and the shape of the plantar pressure data is a 15000x89 array. I will use a regression ...
- 21
0
votes
0
answers
11
views
Why the order of the fearures affects synapse LightGBM predictions?
I am using LighGBM Classifier and Regressor and it seems that the order of the features I am adding, affect the predictions of the model. Everytime I change the order, another result comes up and with ...
0
votes
0
answers
84
views
Where can I get REDD dataset if http://redd.csail.mit.edu/ is not working?
Reference Energy Disaggregation Data Set (REDD) is used for research in Non-intrusive Load Monitoring (NILM). Every paper and blogs points towards http://redd.csail.mit.edu/ as data downling link. ...
- 103
0
votes
0
answers
13
views
why we use chebyshev's inequality when we can convert data into normal distribution?
When there is a normal distribution available, why don't we just converted all data into Russian distribution and then use this equation instead of going with other formula and all we have to buy to ...
0
votes
0
answers
21
views
Any ideas on deciding a hotel competitive group?
Suppose for each hotel, I need to find a group of hotels which are competitive to this hotel. Competitive hotels mean they probably share the same group of customers, and the customers will make a ...
0
votes
0
answers
10
views
What is the best way to set up control limits for customer rating data?
I have 2 years of monthly average ratings for a product given by the customers. Obviously there are fluctuations in the average ratings and I wanted to set up limits for this average monthly ratings ...
- 1
0
votes
0
answers
25
views
Machine learning method to detect correlation between bike counters
I am doing a research master's degree in transportation science. I would like to develop a model for one of my classes to detect the dependence between various bicycle counters.
The database I'm using ...
- 1
0
votes
1
answer
37
views
Predict coordinates from input of coordinates
I'm a newbie at data science and I want to ask how can I predict a set of coordinates from a set of input coordinates? That is (x1, y1) -> (x2, y2).
To give a ...
1
vote
1
answer
40
views
Interpreting Learning Curves of models
I need some help to understand if the models are overfitting and which of these we can consider "the best". On the internet i only find simple examples with learning curves but in these ...
- 41
0
votes
0
answers
10
views
Multicategory choice model resources
I'm a student wanting to do some applied research.
The data I'm working with has 500 consumers, and for each consumer, there are 4 type of goods they bought and each are categorized by company.
For ...
2
votes
4
answers
150
views
99% accuracy in train and 96% in test is too much overfitting?
I have a binary classification problem, the classes are quite balanced (57%-43%), with a GridSearch with Random Forest Classifier I obtained the best hyperparameters and I applied the model to train ...
- 41
1
vote
2
answers
77
views
Are Chi-square and ANOVA (f_classif) to select best features?
I have a binary classification problem (target 0 o 1), I have both variables continuous and categorical as features. I understood that about Chi-square i can use only categorical features to evaluate ...
- 41
1
vote
0
answers
142
views
How to convert an ONNX model with dynamic input shape to Tensorflow model
I am trying to convert an ONNX model with a dynamic input shape to TensorFlow format using the onnx_tf package. I am using TensorFlow 2.11.0, ONNX 1.13.0, and onnx_tf 1.10.0. The input to the model ...
0
votes
0
answers
16
views
Is there a way to quantify uncertainty in classification?
I'm thinking of a way to build an extension to a binary classifier (actually I will get the output probabilities like in logistic regression, so technically you should call this regression) that ...
- 53
0
votes
1
answer
25
views
Differences in OvO SVC decision_function and predict methods in Scikit learn package
I have observed that there are differences in the implementation of the decision_function and the predict methods the one versus one multi class implementation of SVC in the sci-kit learn package. Is ...
0
votes
0
answers
18
views
What does KDE plot tell to me?
What the KDE plot tells to me? How can I evaluate if my model is good by looking at the graph? For example I have this KDE plot of the residuals(it's x_pred-y_pred) of a machine learning evaluation of ...
- 41
0
votes
1
answer
46
views
KMeans is not predicting the correct cluster
k-means clustering is done and created 5 optimal number of clusters. (Clustering is done unevenly).
While using them in my model, the model is not choosing the exact cluster which has the exact data. ...
- 1
0
votes
1
answer
49
views
How to train machine learning on sales forecasting problems of almost 10,000 shops?
I have a dataset of almost 10,000 shops, 'dates', 'shop ID' and 'sales amounts' as their features almost 2 years of data. I want to forecast each shop, the sales amount for 30 next days. I want to ...
- 123
0
votes
1
answer
32
views
What can be the reasons for 95% of samples belong to one cluster when there is 5 clusters?
'''I used the k-means algorithm to clustering set of documents which are textual data only.
The document has 2lack records.
Surprisingly the result for the clustering is
90% of records is storing in 1 ...
- 1
0
votes
1
answer
42
views
Numbering after classification
I have tabular data. Arbitary amount of rows can form a certain class. Data contains multiple instances of each class. I want each class instance to contain its own number. What is the name of this ...
- 101
0
votes
0
answers
7
views
Hidden Markov Models and Probability Distribution functions
I am doing a personal project on a continuous dataset testing some Markov chains and identifying the hidden states of the dataset. Right now i have the model trained and i have predictions, but i want ...
0
votes
1
answer
24
views
Customizing Collaborative Filtering for Product Affinity
I'm trying to build a recommendation system and I am trying to use Collaborative Filtering (please let me know if other models fit better for my use case).
My Data:
My data is for an e-commerce site ...
0
votes
0
answers
69
views
Is it possible to turn the gmail inbox into a dataset for AI fine tuning?
My idea is to have a dataset of my gmail emails and replies. The purpose is to create a bot that can reply new emails based on all past correspondence in my inbox. How do i prepare such dataset from ...
- 1
0
votes
1
answer
26
views
Ideas on how to solve a problem using machine learning
I am fairly new to machine learning. I have been in mechanical simulation field for the past 7-8 years, I realise there are potential areas which I have been doing the same thing day in and day out, ...
0
votes
1
answer
75
views
Reading a model stored as binary
I'm using a s3 bucket to store a model I trained in python. Since I'm using an s3 bucket I convert the file to binary first and then store it on the bucket.
...
