Questions tagged [experiments]
An experiment is a procedure carried out to support, refute, or validate a hypothesis. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated.
41
questions
0
votes
0
answers
36
views
How to properly perform K-fold cross validation with train, eval and test sets while building NN model?
Intro
I am training simple neural network and want to properly evaluate my model. It is not entirely clear to me, which dataset should I divide into folds in K-fold CV working with train, eval and ...
- 103
0
votes
0
answers
8
views
Transfer learning applicability for Psychology experimental research
This is my first question so please be gentle!
I am a Psychologist building a predictive model using experimental data and I want to know how I can do this using limited training data from part of my ...
1
vote
1
answer
43
views
Which test statistic do you recommend for checking the difference between a dynamic for some parameter in control and treatment group?
I have two groups of patients, a control (A) and a treatment (B), and data for patients' weight in the beginning and in the end of a period of treatment in each group. I need to check if the dynamic ...
- 111
0
votes
1
answer
10
views
Blocks, factors and treatments in designed experiments
I have recently began studying a course on Designed Experiments and am having some trouble understanding some of the terminology. I've looked at some other answers on the site and I think that I am ...
- 101
3
votes
1
answer
119
views
Taxonomy of train-test split approaches
I am looking for as close as possible for a exhaustive taxonomy of each train-test split approach.
For example, the 3 main splits that come to mind are:
A non-time based problem - would lead you to a ...
- 435
0
votes
1
answer
107
views
Store preprocessing function along with model in mlflow.keras
The following is a simplified code snipet that is relevant to storing keras LSTM models in MLFlow.
...
- 83
1
vote
0
answers
18
views
Sample Size Calculation for Bayesian A/B Testing with Continuous variables
I'm doing some Bayesian A/B testing and I need to work out an appropriate sample size in order to detect an effect. Unfortunately, there doesn't really seem to be much information about this out there ...
- 11
1
vote
0
answers
8
views
Can I use multi armed bandits to optimize how much both algorithms are weighted when creating a composite score?
So, I'm aware that multi-armed bandits are great for evaluating multiple models and from what I understand, it is mainly used to pick a specific model.
I would still like to evaluate two models but I ...
- 11
0
votes
1
answer
72
views
How do I conduct an experiment on the new pricing if it's impossible to conduct an A/B test?
We want to introduce a new price list for the customers of our international SaaS company.
Beforehand we want to test this new price list in several countries.
A/B test cannot be conducted here ...
2
votes
1
answer
95
views
Feature selection for two seperate datasets
Currently, I'm doing research with experimental data. The data comes from two experiments with two slightly different tasks, but with the same setup in a VR environment. Both experiments were done ...
- 21
1
vote
0
answers
75
views
Practical constraints in A/B testing
I saw an article about an A/B test that google had performed way back. They wanted to decide what shade of blue a button should be and how that affects click-through rate. They divided users randomly ...
- 157
0
votes
0
answers
54
views
What is the right approach to bucket users for algorithms with different coverage for A/B testing
I've couple of recommendation algorithms that I want to A/B test. Algorithm A has 90% user coverage and algorithm B has 95% user coverage. That means if the algorithms are asked to provide ...
- 571
1
vote
0
answers
36
views
paired t-test shows no difference between median and Wilcoxon test p value shows that there is a difference between median values ? How to interpret?
I have a dataset. I wanted to do paired t test on it. So I carried out normality test and it showed that it does not follow normal distribution. So I used Wilcoxon test in place of paired t test. The ...
- 151
5
votes
1
answer
105
views
experimental design in R project
I want to know of any repositories that contain complete experimental design in R covering basic test and analyses?
I want to take a top-buttom approach to learn step by step through a real project ...
- 171
1
vote
0
answers
23
views
How to think about design of experiments in the context of more complex machine learning [closed]
I have experience with multifactor DoE, but in the context of optimizing treatment of a single or or a small number of populations. Are there any articles people recommend to help get my head around ...
- 11
0
votes
0
answers
163
views
Comparing the two feature sets
I am working on a classification of two feature sets derived from a dataset. We first obtain two feature matrices derived from two feature extraction methods. Now, I need to compare them. However, the ...
- 101
1
vote
0
answers
42
views
Test and Control analysis to measure the impact of Change in sales-rep for territories
I hope you all are doing well.
Before I proceed with my problem statement, a few terminologies for reference -
Territory = Sales Territory - Think of it like a county/region
assigned to a particular ...
- 111
1
vote
1
answer
159
views
No statistical significance but observable trends
I have a general inference question regarding scenarios when results from data are not statistically significant but there appears to be an observable trend.
For example, ...
- 133
1
vote
0
answers
61
views
Python experiment results directory structure creation/manipulation package
so we do numerous types of ML experiments using a number of frameworks.
We've ended up writing an awful lot of boilerplate :
read some configuration as to the experiment's parameters, variables, ...
- 111
2
votes
1
answer
35
views
Structuring experiment/training data with months in mind
We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components ...
- 1,310
0
votes
0
answers
27
views
Attractive clustering experiment for missing value datasets
I have an upcoming publication, where I present an algorithm for clustering tabular datasets with missing values. I want to do quantitative evaluations of my algorithm and qualitative evaluation. In ...
- 752
1
vote
1
answer
17
views
Truncating float/doubles for reproducibility
I deploy machine learning models (typically GPU) to a variety of environments. I work sort of at the edge of ML R&D and devops, so I am really big into reproducibility, and one thing that drives ...
- 263
1
vote
1
answer
100
views
Checking if ML model is possible
How can I check if a machine learning model is feasible on a given dataset? What techniques like EDA, correlation etc. can be used to judge if a model is possible i.e. data and predictor variables ...
- 436
1
vote
1
answer
130
views
Annotation tool for classification experiments
What tools are available that provide an interface to present text classification results? I need to keep and compare the results of different classification algorithms for unsupervised data. The tool ...
- 21
5
votes
3
answers
2k
views
Can we use difference-in-differences with a biased A/B test?
We noticed we had a biased sample in our A/B test and was wondering if difference-in-differences would help us make valid conclusions about the data, or if there was another way to proceed.
We ran an ...
- 151
2
votes
1
answer
34
views
Are there any reports on Hardware errors affecting experiments?
I recently wrote this as a list of what I have seen / can think of as problem sources that make it hard to reproduce (replicate?) an experiment. I think I have seen most of them, except for hardware ...
- 18.7k
8
votes
2
answers
1k
views
A/B testing: How to calculate p-value on post test segments?
My question on A/B testing is about doing post test segmentation analysis.
For example:
I run an A/B test on my website to track bounce rate. On the
treatment group, i put a video to explain ...
- 233
1
vote
1
answer
2k
views
Approaches to A/B testing when you can't randomize on the user level
I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.
For example, if I own 10 shops and have a ...
- 1,843
1
vote
1
answer
218
views
Do Data Miners realise Machine Learning cannot establish Causality? [closed]
Do people in Machine Learning (or, more generally, in Data Mining) realise that no causal link can be inferred from a correlation?
My impression of the ML/AI community is that most people involved ...
- 127
5
votes
2
answers
741
views
AB testing : When AA testing doesn't work
After 6 months of AB testing on our CRM tool (Oracle Responsys, but this could be true with anyone), the test exhibited some weird results so we decided to pause everything, and to make some good old ...
- 191
-1
votes
1
answer
204
views
Avoiding leakage on my random forest?
I am training a random forest model. I am wondering if it is safe (leakage?) to use on my training set the parameter average price of a car calculated using all my data points. The issue is that some ...
- 131
1
vote
2
answers
209
views
How to determine if a company decision was successful or not?
I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not.
I have done some research and saw that A/B testing might be a way to do this ...
- 123
5
votes
3
answers
4k
views
What are the methods to ensure that the population split for A/B test is random?
Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.
- 91
3
votes
2
answers
84
views
How do I learn experimental methodology? When is it relevant?
I just graduated in Computer Science, with a very theoretical background but without any kind of Data Science or Artificial Intelligence experience, and I working on my own to discover those two ...
- 131
1
vote
0
answers
23
views
Experiments on time spent by the visual cortex
I'm working on a object detection problem, using convolutional neural networks which have a lot in common with how the human visual cortex works. I've previously seen remarks on tasks in computer ...
- 111
4
votes
2
answers
1k
views
Recommendations for storing time series data
As part of my thesis I've done some experiments that have resulted in a reasonable amount of time-series data (motion-capture + eye movements). I have a way of storing and organizing all of this data, ...
- 558
3
votes
1
answer
72
views
Attributing causality to single quasi-independent variable
Apologies if this isn't the correct place to ask - I'm not sure if this fits best with Stats or Data Science.
I'm using analytics to help marketers identify attributes of their users correspond to ...
3
votes
1
answer
41
views
Timing sequence in MapReduce
I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution ...
- 31
11
votes
2
answers
432
views
Book keeping of experiment runs and results
I am a hands on researcher and I like testing out viable solutions, so I tend to run a lot of experiments. For example, if I am calculating a similarity score between documents, I might want to try ...
- 113
3
votes
1
answer
80
views
Estimating Variance Reduction Resultant from Additional Data
I couldn't quite think of how best to title this, so recommendations are welcome. Same goes for the tags (I don't have the reputation to use the tags that I thought were appropriate). The question is ...
- 131
10
votes
4
answers
411
views
How to debug data analysis?
I've came across the following problem, that I recon is rather typical.
I have some large data, say, a few million rows. I run some non-trivial analysis on it, e.g. an SQL query consisting of several ...
