New answers tagged

1

It depends how deep technichally you want to go. You can apply a slight modification of a Survival methods/ cox models that relate the time that passes, before some event occurs, to one or more covariates that may be associated with that quantity of time. Also if you group de features you can make the problem look like as a classical binary classification ...


2

You have a big dataset and you get new instances//data every 2 months. First you should select with which data you want to train. Since your data is big and there is the probability than data from 2 years ago is not as relevant as the data from the last month you can consider doing a Roll out// slidding window validation. This way you will only select the ...


2

From what I understand - your problem is "sample selection bias" problem. Any kind of pattern to select a subset out of large data may lead to bias. This raises two question. How to choose? Random/stratified random (If you have multiple classes) under sampling to obtain a smaller subset. How big to choose? we can set percentage of undersampling. Reducing ...


1

I know of few though they do not model user behavior, they just try to present best search results and compare to user ideal. lemurproject.org has books based on it though it is old and in java. For example the user activity recorder works with firefox pre big numbers versioning. That is something like 10 years ago. Maybe has a modeler but almost certainly ...


-1

Maybe you could start by actually and precisely defining what you want to do with your data. Apparently you'd like to predict some probability for each type of alert (which I assume is given by the TYPE column), and find which one gets the highest probability (that boiling down to finding what type of alert is the most likely to happen in a given time ...


1

I will try and be as concise as possible. First, let's redefine the way you think about your data points. There can ever only be two types of visits in terms of time. Periodic and Non-Periodic. Let's call each visit an event. Some events could be related to chronic conditions where periodic visits are quite common. Some events could be related to flu, head ...


2

I would separate value with representation in this case. Energy as you mentioned, in the real world, holds a very continuous value. However, we may choose (for various reasons) to represent this value in different forms. We can take values as they are (15.21252, 23.76535), we can round them into integers (15, 24), we can even decide to represent this ...


1

I would like to make the argument that you actually cannot have statistically speaking 100.00% accuracy even in theory but you can get really close. However, you getting too close might mean that your overfitting. This is because you cannot have statistically speaking absolute zero uncertainty in any system of more than 2 predictors that are independent or ...


2

In theory it's of course possible to reach perfect performance: if the algorithm can find what it needs in the features to correctly distinguish between classes (or clusters), then it will perform perfectly. In reality however it's very rare that performance is perfect, because: Text data is noisy and extremely diverse Most of the time when there is a way ...


2

Your assignment is basically the process we call EDA - Explorative Data Analysis. So what should you do? Simply explore! What is the shape of your dataset? How do variables behave, do they have a factor structure, correlate, etc. What are the main descriptives of your dataset, to they tell an interesting story, etc. And once you start doing this you will ...


0

I suggest to check out some introductory books on NLP, e.g. Natural Language Processing with Python. This is a very accessible and practical introduction and useful even if you won't be working in Python. Another, more detailed text book is Speech and Language Processing by Jurafsky and Martin. You need to understand the basics first, and these are ...


2

K-means don't modify the underlying structure of your data. K-means will just provide the 'color' part of your graph. To answer the question about why do you get a cuboid, it's because your underlying data are a cuboid. Not necessarily by construction, but that's what happen when you cap your data. As an exemple, look at the following code : X1 = c(rnorm(...


0

How would you know you have to do cluster analysis before looking at your data ? Setting aside data quality questions (which you should never do), a bare minimum of EDA will help you : Know if it's relevant to do a clustering analysis (rarely imo) Know if K-means is the best clustering tool (rarely imo) Get an idea of the number of the clusters Then you ...


3

If you expect that all zeros is a result of error in the measuring of the features (i.e. the observations should not be all 0s but they are), then I would say: Keep all the data, but increase k (from k-means) by 1. This extra one will hopefully become the class of all these wrong observations. If you expect that all zeros is correct (i.e. these observations ...


3

It's a matter of data quality so it depends how the dataset was built: Either these instances are meaningful, i.e. it makes sense that an observation would have zeros for all the features and that it would happen that often. Or these are the result of an error, typically the complete absence of measurement for these observations. Naturally one wants to ...


0

But can I assign the cluster labels from the pca reduced data to the original data ? would it be a right approach ? I guess not. Yes, that is totally the right approach. Principal components are just the linear combinations of your original features that explain the most variance, so you can definitely use them for clustering. Moreover, since you only kept ...


Top 50 recent answers are included