# Tag Info

21

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through. (More Info)

11

The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable. If you encode time as numeric, then you are imposing certain restrictions on the model. For a linear regression model, the effect of time is now monotonic, either the target will increase or decrease with time. For decision trees, ...

8

I'm not aware of a foolproof way to do this. Here's one idea off the top of my head: Treat values as categorical by default. Check for various attributes of the data that would imply it is actually continuous. Weight these attributes based on how likely they are to correlate with continuous data. Here are some possible examples: Values are integers: +.7 ...

5

I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it is to category 3). This is definitely not the case for hours or months. However, the issue that you raise is that you want to represent hours and months in a ...

4

If you have, for example, number of children of a family (which could range, for example, between 0 and 5), is it a categorical or numerical variable? Actually it depends on your problem and how you intend to solve it. In this sense, you can do the following: Compute the number of unique values of that column Divide this number by the total number of rows ...

4

You can get a reasonably good approximation of steps for exploratory data analysis (EDA) by reviewing the EDA section of the NIST Engineering Statistics Handbook. Additionally, you might find helpful parts of my related answer here on Data Science SE. Methods, related to EDA, are too diverse that it is not feasible to discuss them in a single answer. I will ...

3

It depends on which algorithm you're using. If you're using tree-based algorithms like random forest, just pass this question. Categorical encoding isn't necessary for tree-based algorithms. For other algorithms like neural network, I suggest trying both method(continuous & categorical). The effect differs between different situations.

2

Binning One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that fall within each bin to obtain a density. So, the calculation would be:  \sum_{b_x \in Bins_x} \sum_{b_y \in Bins_y} \frac {\#(b_x, b_y)} N \log ...

2

Since this question has been cross-posted, the initial comments by @nickcox on Cross Validated are highly relevant and true. My views are slightly different. For instance, I would rephrase the question, decomposing it into two parts: first, there is the issue of how one would go about classifying a stream of unknown information by data type and, second, what ...

2

To rephrase the answer provided by @raghu. One major difference between categorical and numerical features is whether the magnitude of the numbers are comparable, i.e., is 2019 bigger than 2018, or December(12) bigger than March (3)? Not really. While there is a sequential order in these numbers, their magnitude is not comparable. Thus, transforming into a ...

2

I don't know exactly what the problem is, but maybe you could try checking the value of your gradients and see if they change a lot around the 8th epoch.

2

Clustering on categories is not something sklearn can do by default. And assigning sequential values to categories like that certainly won't help - clustering tends to work based on distance, by assigning 0, 1, 2 to Yes, No, Don't Know like that, you are suggesting Yes is 'closer' to No than it is to Don't Know. I highly recommend having a look at k-modes, ...

1

I guess this is off-topic but here's a reproducible code: #Preparing mocking data >set.seed(123) #for reproducibility >Party <- sample(c('D', 'N','R'), 10, replace = TRUE) #sample random values >d <- data.frame(Party) #put values into a dataframe # Here's how you substitute the values in just one line. This operation is vectorised. >d\$...

1

One reason might be “exploding gradients”. Although your loss function seems to output quite stable values, it can perhaps be relevant to investigate your gradients and see how they change. Maybe this blogpost can help you out: machinelearningmastery

1

According to your definition (consecutive, order matters, max +/-2 difference), it's not a fuzzy matching case. It's just a minor variant of searching a subsequence: for i=0 to len(source)-len(test) { j=0 while (j<len(test)) && (abs(source[i+j]-test[j]) <= 2) { j++ if (j == len(test)) { // match found } } This is the simple ...

1

The first step would be to find how similar a candidate number is against any number in the reference list. I think this is a perfect case for a character-based string similarity measure, typically the Levenshtein edit distance. In case it's possible to have several matches, there could be a second step which would predict the most likely match, maybe based ...

1

The trick is to convert ODE/PDE into an integral equation and then Monte Carlo comes to play. Here are some examples. http://jotterbach.github.io/2018/08/08/MonteCarloODE/

1

I am not sure why your MinMaxScaler didn't work, but here is a function that should scale your data into the desired range: def rescale(data, new_min=0, new_max=1): """Rescale the data to be within the range [new_min, new_max]""" return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min Looking at the documentation of ...

1

Are there publications which mention numerical problems in neural network optimization? Of course, there has been a lot of research on vanishing gradients, which is entirely a numerical problem. There is also a fair amount of research of training with low precision operations, but the result is surprising: reduced floating point precision doesn't seem to ...

1

Because of all the data you have is well defined I would suggest you a categorical encoding, which is also easier to apply.

Only top voted, non community-wiki answers of a minimum length are eligible