Tag Info

4

I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other resources, like data science books. Together with the repo, I suggest the TidyTuesday videos by David Robinson, where he creates screencasts of complete data ...

3

I think it would be better to use a standard scaler that removes the mean and divides by the standard deviation. See here for more info and an implementation using sklearn. Why? At least you should be aware that dividing by the maximum could hide smaller effects. In the case you have an outlier that has a very high value, you would loose the small changes in ...

3

Some feedback/tips/tricks/opinions here: Problem setup Including requirement analysis. Gotta decide how the system/solution should work, how to know ho how well we are doing, and then how to get there. Model evaluation. It is very desirable to have a quantitative way to evaluate our model performance. For that we want some labeled data. It is very quick to ...

2

You could create a second label for your usernames according to whether they contain london or not (pseudocode below): for idx, username in df['Usernames']: if 'London' in username: df['London'].iloc[idx] = 1 else: df['London'].iloc[idx] = 0 Consequently given you want to go with correlation and that you are comparing binary ...

2

In short, I think the focus of your question is mostly on how to deal with covariates (watch out for the not always clear use of this term and its synonyms - see here and here. What I mean by covariates is "In statistics, a covariate represents a source of variation that has not been controlled in the experiment and is believed to affect the dependent ...

2

Well, there are a few ways to do the job. Here are some I thought of: Scatterplots with noise: Normally, if you try to use a scatter plot to plot two categorical features, you would just get a few points, each one containing a lot of instances from the data. So, to get a sense of how many there really are in each point, we can add some random noise to each ...

2

Target encoding calculated using an appropriate cross-validation strategy can also be powerful for high-cardinality categorical features. In some instances, frequency encoding can also be useful.

2

You could use Binary or BaseN encoding because they are commonly used with high dimensional nominal categories. Because binary or baseN encoding encodes the categories into ordinal numbers and then into binary or base of N respectively in an effective way. They are so efficient in terms of complexity and the dimensionality. I recommend you to read this for ...

1

So, Clustering is "Unsupervised" learning : You make groups in which elements look like each-other. In Unsupervised learning, you don't have a Label that you look for. Here, your problem is to Classify text between 3 categories : Sports, Foreign, Local. Those 3 categories ARE labels : You know you have news about those 3 subjects, and want to make ...

1

The standard method to encode a categorical variable is one hot encoding. Replacing categories with numbers (ordinal encoding) would certainly introduce errors in the model because it would rely on numerical comparisons which are meaningless with categorical values. The high number of dimensions can be a problem if the number of instances is too low and/or ...

1

First let me answer your specific question: If you want to decide which feature of two highly correlated, high impact features I would look at the following additional attributes of your features: How is the data quality or amount of data? Is one better or higher than the other? Choose this one. Is it in any way harmful to remove one of the features? If yes,...

1

The concept of cross-correlation is used in signal processing to find delay in signal and also in image processing to match-images(known as template matching) The general approach is to go on shifting the signal by 1 and compute correlation and find where the maximum values, here's a solution using matplotlib, if someone wants a ready to use implementation x=...

1

Doing EDA before is always helpful. It helps to create a better model. Suppose you have an image dataset and you do EDA by looking at the various sizes it contains and the various types of images it contains. It gives you an idea as to how big the image you are going to use in the network and also, what type of augmentations you are going to use. Same goes ...

1

You should start with exploratory data analysis to understand the data in as much detail as possible. Once you have a good understanding of the obvious patterns in the dataset, you should then turn your focus towards answering the less obvious questions using modelling techniques. The final item on the list that hints at building a recommender is an example ...

1

Okay, so since the visuals are done in excel, I think you can start with simple linear regression in excel. Im not an excel expert in this regard, but I think this article will be a good starting point https://www.ablebits.com/office-addins-blog/2018/08/01/linear-regression-analysis-excel/ So, what you can do: Get your data and fit linear regression on it. ...

1

This is a tentative answer. One can try to determine first the relative importance of each feature (eg by Factor Analysis or Principal Component Analysis, ..) Once the more significant features have been identified (or guessed perhaps), then one can try combinations of scores with these features. Take simply the average as a composite score. Do a linear ...

1

Question 1: Geograhic location, market sector, it all depends on what you're trying to achieve with the clustering. Question 2: Yes. Adding too many indicators will cause your data to overfit and at some point you'll end up with every cluster having 1 member. I would use a regularisation penalty term to counter this. Either L1 (Lasso Regression) or L2 (Ridge ...

1

Re-scaling means that you multiply your variable by a factor, i.e. $x \to x/a$. A normalization is a specific kind of re-scaling, where the factor $a$ is such that the values of $x$ become of order one. Its form depends on the context and what you are trying to do. Examples are $a= \langle x \rangle$ or $a= \textrm{max}(x)$. Perhaps the most common one is ...

1

About the question whether to scale only a subset of features, I would tell you to do it over all the features (at least the continuous numeric ones) since the goal of data-scaling is to put these data on the same "reference scale" to be fairly compared. Nevertheless, having mixed data types (continuous numerical, categorical...) for your ...

1

There's a lot to say on this topic. First of all, outside of academia, there is an unofficial classification (more slanted toward business types) of machine learning systems (and thus neural networks), which splits them into three categories in order of increasing utility/autonomy: descriptive, predictive and prescriptive. A descriptive system summarizes ...

1

To start with, you could use a simple thresholding. If you have the dataset $S$ where an element has the form $(x,y,c) \in S$, $x$ denotes the year, $y$ is a binary value (exam passed or not), and $c$ is the student id. you can obtain a classifier by using $\{(x,y,c) \in S \mid x \leq \theta\}$ and $\{(x,y,c) \in S \mid x > \theta\}$. Now you can check ...

1

I dont think there is one correct way, but what you can do is Use PCA if you have many features. This will reduce some number of features based on the amount of variance in each feature. You may use other dimensionality reduction techniques. You can use models like Lightgbm or random forest and know which feature are important. 3. You may use Lasso ...

1

The current approach use 70/30 or 80/20, the most used is 80/20 (train/test). However there is other things you should check, for example if you data is balanced. If your data is not balanced you might want to use undersample or oversample.

1

When making decisions about which data to use in a model you have to be aware of several pitfalls. One of them is information leakage i.e. including data that contains information that you shouldn't have at the time of prediction. Both Duration and Goals are data points that you do not have at the time of prediction (that is before a match) and therefore ...

1

Shouldn't an index have only one value in the feature axis? Yes, that's correct. On the graph given as example this is not visible because there are too many row indexes (50000). As a consequence it's impossible to distinguish a particular index from its neighbors, but if the X axis was stretched long enough one would see a single feature value for every ...

1

Rationale Some of the terms are a little vague, particularly what you refer to as eligible students and returned students. I'll set some variables for clarity, but tell me if I defined them incorrectly. I assume them to mean: eligible students $= A$ being the set of all students in the after-school program 2019-2020 returning students $= A\cap S$ where ...

1

Most probably, you are using Pearson's correlation method. This method is used for two Continuous features. Here, both the price_drop and the OHE features are Binary Categorical features. So, you can use these methods - Phi - Phi is a measure of the degree of association between two binary variables (two categorical variables, each of which can have ...

1

You could start with plotting the relation between any relevant pair of variables, typically with a simple scatter plot and possibly using color to represent a third variable. Pearson correlation coefficient is a simple but useful measure of association between two variables. By calculating it between two variables on the whole data and then on subsets based ...

1

Regarding the scores between 10-90, I'd think the training data could be such that there are very few samples in that set. This looks like a regression problem, try training an XGB regressor over your training data. The implementation for which can be found in sklearn documentation. It would help if your training samples are of good quality i.e. Sufficient ...

1

I assume that currently you're training a binary classification model, right? You could try training a regression model which predicts the score between 0 and 100. It would be better to have some examples in your training data which are between 10 and 90, because that would make the model learn the distribution of the scores. But even if you don't have this ...

Only top voted, non community-wiki answers of a minimum length are eligible