Hot answers tagged

8

What are Bias and Variance? Let's start with some basic definitions: Bias: it's the difference between average predictions and true values. Variance: it's the variability of our predictions, i.e. how spread out your model predictions are. They can be understood from this image: (source) What to do about bias and variance? If your model suffers from a bias ...


7

Unfortunately I don't think a generative model could prevent from leaking private information from the original dataset. Like any other kind of model, the generative model is based on the values obtained from the training data. The idea of using such a model in "generation mode" is indeed interesting since it would make it difficult to reverse-engineer the ...


7

Yes it is. That is, in theory atleast. So we already have mathematical tools to prove whether privacy (and how much-thats the parameter epsilon) is perserved. Its called differential privacy. I highly recommend this non technical introduction Long story short, we coud let a generative model learn a prediction function of a model that guarantees privacy. ...


4

First, find out when the user switch and give a separate id to each message group: df['group_id'] = ((df['user'] != df['user'].shift()).cumsum()) user message group_id A Hi. 1 B Hello. 2 B How are you? 2 A I am stuck. 3 B How can I help you? ...


4

TL;DR Yes, with overfitting all data becomes (non-linearly) separable (as long as the points don't precisely overlap). Explanation The problem with your argument is that you are using circles on a 2D plane, which is very difficult to learn. However, I think your argument can be made stronger with a decision-tree. (0.2, 3.1)? --> yes -> star ...


4

If you want to test whether your algorithm works as expected, I'd use sklearn datasets. They allow you to create simple synthetic 2D data with certain properties: circles, half moons, etc. If you want "real" datasets, here is an interesting resource found after a brief search: https://uni.hi.is/helmut/2019/06/20/datasets-for-dbscan-evaluation/ It seems to ...


4

Can I use any machine learning methods having only one feature? Yes! In fact, many NLP classifications tasks are in this format. Given 1 piece of text, classify something. For example: Given 1 review, classify the sentiment Given 1 news article, classify the topic Given 1 chat message, classify the intent And now you have: Given 1 name, classify the ...


4

One option would be to feed an array of both variables to the stratify parameter which accepts multidimensional arrays too. Here's the description from the scikit documentation: stratify array-like, default=None If not None, data is split in a stratified fashion, using this as the class labels. Here is an example: import numpy as np import pandas as pd ...


4

Usually predictive power refers to the model, rather than the data. I've occasionally seen some people use it in the way that the author of your book uses it (see this for example). In the context of your book, yes, predictive power refers to whether input can be mapped to target output $X\rightarrow Y$. We can infer a dataset's "predictive power" ...


3

On the page you linked there is actually a Python example on how to get the data. It is in Python 2, but I will show you how to make it work in Python 3. import urllib import json # Used to load data into JSON format from pprint import pprint # pretty-print url = "https://data.sa.gov.au/data/api/3/action/datastore_search?resource_id=...


3

You can approximate a time series with a polynomial of degree n_degree by using ridge regression. You can try different degree numbers (e.g. [2,3,4,5,6]) and choose the best one. Keep in mind that higher degree models, always get lower error values. So you should somehow penalize higher degrees. from sklearn.linear_model import Ridge from sklearn....


3

The term you are looking for is stratified sampling : https://en.wikipedia.org/wiki/Stratified_sampling. It's a way to sample from population that can be partitioned into sub-populations. More specifically for your problem, look at what they do for clinical trials, when they parition patients in sub-groups.


3

train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples. --> can't that be solved by using stratify=True? --> yes, that's what stratify=True is for. However you still only train on the data of your training set and test with the data from the test set train_test_split doesn'...


3

I think it doesn't really matter that there can be negative values after as long as you rescale your data correctly at the end. Think about what positive/negative values mean for z-score. This has nothing to do with whether your use case (for example speed) can realistically have negative values or not. With z-score positive values simply mean that the ...


3

The original paper mentions two corpora: CoNLL 2003 (apparently here now) and the "CMU Seminar Announcements Task". However according to the page linked in the question the actual NER was trained on a larger combination of corpora: Our big English NER models were trained on a mixture of CoNLL, MUC-6, MUC-7 and ACE named entity corpora, and as a result the ...


3

There are no general rules such that n features, m observations with a X type learner give q accuracy. Your predictor depends on features in your model. Let's say you wanna predict volume of trade between two countries. And lets say (hypothetically) this trade regulated and its the only variable. If you add features that can explain regulation changes ...


3

Having consulted my professor, the person that wrote the question from the exercise book featured in the OP, here is their perspective: Groups of data points can always be separated. The exception is when two points are at the same location. However, the thing to consider is whether or not your decision boundary can separate unseen data, generated by the ...


3

I am building on the first part of @Dylan's answer: For general items like "dogs" pre-trained models are easily available. A good starting point is ImageNet. There are plenty of pre-trained models available for this dataset, e.g. see here for PyTorch. Since ImageNet includes multiple categories for a given item you can check this list to see which indexes ...


3

First: I think you want the product functionality, not zip, since you are checking every df with every ref. In zip, you would check df_a with ref_1 and df_b with ref_2 only. Second: Your can look at the equation $(1+2+3+4)−(5+5+5+5)$ as $(1-5) + (2-5) + ...$ which is simply subtracting data frames and sum over columns. With these two consideration, ...


3

You can have a look at the kaggle stock dataset. https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs This questions are normally done in OpenData stack exchange. https://opendata.stackexchange.com/


3

So, after a lot of digging, I found something in the comment section. They are document embeddings. There is a github repo that specifies an API. Paper on arxiv Example usage of a similar approach Relevant Comments from the Kaggle Comment section on the Data Update Log for the CORD19 Dataset: Comment 1 Comment 2 Examples how to visualize the ...


2

I was facing the same task. However, all the provided answers have one flaw: They save the large file to EC2 disk. This has the drawback that you need to make sure that there is enough disk space on your instance and/or that the file system is encrypted (if you process sensitive data). So, better avoid saving to disk. Instead, stream the curl stdout as ...


2

The graph resulting from this kind of dataset is also known as a Network Graph and the kind of analysis you are trying to do is known as Social Network Analysis. There are many prominent Python libraries for visualization and subsequent analysis of network graphs. The most widely used is NetworkX. It is easy to add nodes and directed edges in a NetworkX ...


2

Answering the questions, Indeed, if you aren't careful when creating dataset partitions, you can end up with unbalanced partitions. In that case, as you said, using stratified sampling can mitigate the problem. Another option is to perform data augmentation on feature space (i.e. SMOTE [1]). For comparable results, you should use a seed/random_state. That ...


2

You can think of the data distribution as the world that the model lives in. You want to train it to perform well in this world and to do this you have to train it on examples representing this world. Further you have to estimate its performance in this world by testing it on more examples from this world. If you train your model in one world, test and ...


2

There is no theory or general case that sets the size of dataset required to reach any target accuracy. Everything is dependent on the underlying, and usually unknown, statistics of your problem. Here are some trivial examples to illustrate this. Say want to predict the sex of a species of frog: It turns out the skin colour is a strong predictor for the ...


2

There is no general rule but you better to use such techniques (SMOTE, sampling etc) and try to obtain 50:50 if you can. You can utilize my another answer (just ignore multiclass part). My suggestion is you should create more samples by synthetic data generators like SMOTE. You have a model with very less observations (197). The model may not fit well thus ...


2

K-fold means that the validation step will be performed k times, each of them using a fraction $\frac{k-1}{k}$ for training and $\frac{1}{k}$ for validation. If you want a fixed validation fraction, choose the number of folds that fits: 90%/10% : 10-fold 75%/25% : 4-fold etc. I don't know about Matlab libraries or functions, but if you want to do custom ...


2

I'd start here. Most basic idea is to run statistical tests to see how target variable depends on each feature. These include tests like chi-square or ANOVA. Tree-based models can also output feature importance. Check this post. There's plenty of posts on kaggle with code. Might be worth checking those: https://www.kaggle.com/willkoehrsen/introduction-to-...


2

You said: Yes, I already ran few feature selection algorithms like SelectKbest, SelectFrom Model, RFE, Feature Importance etc which outputs both min and max. For example - Min_bp and Max_bp. When I did a sanity check by running correlation, I was able to see that they all are correlated. In general you have 2 options. You can remove features that ...


Only top voted, non community-wiki answers of a minimum length are eligible