Hot answers tagged

41

Yes, you put it quite correctly. As a teacher, you wouldn’t give your students an exam that’s got the exact same exercises you have provided as homework: you want to find out whether they (a) have actually understood the intuition behind the methods you taught them and (b) make sure they haven’t just memorised the homework exercises.


11

It is wrong because: it is fundamentally incorrect (a theoretical concern) it leads to bad results (a practical concern) It is fundamentally incorrect because usually the objective of testing a model is to estimate how well it will perform predictions on data that the model didn't see. It's quite hard to come up with good estimates of real-world ...


8

What are Bias and Variance? Let's start with some basic definitions: Bias: it's the difference between average predictions and true values. Variance: it's the variability of our predictions, i.e. how spread out your model predictions are. They can be understood from this image: (source) What to do about bias and variance? If your model suffers from a bias ...


8

One way or another any data can be represented in a table or even in a big binary string, since after all the physical memory of a computer is just one big binary sequence. But the question is whether the table format adequately represents the semantics of any data, and the answer is definitely no: while there are tabular representations for graphs, text, ...


8

It can happen that the model you train learns "too much" or memorizes the training data, and then it performs poorly on unseen data. This is called "overfitting". The problem of training and testing on the same dataset is that you won't realize that your model is overfitting, because the performance of your model on the test set is good. ...


7

Unfortunately I don't think a generative model could prevent from leaking private information from the original dataset. Like any other kind of model, the generative model is based on the values obtained from the training data. The idea of using such a model in "generation mode" is indeed interesting since it would make it difficult to reverse-engineer the ...


7

Yes it is. That is, in theory atleast. So we already have mathematical tools to prove whether privacy (and how much-thats the parameter epsilon) is perserved. Its called differential privacy. I highly recommend this non technical introduction Long story short, we coud let a generative model learn a prediction function of a model that guarantees privacy. ...


6

Simple answer: circular reasoning. The fact that your model "knows" the answer to something you've already told it the answer to really doesn't prove anything. Put another way: the entire point of testing is to get some sense of how well your model would do with data it hasn't seen yet, and testing it with data that it has already seen doesn't do ...


5

How to fetch Kaggle data from python code? Install kaggle package C:\Users\TalgatHafiz> pip install kaggle login to your Kaggle account click on the icon in the upper right corner -> My Account Scroll down to API section Click "Create New API Token" "kaggle.json" file is created and saved locally Create ".kaggle" dir C:...


4

If you want to test whether your algorithm works as expected, I'd use sklearn datasets. They allow you to create simple synthetic 2D data with certain properties: circles, half moons, etc. If you want "real" datasets, here is an interesting resource found after a brief search: https://uni.hi.is/helmut/2019/06/20/datasets-for-dbscan-evaluation/ It seems to ...


4

Can I use any machine learning methods having only one feature? Yes! In fact, many NLP classifications tasks are in this format. Given 1 piece of text, classify something. For example: Given 1 review, classify the sentiment Given 1 news article, classify the topic Given 1 chat message, classify the intent And now you have: Given 1 name, classify the ...


4

One option would be to feed an array of both variables to the stratify parameter which accepts multidimensional arrays too. Here's the description from the scikit documentation: stratify array-like, default=None If not None, data is split in a stratified fashion, using this as the class labels. Here is an example: import numpy as np import pandas as pd ...


4

Usually predictive power refers to the model, rather than the data. I've occasionally seen some people use it in the way that the author of your book uses it (see this for example). In the context of your book, yes, predictive power refers to whether input can be mapped to target output $X\rightarrow Y$. We can infer a dataset's "predictive power" ...


3

First: I think you want the product functionality, not zip, since you are checking every df with every ref. In zip, you would check df_a with ref_1 and df_b with ref_2 only. Second: Your can look at the equation $(1+2+3+4)−(5+5+5+5)$ as $(1-5) + (2-5) + ...$ which is simply subtracting data frames and sum over columns. With these two consideration, ...


3

You can have a look at the kaggle stock dataset. https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs This questions are normally done in OpenData stack exchange. https://opendata.stackexchange.com/


3

So, after a lot of digging, I found something in the comment section. They are document embeddings. There is a github repo that specifies an API. Paper on arxiv Example usage of a similar approach Relevant Comments from the Kaggle Comment section on the Data Update Log for the CORD19 Dataset: Comment 1 Comment 2 Examples how to visualize the ...


3

Here is some script for "openml" collection of datasets. Hopefully one can provide something similar for other databases. #see docs: https://docs.openml.org/Python-guide/ !pip install openml import openml import numpy as np import pandas as pd import time # Get information on all collection of openml datasets: datalist = openml.datasets....


3

OpenML has a gallery of different use case examples, including browsing and downloading datasets through python, and running benchmarks: https://openml.github.io/openml-python/master/examples/index.html When you want to benchmark new algorithms, this is the gist: import openml from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import ...


3

Not sure that's impossible but some data is inherently hard to represent tabularly : Cloud points like the ones processed in this paper : https://arxiv.org/pdf/1612.00593.pdf Similarly time series with variable sampling frequencies Graphs that evolve in time like the relations in a social network


3

It looks like that your prediction is clamping at 750. Be mindful of the fact that Tree can't predict a Regression value that is outside the range it has been trained on. So, first of all, please assure that your data doesn't have a trend.


3

Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic? If the model memorizes the training data when that same data is used for the "test" set, it would still memorize the training data when different data was used for the "test" set. Using a separate "test" set cannot ...


2

In general tidy data is great... but it can quickly become unreasonably large. This is the main reason why I usually try to refactor my data in a tidy format as late as possible in the process. Example: imagine a dataset containing $N$ instances, with columns feature1 ... featureX and result1... resultY, where the result? columns represent some value based ...


2

Well, what you should do depends highly on what your goal here is. Do you - for whatever reason - have to use the given test and training sets as such? If not, a valid approach would be to shuffle the available data and determine a new test and training set, so that they each contain all classes. Do you want a classifier only for d and e, and have to use the ...


2

Any good hash will be uniformly distributed, which means that you can assume a uniform distribution when you apply modulo n, as long as $n < 2^{M/2}$, where M is the number of bits in your hash, see here. So for SHA1-32 you would at most modulo by $2^{16}$. There is no approach to calculating an integer value; what you have there is an hexadecimal ...


2

I was facing the same task. However, all the provided answers have one flaw: They save the large file to EC2 disk. This has the drawback that you need to make sure that there is enough disk space on your instance and/or that the file system is encrypted (if you process sensitive data). So, better avoid saving to disk. Instead, stream the curl stdout as ...


2

For anyone who's still facing the issue: None of the other suggestions worked for me or was too much work to do. I simply replaced all \n with \\n before saving to CSV and it'll preserve the newline character. df.Column_Name = df.Column_Name.apply(lambda x : x.replace('\n', '\\n')) df.to_csv("df.csv", index=False)


2

Since the variables isPWKinText and H1_2_Len have the same value for all examples in your dataset they have zero variance and contain no information. There is simply just no inference you can make based on them. That is why they are not relevant and shown blank. Please note, that this depends on your dataset, of course. The two variable might just be ...


2

Replacing missing values with mean, median and mode is feasible when the number of missing data is not small enough. In your cases, only a handful of data are missing. We can use the rest of the data to come up with estimates better than mean, median and mode. Let's see the possible approaches: Scenario 1: using the reading for breakfast and lunch for day ...


2

You can use flatten numpy for this task. Simply convert your data to a numpy array and use the function as: In [1]: import numpy as np In [2]: x = [[[1,2], [3,4]]] In [3]: x Out[3]:...


2

Discrimination is the separation of the classes while calibration gives us scores based on risk of the population. For example, there are 100 people that we’d like to predict a disease for and we know that only 3 out of 100 people have this disease. We get their probabilities from our model. Due to good predictability power, our model predicts probabilities ...


Only top voted, non community-wiki answers of a minimum length are eligible