New answers tagged dataset
0
Why don't you try contacting him via Twitter and ask for permission?
https://twitter.com/favstats?lang=es
Maybe checking his blog you can find the data needed
https://favstats.eu/post/demdebates/
1
Most classical machine learning algorithms assume targets are labeled completely correctly. If you think your labels are noisy, it would make sense to apply techniques to handle that issue.
As far as splitting a dataset into subgroups to improve performance, you are describing any tree-based machine learning algorithm. The goal of any tree-based learning ...
0
Honestly once you get to something serious (big or evolving) the main problem is about sharing and updating data. Once a solution is devised for data, it is not really hard to adapt it to models.
Depending on the volume and the usage, the data can be stored, exchanged and acessed in a wide range of solutions. It might be old historical/external solution with ...
0
One possible option would be operate directly on the zip files using zipfile.ZipFile.
Counting the number of items in a zip file:
from contextlib import closing
from zipfile import ZipFile
with closing("/content/gdrive/My Drive/path.zip") as zip_file:
count = len(zip_file.infolist())
0
Since Invoice analysis comes under trend analysis. It's better to take a look at the data from a statistical point of view. Plot graphs, manually check the relation and so on. if there is a trend which is not monotonic, or if it's recurring weekly it's better to take a look at (Recurrent Neural Network)RNN's to predict based on the day or week.
1
If the model is a discriminative model (e.g. a classification model), it is highly unlikely that you can identify whether it was trained with some specific dataset.
If the model is generative, (e.g. a language model or a machine translation system), you may be able to try to identify if the model was trained with your data by trying to extract from it ...
1
A slightly hacky way to get there maybe but you can do this to get what you want from the second table;
df2['count'] = 1
pivot = df.pivot_table(df, index='userid', columns='productid', values = 'count').reset_index()
pivot = pivot.fillna(0)
You would then want to merge this to the first dataset like this;
finaldf = pd.merge(df1, pivot, left_on='userid', ...
3
It looks like that your prediction is clamping at 750.
Be mindful of the fact that Tree can't predict a Regression value that is outside the range it has been trained on.
So, first of all, please assure that your data doesn't have a trend.
1
If my understanding is right, you have a regression problem, with categorical features with high cardinality and "outliers" (or just big numbers).
How have you encoded categories? Target Encoding?
There is another option that is not encoding with the mean but with the median that on some cases can perform better.
On this notebook , you can see an ...
0
After having a look at the article you quote of KDNuggets, in this article they analyze the data from this last year (poll), but they have been doing that for several past years.
Perhaps by looking at past years you can extract some better data.
Also in this other forum https://opendata.stackexchange.com/
they talk about open data, maybe here you can get a ...
0
When to scale or normalize a column?
When you are using an algorithm that assumes your features have a similar range, you should use feature scaling.
The main example is distance algorithms like Euclidean distance.
Thus, the algorithms that are distance-based algorithms like K-means, SVM, etc also require feature scaling.
Gradient Descent Based Algorithms ...
1
You can find a complete repository of around 80 multi-label datasets here :
0
If you are using a sparse matrix, you can split train/test dataset using this function from LightFM:
lightfm.cross_validation.random_train_test_split
The link above says:
Randomly split interactions between training and testing.
This function takes an interaction set and splits it into two disjoint sets, a training set and a test set. Note that no effort is ...
1
Your test and validation dataset should reflect the type of data you would expect when you deploy your model in the actual setting. So usually you do not apply augmentation to the validation and test dataset, since in the real setting you will not receive some strange augmented images.
Another way to think of it is if you apply augmentation to your ...
0
Actually you should never use any sampling techniques on your testing/evaluation data because this could lead to wrong classification results.
If your dataset is imbalanced you could perform upsampling or downsampling techniques (like SMOTE) on your training data only.
If you want to benchmark your multi-class classification you need to rely on e.g. the ...
1
TL;DR
While the data comes from the NYTimes and seems legit, the presentation is intentionally misleading and the subsequent assertions are baseless. I say "intentionally" because an unbiased and reputable analysis would not propagate such major allegations from the data they have presented. The data does not prove nor disprove voter fraud, so the ...
0
I read the thread but didn't analyze the data.
It's very difficult to answer this question in any conclusive way: assuming the graphs are correct, interpreting them is a highly risky/subjective game because there's so many hidden factors: the way votes are collected at different times, in different places which have different population density, under ...
1
Strictly theoretically it makes no difference on DNN, I answered it today here and I said:
Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. ...
0
Differential privacy. Where you can inject targeted noise anonymise the data, but you can still make inference.
1
If you check the pypi source package you can see that they are most likely depracated and they advise you to use imbalanced-learn instead.
1
As you can see from the Train and Validation loss (and also accuracy). While your model is able to learn, your validation results do not improve. This means underfitting, or in this specific case, your validation data is different from your train data.
The reason is following:
training_generator = DataGenerator(train, **params)
validation_generator = ...
8
One way or another any data can be represented in a table or even in a big binary string, since after all the physical memory of a computer is just one big binary sequence. But the question is whether the table format adequately represents the semantics of any data, and the answer is definitely no: while there are tabular representations for graphs, text, ...
3
Not sure that's impossible but some data is inherently hard to represent tabularly :
Cloud points like the ones processed in this paper : https://arxiv.org/pdf/1612.00593.pdf Similarly time series with variable sampling frequencies
Graphs that evolve in time like the relations in a social network
2
Generally numbers (percentages) do not matter.
What matters is that your Splitting (Train/test/Validation) does 2 things. Represent the real world sitatution and making sure the model can generalise given that ist evaluated on the holdout sets.
So what does that mean here exactly? You have 30k Images and 400 patients. Most likely patients(scans) will differ ...
0
Generally you should have a 60% train dataset and a 20% validation as well tests set. I'm not familiar with the tumor segmentation thing but as long as the images for the same pacient are different and with a relevant level of difference that must be enough.
1
You can use the following techniques to mask sensitive data:
Substitution cipher - any character of plain text from the given fixed set of characters is substituted by some other character from the same set depending on a key.
Tokenization masking - mask source string data based on criteria that you specify in an algorithm.
Principal Components Analysis (...
Top 50 recent answers are included
