New answers tagged dataset
0
Dynamic Time Warping measures the similarity between two sequences of values across time. It combines the idea of comparing a pair of points $(a\in A,b\in B)$ with the idea of finding the optimal alignment of points $(a,b)$ between the two sequence. The alignment algorithm is similar to the edit distance between strings, and can be used to extract a mapping ...
1
For this task, you need to decide 3 things:
(1) What are the allowed transformations.
For example do you allow an arbirtrary affine transformation (rotation, scaling, translation). Then you need a matrix $f \in \mathbb{R}^{3 \times 3}$ (using homogeneous coordinates) that maps the points from the left image to the right one. Each entry of the matrix is a ...
0
There are several approaches to this as you need both the input (images) and if your problem is a classification one, you need to reliably store the labels. You might also have some additional information about the images that could be useful for your problem:
you can store the images in such a way that all information is contained in the permanent store (...
1
There are many methods to connect two different kinds of datasets
Python Pandas - Merging/Joining
left − A DataFrame object.
right − Another DataFrame object.
on − Columns (names) to join on. ...
left_on − Columns from the left DataFrame to use as keys. ...
right_on − Columns from the right DataFrame to use as keys. ...
left_index − If True, use the index (...
1
I dont think there is one correct way, but what you can do is
Use PCA if you have many features. This will reduce some number of features based on the amount of variance in each feature. You may use other dimensionality reduction techniques.
You can use models like Lightgbm or random forest and know which feature are important. 3. You may use Lasso ...
0
See if this can help - Publicly Available Datasets
Also you can use SMOTE technique if you have insufficient data.
0
In reference with your little found data either augment it or apply cross validation on top of it.
else
Look for your expected data in https://datasetsearch.research.google.com/
0
If I understand well your main issue is to transform some Categorical variables into Continuous, or some Continuous into Categorical.
You have many ways to do that :
Continuous to Categorical :
You can yourself set intervals, transforming your continuous value into a category. For example, for a variable Age, you define the following intervals : [0;10[, [10;...
0
Add an indicator column while concatenating the two dataframes, so you can later seperate them again:
df = pd.concat([test.assign(ind="test"), train.assign(ind="train")])
Then later you can split them again:
test, train = df[df["ind"].eq("test")], df[df["ind"].eq("train")]
0
Method 1:
Develop a function that does a set of data cleaning operation. Then pass the train and test or whatever you want to clean through that function. The result will be consistent.
Method 2:
If you want to concatenate then one way to do it is add a column "test" for test data set and a column "train" for train data set. Perform you ...
0
There are several methods to choose from. If you insist on concatenating the two dataframes, then first add a new column to each DataFrame called source. Make the value for test.csv 'test' and likewise for the training set.
When you have finished cleaning the combined df, then use the source column to split the data again.
An alternative method is to record ...
0
not sure if I understand correctly but it might be done with a command like this
df['category'] = np.where(df['category']!='0', '1', '0')
0
In general it is better to have a good mix of positive and negative labels for a model to learn from. That said, many classification problems suffer from significant label imbalance (e.g. fraud detection).
I don't see an obvious problem with a 70/30 mix of labels, but as you start modeling be sure to check the confusion matrix and look for where you model ...
0
You haven't mentioned any language preference but there are ways to do pattern recognition - I personally recommend using Python. The K-nearest neighbors, decorrelation and a dissimilarity matrix are a few ideas once you have defined what a pattern is.
Since you mentioned that you are new to all this - here are some really good sources to understand what ...
0
You can find the dataset from the following links.
IEEE Dataport (https://ieee-dataport.org/datasets)
https://datasetsearch.research.google.com/
1
Rather than asking about the predictive power of a dataset, I think it's intuitive to ask about the predictive power of a model. My reasoning is as follows;
A dataset can be univariate, bivariate or multivariate types. The dataset can contain only numerical features or categorical features or both. Suppose there is a univariate dataset with a negative skewed ...
1
For anyone who's still facing the issue:
None of the other suggestions worked for me or was too much work to do. I simply replaced all \n with \\n before saving to CSV and it'll preserve the newline character.
df.Column_Name = df.Column_Name.apply(lambda x : x.replace('\n', '\\n'))
df.to_csv("df.csv", index=False)
0
Instead of downsampling the dataset, you should try using the Upsampling technique. Sometime downsampling leads to loss of data. Use augmentation techniques to increase the size of your dataset.
0
These are some ways by which you can improve your model's accuracy and reduce overfitting.
Split your dataset into training, validation, and testing.
Add dropout.
4
Usually predictive power refers to the model, rather than the data. I've occasionally seen some people use it in the way that the author of your book uses it (see this for example).
In the context of your book, yes, predictive power refers to whether input can be mapped to target output $X\rightarrow Y$. We can infer a dataset's "predictive power" ...
1
I can think of these possible solutions:
The basic one, club the whole data and try different algorithms and evaluate the results.
If Age distribution among the different samples(data set) are not proportionally distributed, i.e. if your dataset have huge samples of 'Young' in comparison to 'Adult' one or vice-versa, then I definitely try to tune my model ...
0
Hope this link would help you. It is always better to have more data, so concatenating the data sources would at least slightly increase the performance of you model than it used to be.
0
It's important to remember you can always save objects (like dicts or json) into individual cells in Pandas. Especially if you're not sure of how you'd to analyze at the moment.
The Google Analytics Customer Revenue Prediction data uses a lot of JSON
You can see how people analyze the data on the Notebooks section too
https://www.kaggle.com/c/ga-customer-...
0
Seems to me the best way to do this is to train a single model on all of your data and let it sort out the features that distinguish one data set from another.
If you don't know in advance which set you're getting data from, you can't know which model to choose so having more than one would limit you
1
Add all data together, and make sure you have features representing all possible insights. In your case one feature with age/maturity (young, adults...). Lets say you fit a decision tree (or Random Forest, gradient boosting...) the model will decide whether to do a split or not on this feature if it contains meaningful information.
If you combine you should ...
1
No, that isn't what it means.
For one thing it is not clear what parameter the confidence interval that you calculated is for.
In any case, some care is needed in the interpretation of (frequentist) confidence intervals.
In frequentist statistics, a confidence interval is random, and the parameter that the interval is for is fixed. In the case of a 99% ...
1
I could think of 2 solutions:
Since you mention stripping of the words why not make it a 2 step program where in the first classifier is a binary where in 1-3 is one class of Actions performed and the second class is 4 where there is No Action performed. If the word happens to be in the first category you can further run it for classification in between the ...
1
Just imagine it practically, If class A data is 90% and class B data is 10% ,then if you just randomly classify the label you prediction as class A then your accuracy will be 90%.
So biased data will lead your model to be biased over the class which has more data as it will give better predictions in your model.
0
On the basis that difficult words are difficult because they are not commonly used, I think something as simple as TF-IDF would work well.
0
You should scale your data before training your data. Try something simple:
df = (df - df.min()) / (df.max() - df.min())
If you're using the L2 distance in your knn model this means some values are squared (which is probably why you max out the float64). Btw float64 max is $2^{31} − 1$ so check the range of the columns just to be sure there might be an ...
0
I'm almost certain that you have missing values, check that and fill them beforehand
Assuming you data frame is called df
df.info() will give you that info
Additionally df.describe() is a good method to validate the maxmimum value is not np.inf
0
If I merge on dates, I'd have multiple repeated rows with fires occurring at the same time in different places, would that be the best way?
Probably not, since you don't want to lose the location information. You should probably find a way to map the latitude/longitude to borough/county between the two datasets, so that you obtain a semantically consistent ...
0
Alright, I believe I found one:
https://github.com/dchen236/FairFace
1
You can use inner join:
import pandas as pd
df1 = pd.read_csv('file1.csv')
df2 = pd.read_csv('file2.csv')
df = pd.merge(df1, df2, on="Column1", how="inner")
0
you know, when they say the business team needs insights, it doesnt always imply machine learning.
You could also do a lot of exploratory analysis and visualize spending trends, seasonality among the demographic spending, when the customers are most active during the day, highlight the towns with the highest income growth rate, which age group is your ...
0
One way to do is to use target encoding:
https://medium.com/analytics-vidhya/target-encoding-vs-one-hot-encoding-with-simple-examples-276a7e7b3e64
https://towardsdatascience.com/target-encoding-and-bayesian-target-encoding-5c6a6c58ae8c
(There are a million resources to learn target encoding)
This way your categories will not only be ordered by the number ...
0
It seems that replacing "updates.shape[0]" with "tf.shape(updates)[0]" solves that particular problem. However, this leads to another problem described below.
import numpy as np
import tensorflow as tf
def main( ):
inputData = np.zeros((1000, 3), dtype=np.float32)
inputData[:,0] = np.sin(np.arange(1000)/360)
inputData[:,...
Top 50 recent answers are included
