New answers tagged data
0
Scikit-learn has an impute module that supports many of those imputation methods.
0
If you are going for the approach with train, validation and test sets and want to train the model using both train and validation sets than yes, it would be appropriate to perform new normalization on the data in its raw form. Than just like before, you use the mean and standard deviation to normalize test set data.
3
Scatter plot and box plots are the most preferred for visualizing outliers.
Parallel plots can also be utilized for detecting outliers. For large datasets it can be bit confusing, highlighting outliers comes in handy then.
parallel plot case study
outliers in parallel plot
1
No the cleanest solution but it works
currentyear = 2021
df["Vals"] = df.apply(lambda x: x["column1"] if x["y1"] == currentyear else x["column2"] if x["y2"] == currentyear else x["column3"], axis = 1)
Hope it helps
0
I think this should work
df['vals'] = [x1 if y1 == currentyear else x2 if y2 == currentyear for x1, x2, y1, y2 in zip(df['col1'],df['col2'],df['y1'],df['y2'])]
0
Correlation between categorical variables can be calculated with Spearman's rank correlation coefficient. If Spearman's rank correlation is high enough, the variables can be dropped.
2
One way is to convert .data file to excel/csv using Microsoft Excel which provides an option to get external data (use Data tab --> from Text). Check this video for demo
Other way you can utilize python to read .data files and directly work with dataframes
import pandas as pd
df = pd.read_csv("adults.data")
(OR)
df = pd....
0
This might helps you
Before loading the data make sure check the file format.
Load the required libraries
First read the excel file using 'pd.read' function
save the file into csv format using function 'to_csv'
. example:
pp = pd.read_excel('paul_palmer.xlsx')
pp.to_csv('paul_palmer.csv', index=False)
0
Another solution is to use pandas to read the file and then export it to an excel form as following:
import pandas as pd
df = pd.read_csv("file.data", sep=",", header=None)
df.to_excel("output.xlsx")
You could read more here
0
To deal with mismatch of timesteps, you have to find a balance between all those mismatched timesteps according to your objective.
For example, if your objective is to have a prediction for the next quarter, your learning data should be adapted to quarters.
learning data < 1 quarter: get the mean values of all weeks that last quarters.
learning data > ...
0
Association rules are a relatively straightforward class of algorithms. Those earlier papers cover most of the interesting properties association rules.
The field of recommender systems moved towards collaborative filtering and matrix factorization around that time. Those methods have increased empirical performance and are more interesting to study.
2
I believe you are looking to work along with the missing values in particular column(X) where column(W,Y,Z) have important values in these rows and can't discard or do imputation, especially for plotting them visually.
Yes its possible, considering:
When you only plan to plot other columns(W,Y,Z excluding column X) to view them visually
When you only plan ...
3
Finding corpora for NLP research can be hit and miss, my advice would be to study the availability of adequate data when deciding about the research direction, not afterwards. Of course this completely depends on the type of requirement for the data. In case you have to create your own corpus, design the corpus collection and annotation very carefully ...
1
It's not obvious but if you go resource>manage added data sources>edit action on source and then simply click the connection name it's actually a text box and you can rename it there.
2
If the range of possible integers is small, encode the presence of each integer as a boolean column in a feature vector.
Example with a value range of 0-5.
[1,3,4], [4,3,1], [3,1,4] would all be encoded as [0,1,0,1,1,0]
2
You can use a dimensional reduction algorithm like t-SNE:
https://youtu.be/wvsE8jm1GzE
https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html
It is quite easy to implement and you will see correlations between countries clearly.
Top 50 recent answers are included
