9
Don't worry - your hard-earned Python skills are still important ;)
Tableau is not a replacement - it is essentially a means of sharing your insights/findings. It is a wrapper around your normal toolkit (Pandas, Scikit-Learn, Keras, etc.). It can do some basic analysis (just using basic models from sklearn), but the powerful thing is it can deploy your ...
4
Could you reduce data types, for example, int32 to int16, but you should be careful, you have to be sure that you don't lose important a piece of information by reducing of memory.
Iteratively read CSV and dump lines into the SQLite table. Working on database is faster than on CSV file.
Use the library for parallel computing in Python like Dask or ...
3
There is the official answer and the realistic answer (from a business perspective):
Official
Officially the greatest thing your Python skills will bring you is flexibility. If you are going to run some economical model where you want to show a gradient uncertainty or something else crazy, doing that manually in any Data Visualization/Business Intelligence ...
2
Try to use PySpark and its optimizations for huge amount of data (paralellization, batch reading, etc.)
Do your pre-processing step by step and save the result of each step in a file/table.
Do your pre-processing in one DataFrame and at the end just write it in your files, just truncating it in order to get X, Y and X_test set.
2
The choice is mostly about your specific task: what do you need/want to do?
Many-to-one (single values) models have lower error, on average, since the quality of outputs decreases the more further in time you're trying to predict. Many-to-one (multiple values) sometimes is required by the task though.
An alternative could be to employ a Many-to-one (single ...
2
First: I think you want the product functionality, not zip, since you are checking every df with every ref. In zip, you would check df_a with ref_1 and df_b with ref_2 only.
Second: Your can look at the equation $(1+2+3+4)−(5+5+5+5)$ as $(1-5) + (2-5) + ...$ which is simply subtracting data frames and sum over columns.
With these two consideration, ...
2
LabelEncoding your features is a bad practice
You should avoid using LabelEncoder to encode your input features! Don't believe me? Here's what scikit-learn's official documentation for LabelEncoder says:
This transformer should be used to encode target values, i.e. y, and not the input X.
That's why it's called LabelEncoding.
Why you shouldn't use ...
1
As someone who worked on a competitor to Tableau, Data Science skills have largely superseded the need for Bi Software for data munging, complex analysis and ad hoc reports.
But BI Software can still be beneficial if you need to deploy your results to lots of people, often with varying rights to view something (e.g. you can only see your performance stats, ...
1
To answer your question no.
The term "imbalance" usually refers to classification problems. For your case, i.e. a regression problem you can only look at the distribution of your target variable.
If by "balance" you mean them having a uniform distribution, you could argue that they are, if fact imbalanced. However, I'd argue that this is not the problem ...
1
Do you think I implemented the code in the right way?
Code is correct, but he is minimal possible.
The features (X) have low correlation with the labels (Y).
It's could be the biggest problem, features have to have correlation with labels.
Do you have any suggestion to enhance the accuracy?
Make transformation of labels(X). Preparing of labels its ...
1
What you are referring to is a problem of estimation regression model uncertainty.
The uncertainty estimation method depends on the model that you are using.
Take a look at this tutorial, it provides a detailed explanation of what to do when you are using Linear Regression. It also points to the more sophisticated paper, which describes what to do when ...
1
After you apply dropna on your out, some rows are removed but the same is not done for your input. Hence, they have different number of rows, leading to your error.
1
One option is to move to a cloud computing service and rent a larger, faster computer that is not memory constrained.
1
What you want to do is exactly the default behavior of the category type.
Convert your month value to the type category declaring all months (it has a somewhat weird interface to create a categorical type)
df.month= dd.month.astype(pd.api.types.CategoricalDtype(categories=range(12)))
df.month.value_counts()
will give you:
id code month
sally s_A ...
Only top voted, non community-wiki answers of a minimum length are eligible
