4
There are 2 things you can do here:
1.) Use libraries like Dask to speed up your data preprocessing. Here is the link
2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...
3
The term learning curve can mean different things in different context, which is confusing.
When talking about neural networks (and other iteratively trained models) the learning curve describes the model's training progress. It is often used to determine when it's time to stop training.
In scikit-learn, the learning curve is interpreted differently. It ...
2
A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort.
For the initial data exploration and ...
2
The values for LOSS TOK2VEC and LOSS NER are the loss values for the token-to-vector and named entity recognition steps in your pipeline. The ENTS_F, ENTS_P, and ENTS_R column indicate the values for the F-score, precision, and recall for the named entities task (see also the items under the 'Accuracy Evaluation' block on this link. The score column shows ...
2
Even though virtually any supervised classification algorithm can be used when having categorical features by applying some encoding technique, my first thought is using Catboost, an algorithm specially designed just for handling categorical features without a necessary explicit preprocessing/encoding phase. In short this algorithm will use an adaptation of ...
2
You can try applying your preprocessor to your X_train and X_test:
preprocessor = ColumnTransformer(
transformers=[
('num', numeric_transformer, numericas_all)
,('cat', categorical_transformer, categoricas_all)
])
X_train_pipe = preprocessor.transform(X_train)
X_test_pipe = preprocessor.transform(X_test)
Edit:
Since you did not use any transformer that ...
1
Preprocessing is needed for both train and test sets. But you should be aware of data leakage, meaning no information from the test set should be used to preprocess the training set.
For example, if you are trying to apply One-Hot encoding to your classification labels you should train the encoder (e.g. sklearn.preprocessing.OneHotEncoder) on training set ...
1
The path you are providing to the flow_from_directory method is one level to deep. The data generator expects a path to a directory which contains one subdirectory for each class in your dataset, see tensorflow documentation. This github gist shows how to apply the ImageDataGenerator to a dataset (coincidentally also using 'cat' and 'dog classes') together ...
1
It is correct that calling learning_curve will refit your model multiple times for different training dataset sizes. You can simply pass specific hyperparameters when initializing the model you want to use, which you can then pass to learning_curve for the estimator argument. The actual loss funtion that is used depends on the type of estimator you are ...
1
There are some ML models which use both categorical and numerical data
Decision trees(with bagging),
Random forest(with bagging & random
subspace)
Naive Bayes(numeric by Gaussian distribution or kernel
density estimation)
KNN based approach
Ensemble Techniques
linear regression
Note: you can always use different encoding techniques to transform ...
1
Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala/spark/hadoop if the problem wont fit on a single machine.
1
You could use np.where, but just using a series comparison should work as well:
df["col"] = (df["A"] > 2) & (df["B"] < 30)
1
I answer a similar question here https://stats.stackexchange.com/a/548006/226796
Hope that it helps you to understand the essence better. Below is my answer.
The two forms of rewards are equivalent in the following sense. Now suppose you have an offline dataset consist of s1,s2,..,sk with reward r1,r2,...,rk. Based on the data, you can estimate the MDP model ...
Only top voted, non community-wiki answers of a minimum length are eligible