Syenix
  • Member for 2 years, 7 months
  • Last seen more than a month ago
  • Bangalore, Karnataka, India
1 answers
1 votes
57 views
Evaluating information extraction from structured documents
3 votes

I have worked with Structured text using OCR. OCR is prone to errors even while reading the content and a slight change in string arrangement would lead to false positives. I used Cosine Similarity ...

View answer
4 answers
8 votes
3k views
How to deal with spelling errors NLP
2 votes

You can try Flashtext to easily create alternatives as you find and FuzzyWuzzy to get the similarity between word tokens by n grams

View answer
2 answers
2 votes
102 views
Hyperparameter optimization, ensembling instead of selecting with CV criteria
2 votes

I think you are looking for hyperopts, Optuna and Gpopy for Hyperparameters search without burning much of CPUs.

View answer
4 answers
1 votes
305 views
Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?
2 votes

Firstly any imbalance class text classification will have biased towards majority population and will result in overfit/underfit. You can use Smote, imbalancelearn library and imbalance from Scikit ...

View answer
2 answers
1 votes
177 views
Which ML algorithm should I use for following use case for classification and Why?
1 votes

You can check https://autogluon.mxnet.io/ from autogluon import TabularPrediction as task predictor = task.fit(train_data=task.Dataset(file_path=TRAIN_DATA.csv), label=COLUMN_NAME) predictions = ...

View answer
2 answers
2 votes
1k views
Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?
1 votes

I think you have misunderstood the koalas library. You can say its Pandas on Distributed System. You can use Koalas similar to pandas. There are few drawbacks with respect to APIs which is documented ...

View answer
2 answers
1 votes
999 views
How to extract entities from text using existing ontologies?
1 votes

I think you are looking for Spacy, Polyglot and AllenNLP to find your NERs.

View answer
1 answers
2 votes
15 views
Technique to determine variation in metric due to varying parameters
Accepted answer
1 votes

If the variation in values of feature then should not you remove the column? Your model will not learn from it if the variance is low. you can Q-Q plots when you vary the features to check how close ...

View answer
2 answers
0 votes
24 views
Support Vector Machine analysis - Python, how do I determing whether to use linear, square, or other types of SVM models?
1 votes

By Default, SVM in Sklearn uses RBF Kernel. You have to try out all the 3 kernels, with different Gamma and C. SVM treats outliers better and add a penalty on every outlier it detects. You should ...

View answer
1 answers
0 votes
20 views
How to define the adequate cash prize sizing for hosting a Kaggle or similar compeition?
1 votes

This is a generic question that may need more than just cash prize. 1) Organizational Reputation 2) Dataset Size 3) Defining Problem statements with Domain knowledge 4) Time period 5) Computation ...

View answer
2 answers
1 votes
161 views
Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?
Accepted answer
1 votes

Looking at your use case, Dask, H2O, Modin, Koalas and Vaex would better for scaling your data preprocessing apart from Pyspark. They have API's similar to pandas thus porting your existing code would ...

View answer
2 answers
2 votes
211 views
Training machine learning models from log files
1 votes

Anytime that you are looking at logs, basically you want to do anamoly detection. There are various methods I have tried and want to use few more: 1) Topic Modelling 2) Supervised Learning and can ...

View answer
3 answers
8 votes
6k views
How to combine GridSearchCV with Early Stopping?
1 votes

GridSearchCv with Early Stopping - I was curious about your question. As long as the algorithms has built in Early Stopper feature, you can use it in this manner. when it comes to other algorithms, ...

View answer
2 answers
2 votes
2k views
Semantic text similarity using BERT
0 votes

Other way is to use pip install sentence-transformers I am posting it from mobile, sorry if there are any indentation issues `from sentence_transformers import SentenceTransformer from sklearn....

View answer
2 answers
1 votes
108 views
Problem of continuous training - Supervised learning
0 votes

One Solution is "Human in the Loop" with Sentence Encoder. You can use hybrid approach using cosine similarity + Topic modelling + fuzzywuzzy + Bert. I totally understand the NLP world and ...

View answer
1 answers
0 votes
85 views
RAM crashed for XML to DataFrame conversion function
0 votes

import dask import dask.bag as db import dask.dataframe as dd from dask.dot import dot_graph from dask.diagnostics import ProgressBar dask.set_options(get=dask.multiprocessing.get) tags_xml = db....

View answer
1 answers
0 votes
22 views
How to run two different models in single frame?
0 votes

Use ray or Numba for Parallel Computation. Question is "are not you able to do transfer learning to detect both in one single model"?

View answer
1 answers
0 votes
89 views
How to write custom de-identification algorithm in Python?
0 votes

Unless you want to build your own, try the Faker Library for anonymity of PPI info. pip install Faker

View answer
1 answers
1 votes
91 views
Question about balancing training data for sentiment analysis (machine learning)
0 votes

I will try to answer with best of my capabilities and knowledge acquired . 1) Do not drop Good data just to balance it with lesser class 2) You can use imbalanced-learn and Smote based Python ...

View answer
2 answers
0 votes
59 views
Is it okay to use training data for validifying the trained model?
0 votes

No, its similar to leaking the question paper for your final exams. And always trust your Cross val. Check your classes and try extracting features and check if it performs better

View answer
1 answers
0 votes
282 views
What is the best way to encode features when clustering data?
0 votes

Try Hierarchical Clustering, Kmeans will not serve the purpose if you have categorical data.

View answer
4 answers
3 votes
200 views
Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression?
0 votes

You can use Graph based machine learning: Stellar

View answer
3 answers
6 votes
2k views
XGBoost Huge Dataset ~1TB
0 votes

You can use Vaex and Koalas with Sparkling water from H2O. All the three combined if you have computation infra. You can process than pretty much very distributed with ease

View answer
3 answers
0 votes
4k views
pandas apply function with multiple condition?
0 votes

You can use Nested List comprehension within the lambda function. Or Write a function and call the function on your series using Lambda

View answer
2 answers
0 votes
63 views
SVM, which range to use when normalizing
0 votes

Nope. There no definite answer. You cannot arrive at a conclusive decision point unless you experiment and analyze for yourself on the data with various kernel tricks. So feel Free to experiment as ...

View answer
2 answers
1 votes
212 views
Training model on a Balanced vs Imbalanced dataset?
0 votes

You can also try CalibratedClassifierCV if your data is imbalanced. The plots have been really useful to get the insights of the data.

View answer
7 answers
18 votes
4k views
Monitoring machine learning models in production
0 votes

If I understand your query correctly,you are looking for MLFLOW where you can track your experimentation and vizualize them using APIs MLFLOW

View answer
1 answers
0 votes
51 views
the library 'transformers' works also with older version of Tensorflow?
Accepted answer
0 votes

It does work but I recommend using TF2 for any new developments. In the end, if something does not work, you will have to port it back to TF2. Even though there are not significant difference in the ...

View answer
4 answers
2 votes
781 views
How do you predict a continuous variable when all your independent variables are categorical
0 votes

Start with Logistic Regression, NaiveBayes and SVMs. Linear regression does not work well on Categorical data even after encoding. As mentioned, you can encode your categories using Pd.dummy(One hot ...

View answer
1 answers
1 votes
212 views
Algorithmically extract seasonality in time series data
0 votes

You can try FBProphet for your timeseries analysis. You will be able to check the seasonality, stationarity on your data.

View answer