Syenix
  • Member for 2 years, 8 months
  • Last seen more than a month ago
  • Bangalore, Karnataka, India
Evaluating information extraction from structured documents
3 votes

I have worked with Structured text using OCR. OCR is prone to errors even while reading the content and a slight change in string arrangement would lead to false positives. I used Cosine Similarity ...

View answer
How to deal with spelling errors NLP
2 votes

You can try Flashtext to easily create alternatives as you find and FuzzyWuzzy to get the similarity between word tokens by n grams

View answer
Hyperparameter optimization, ensembling instead of selecting with CV criteria
2 votes

I think you are looking for hyperopts, Optuna and Gpopy for Hyperparameters search without burning much of CPUs.

View answer
Which ML algorithm should I use for following use case for classification and Why?
1 votes

You can check https://autogluon.mxnet.io/ from autogluon import TabularPrediction as task predictor = task.fit(train_data=task.Dataset(file_path=TRAIN_DATA.csv), label=COLUMN_NAME) predictions = ...

View answer
Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?
1 votes

I think you have misunderstood the koalas library. You can say its Pandas on Distributed System. You can use Koalas similar to pandas. There are few drawbacks with respect to APIs which is documented ...

View answer
How to extract entities from text using existing ontologies?
1 votes

I think you are looking for Spacy, Polyglot and AllenNLP to find your NERs.

View answer
Technique to determine variation in metric due to varying parameters
Accepted answer
1 votes

If the variation in values of feature then should not you remove the column? Your model will not learn from it if the variance is low. you can Q-Q plots when you vary the features to check how close ...

View answer
Support Vector Machine analysis - Python, how do I determing whether to use linear, square, or other types of SVM models?
1 votes

By Default, SVM in Sklearn uses RBF Kernel. You have to try out all the 3 kernels, with different Gamma and C. SVM treats outliers better and add a penalty on every outlier it detects. You should ...

View answer
How to define the adequate cash prize sizing for hosting a Kaggle or similar compeition?
1 votes

This is a generic question that may need more than just cash prize. 1) Organizational Reputation 2) Dataset Size 3) Defining Problem statements with Domain knowledge 4) Time period 5) Computation ...

View answer
Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0?
Accepted answer
1 votes

Looking at your use case, Dask, H2O, Modin, Koalas and Vaex would better for scaling your data preprocessing apart from Pyspark. They have API's similar to pandas thus porting your existing code would ...

View answer
Training machine learning models from log files
1 votes

Anytime that you are looking at logs, basically you want to do anamoly detection. There are various methods I have tried and want to use few more: 1) Topic Modelling 2) Supervised Learning and can ...

View answer
How to combine GridSearchCV with Early Stopping?
1 votes

GridSearchCv with Early Stopping - I was curious about your question. As long as the algorithms has built in Early Stopper feature, you can use it in this manner. when it comes to other algorithms, ...

View answer
Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?
1 votes

Firstly any imbalance class text classification will have biased towards majority population and will result in overfit/underfit. You can use Smote, imbalancelearn library and imbalance from Scikit ...

View answer
Semantic text similarity using BERT
0 votes

Other way is to use pip install sentence-transformers I am posting it from mobile, sorry if there are any indentation issues `from sentence_transformers import SentenceTransformer from sklearn....

View answer
Problem of continuous training - Supervised learning
0 votes

One Solution is "Human in the Loop" with Sentence Encoder. You can use hybrid approach using cosine similarity + Topic modelling + fuzzywuzzy + Bert. I totally understand the NLP world and ...

View answer
RAM crashed for XML to DataFrame conversion function
0 votes

import dask import dask.bag as db import dask.dataframe as dd from dask.dot import dot_graph from dask.diagnostics import ProgressBar dask.set_options(get=dask.multiprocessing.get) tags_xml = db....

View answer
How to run two different models in single frame?
0 votes

Use ray or Numba for Parallel Computation. Question is "are not you able to do transfer learning to detect both in one single model"?

View answer
How to write custom de-identification algorithm in Python?
0 votes

Unless you want to build your own, try the Faker Library for anonymity of PPI info. pip install Faker

View answer
Question about balancing training data for sentiment analysis (machine learning)
0 votes

I will try to answer with best of my capabilities and knowledge acquired . 1) Do not drop Good data just to balance it with lesser class 2) You can use imbalanced-learn and Smote based Python ...

View answer
Is it okay to use training data for validifying the trained model?
0 votes

No, its similar to leaking the question paper for your final exams. And always trust your Cross val. Check your classes and try extracting features and check if it performs better

View answer
What is the best way to encode features when clustering data?
0 votes

Try Hierarchical Clustering, Kmeans will not serve the purpose if you have categorical data.

View answer
Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression?
0 votes

You can use Graph based machine learning: Stellar

View answer
XGBoost Huge Dataset ~1TB
0 votes

You can use Vaex and Koalas with Sparkling water from H2O. All the three combined if you have computation infra. You can process than pretty much very distributed with ease

View answer
pandas apply function with multiple condition?
0 votes

You can use Nested List comprehension within the lambda function. Or Write a function and call the function on your series using Lambda

View answer
SVM, which range to use when normalizing
0 votes

Nope. There no definite answer. You cannot arrive at a conclusive decision point unless you experiment and analyze for yourself on the data with various kernel tricks. So feel Free to experiment as ...

View answer
Training model on a Balanced vs Imbalanced dataset?
0 votes

You can also try CalibratedClassifierCV if your data is imbalanced. The plots have been really useful to get the insights of the data.

View answer
Monitoring machine learning models in production
0 votes

If I understand your query correctly,you are looking for MLFLOW where you can track your experimentation and vizualize them using APIs MLFLOW

View answer
the library 'transformers' works also with older version of Tensorflow?
Accepted answer
0 votes

It does work but I recommend using TF2 for any new developments. In the end, if something does not work, you will have to port it back to TF2. Even though there are not significant difference in the ...

View answer
How do you predict a continuous variable when all your independent variables are categorical
0 votes

Start with Logistic Regression, NaiveBayes and SVMs. Linear regression does not work well on Categorical data even after encoding. As mentioned, you can encode your categories using Pd.dummy(One hot ...

View answer
Algorithmically extract seasonality in time series data
0 votes

You can try FBProphet for your timeseries analysis. You will be able to check the seasonality, stationarity on your data.

View answer