0
$\begingroup$

I am working on a quite big dataset that will be processed on the cluster, so this is why I am using PySpark for that purpose.

The presentable records of this dataset have a such structure:

+----------+------------+--------------------+--------------------+--------------------+
|         0|  07/29/2013|       Consumer Loan|        Vehicle loan|Managing the loan...|
|         1|  07/29/2013|Bank account or s...|    Checking account|Using a debit or ...|
|         2|  07/29/2013|Bank account or s...|    Checking account|Account opening, ...

After some preprocessing/data cleansing operations I would like to create and then obviously train a model that will classify issues (Issue) into some categories, that are still unknown. I have read some articles about TF-IDF, but not sure if this could be suitable for this case.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If you want to categorise your text using machine learning techniques, you have to get fixed length features from text to train any ML model. You can do That using bag of words, tf-idf, averaging word vectors. If you are using any deep learning based models, you can use LSTM with word vectors or CNN’s.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Ok, thank you very much for your answer, but what next? Can I use any ML agorithm once I have fixed lenght features? Could it be e.g. K-means? If not, which one do you recommend? Thank you in advance. $\endgroup$
    – user83701
    Commented Oct 17, 2019 at 18:56
  • $\begingroup$ If you don’t have any label, go for kmeans or dbscan. If you have those, go for supervised algorithms like SVM, Logistics Regression, Boosting Trees. $\endgroup$
    – Uday
    Commented Oct 18, 2019 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.