0
$\begingroup$

I'm looking for a method of allocating documents (30K and growing) to a set of some 200 categories.

The categories will be user defined and will grow over time.

As my data is unlabelled my thought process is to try an build a system that aids in the manual classification by providing a rough first pass classifier. User can then quickly go through the classified documents and accept/reject the classifications.

Once we have a sufficient large set of classified documents then I'm hoping to use some AI system to automatically classify the remaining documents and new ones as they are published.

I've had a play with LDA but as I understand it, LDA essentially chooses the topics which don't necessarily map to the categories that I want to define.

I've have also built a rule engine that allows me to manually define common key words to map each document to a category but this is still a fairly manual process as I need to define keywords for each of the categories.

$\endgroup$

1 Answer 1

0
$\begingroup$

Multi-output classification for text input could be done thanks to Random Forest Classifier.

First, you need to vectorize your text to make your data multidimensional and allow the Classifier to segregate the data easily.

from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split

# Extract features with a word frequency (for example)
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

Then, you can apply the Random Forest Classifier from sklearn that would automatically classify the vectorized data easily.

#Train/test split
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.2)

# Training
clf = RandomForestClassifier(n_estimators=200, max_depth=15)
clf.fit(X_train, y_train)

# Make predictions on the test set
y_pred = clf.predict(X_test)

# Model accuracy
accuracy = clf.score(X_test, y_test)

X_test would be the text input and y_pred the multi-output classification result.

Alternatively, you can use TfidfVectorizer instead of CountVectorizer.

$\endgroup$
4
  • $\begingroup$ Does it answer your question? If not, please let me know to provide additional information or other solutions. $\endgroup$ Commented Dec 27, 2022 at 8:15
  • $\begingroup$ I don't understand how this allocates documents to my predefined categories? $\endgroup$ Commented Dec 29, 2022 at 8:21
  • $\begingroup$ Could you add a simple example with some input/outputs so that I could answer more accurately? $\endgroup$ Commented Dec 29, 2022 at 8:24
  • $\begingroup$ The plan is to categorize all of the readmes on pub.dev. eg. pub.dev/packages/chopper. An example category would be communications.http.client. I will pre-define the set of categories and want each readme allocated to one. $\endgroup$ Commented Dec 30, 2022 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.