Where does tools like spacy, sklearn, prodigy, nltk fit in the below AI project architecture and what are some common alternatives to these: enter image description here


1 Answer 1


I am not sure you can neatly fit the above tools and libraries into your schematic. What even is the meaning of "feature" here, is it an output (like "text classification" is an output of the algorithm)? In which case I challenge the usefulness of this schematic.

Nevertheless nltk, sklearn, etc. are libraries that contain multiple and diverse tools to help you do everything along the NLP AI modeling workflow. From feature extraction to modelling and validation.

As an example all three libraries of spacy, sklearn and nltk have the ability to construct models. So they certainly fit into "ML algorithm" but they also help in feature extraction so they fit in that bucket.

Beyond that they offer you general data wrangling capabilities as well as validation measures which do not appear in your schematic.

The main difference

NLTK and spacy are mainly focused on NLP and text-based data whereas sklearn is very much multipurpose.

The Alternatives

The alternatives really depend on your use cases, NLTK and spacy do very similiar stuff so are already alternatives to each other for NLP. Sklearn is almost standard when it comes to ML in python but if you want to build deep-learning you will probably work more with Keras, pytorch, etc.

  • $\begingroup$ It is right that spacy forces you to choose a specific algorithm which is the most effective one, but nltk allows you to choose any algorithm that you wish to use? $\endgroup$
    – variable
    Commented Feb 7, 2020 at 12:47
  • $\begingroup$ @variable That is not a 100% accurate way to describe this. Spacy contains pretrained models so called encoders which tokenize the text in a specific way like BERT for example. You can however choose between those, the same is true for NLTK. However I suspect you can also build your own tokenizers and encoding models with NLTK and "unlink" the actual model development and the encoding/embedding. $\endgroup$
    – Fnguyen
    Commented Feb 7, 2020 at 12:58
  • $\begingroup$ OK, and what about for example - training a model using SVM / SGD. I know that sklearn allows us to do this. But can this also be done using nltk and spacy? $\endgroup$
    – variable
    Commented Feb 8, 2020 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.