Best python library for neural networks

Question

I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?

Answer 1

UPDATE: the landscape has changed quite a bit since I answered this question in July '14, and some new players have entered the space. In particular, I would recommend checking out:

• TensorFlow
• Blocks
• Lasagne
• Keras
• Deepy
• Nolearn
• NeuPy

They each have their strengths and weaknesses, so give them all a go and see which best suits your use case. Although I would have recommended using PyLearn2 a year ago, the community is no longer active so I would recommend looking elsewhere. My original response to the answer is included below but is largely irrelevant at this point.

---

PyLearn2 is generally considered the library of choice for neural networks and deep learning in python. It's designed for easy scientific experimentation rather than ease of use, so the learning curve is rather steep, but if you take your time and follow the tutorials I think you'll be happy with the functionality it provides. Everything from standard Multilayer Perceptrons to Restricted Boltzmann Machines to Convolutional Nets to Autoencoders is provided. There's great GPU support and everything is built on top of Theano, so performance is typically quite good. The source for PyLearn2 is available on github.

Be aware that PyLearn2 has the opposite problem of PyBrain at the moment -- rather than being abandoned, PyLearn2 is under active development and is subject to frequent changes.

Answer 2

Tensor Flow (docs) by Google is another nice framework which has automatic differentiation. I've written down some quick thoughts about Google Tensor Flow on my blog, together with the MNIST example which they have in their tutorial.

See also: My Tensorflow XOR tutorial

Lasagne (docs) is very nice, as it uses theano (→ you can use the GPU) and makes it simpler to use. The author of lasagne won the Kaggle Galaxy challenge, as far as I know. It is nice with nolearn. Here is an MNIST example network:

#!/usr/bin/env python

import lasagne
from lasagne import layers
from lasagne.updates import nesterov_momentum
from nolearn.lasagne import NeuralNet

import sys
import os
import gzip
import pickle
import numpy


PY2 = sys.version_info[0] == 2

if PY2:
    from urllib import urlretrieve

    def pickle_load(f, encoding):
        return pickle.load(f)
else:
    from urllib.request import urlretrieve

    def pickle_load(f, encoding):
        return pickle.load(f, encoding=encoding)

DATA_URL = 'http://deeplearning.net/data/mnist/mnist.pkl.gz'
DATA_FILENAME = 'mnist.pkl.gz'


def _load_data(url=DATA_URL, filename=DATA_FILENAME):
    """Load data from `url` and store the result in `filename`."""
    if not os.path.exists(filename):
        print("Downloading MNIST dataset")
        urlretrieve(url, filename)

    with gzip.open(filename, 'rb') as f:
        return pickle_load(f, encoding='latin-1')


def load_data():
    """Get data with labels, split into training, validation and test set."""
    data = _load_data()
    X_train, y_train = data[0]
    X_valid, y_valid = data[1]
    X_test, y_test = data[2]
    y_train = numpy.asarray(y_train, dtype=numpy.int32)
    y_valid = numpy.asarray(y_valid, dtype=numpy.int32)
    y_test = numpy.asarray(y_test, dtype=numpy.int32)

    return dict(
        X_train=X_train,
        y_train=y_train,
        X_valid=X_valid,
        y_valid=y_valid,
        X_test=X_test,
        y_test=y_test,
        num_examples_train=X_train.shape[0],
        num_examples_valid=X_valid.shape[0],
        num_examples_test=X_test.shape[0],
        input_dim=X_train.shape[1],
        output_dim=10,
    )


def nn_example(data):
    net1 = NeuralNet(
        layers=[('input', layers.InputLayer),
                ('hidden', layers.DenseLayer),
                ('output', layers.DenseLayer),
                ],
        # layer parameters:
        input_shape=(None, 28*28),
        hidden_num_units=100,  # number of units in 'hidden' layer
        output_nonlinearity=lasagne.nonlinearities.softmax,
        output_num_units=10,  # 10 target values for the digits 0, 1, 2, ..., 9

        # optimization method:
        update=nesterov_momentum,
        update_learning_rate=0.01,
        update_momentum=0.9,

        max_epochs=10,
        verbose=1,
        )

    # Train the network
    net1.fit(data['X_train'], data['y_train'])

    # Try the network on new data
    print("Feature vector (100-110): %s" % data['X_test'][0][100:110])
    print("Label: %s" % str(data['y_test'][0]))
    print("Predicted: %s" % str(net1.predict([data['X_test'][0]])))


def main():
    data = load_data()
    print("Got %i testing datasets." % len(data['X_train']))
    nn_example(data)

if __name__ == '__main__':
    main()

Caffe is a C++ library, but has Python bindings. You can do most stuff by configuration files (prototxt). It has a lot of options and can also make use of the GPU.

Answer 3

Pylearn relies on Theano and as mentioned in the other answer to use the library is quite complicated, until you get the hold of it.

In the meantime I would suggest using Theanets. It also built on top of Theano, but is much more easier to work with. It might be true, that it doesn't have all the features of Pylearn, but for the basic work it's sufficient.

Also it's open source, so you can add custom networks on the fly, if you dare. :)

EDIT: Dec 2015. Recently I have started using Keras. It is a bit lower level than Theanets, but much more powerful. For basic tests the Theanets is appropriate. But if you want to do some research in field of ANN Keras is much more flexible. Plus the Keras can use Tensorflow as a backend.

Answer 4

TensorFlow (by Google, released on 2015-11-09) looks promising.

• open source (Apache 2.0 License) (GitHub)
• Python (backend in C++)
• CPU/GPU
• Auto-Differentiation
• Portable (even works on mobile devices)

FYI:

• Evaluation of Deep Learning Toolkits
• Easy benchmarking of all public open-source implementations of convnets.
• Soheil Bahrampour, Naveen Ramakrishnan, Lukas Schott, Mohak Shah. Comparative Study of Deep Learning Software Frameworks. arXiv:1511.06435.
• Shaohuai Shi, Qiang Wang, Pengfei Xu, Xiaowen Chu. Benchmarking State-of-the-Art Deep Learning Software Tools. arXiv:1608.07249
• Was a new deep learning toolkit released once every 22 days in 2015?

Answer 5

Pylearn2 seems to be the library of choice, however I find their YAML configuration files off-putting.

Python itself was designed to be an easy language for prototyping, why would you not use it to define the network properties themselves? We have great editors with autocompletion that would make your life much easier and Python is not like C++ where you have to wait for long builds to finish before you can run your code.

YAML files on the other hand you have to edit using a standard text editor with no assistance whatsoever and this makes the learning curve even steeper.

I may be missing the big picture but I still don't understand what were they thinking, I don't think prototyping in code would be much slower. For that reason I'm considering Theanets or using Theano directly.

Answer 6

I like Blocks, which is also built on top of Theano. Way more approachable than PyLearn2, and more feature rich than Lasagne. Neatly written, too.

Updated Jan 2016:

At the time of writing, Keras has by far the most momentum. It is highly modular and can run on both Theano and Tensorflow, giving it great opportunities.

Answer 7

MXNet:

• written in C++ but has an API in Python (and a few other programming languages such as R, Julia, and Go)
• Scales up to multi GPUs and distributed setting with auto parallelism.
• Automatic differentiation
• Decent performances:

Answer 8

From what I heard, Pylearn2 might be currently the library of choice for most people. This reminds me of a recent blog post a few month ago that lists all the different machine learning libraries with a short explanation

https://www.cbinsights.com/blog/python-tools-machine-learning

The section you might be interested in here would be "Deep Learning". About Pylearn2, he writes

PyLearn2

There is another library built on top of Theano, called PyLearn2 which brings modularity and configurability to Theano where you could create your neural network through different configuration files so that it would be easier to experiment different parameters. Arguably, it provides more modularity by separating the parameters and properties of neural network to the configuration file.

Answer 9

I wrote up this post detailing some of my personal favorites:

The Best Machine Learning Libraries in Python

Since over 30 different libraries are mentioned, I won't post them all here, but these are among the most popular:

• Tensorflow
• scikit-learn
• Theano
• Pylearn2
• Pyevolve
• NuPIC
• Pattern
• Caffe

(Sorry, can't link to the Github repos since my rep is still < 10...)

Edit: Added links to Github repos.

Answer 10

neon:

• Python based
• Open source (Apache 2.0 License)
• Developed by Nervana Systems (if you don't know them: Deep learning startup Nervana raises $20.5M; Nervana Systems raises 3.3M USD to build hardware designed for deep learning )
• Supports CPU and GPU
• Basic automatic differentiation support
• Framework for visualization

Answer 11

Just to add some more resources. Recently there was a paper studying the differences between several packages of neural networks and deep neural networks.

Here you can find the information. It looks that Torch and TensorFlow are the winners.

Note: not all of them are in python. However, I posted it to open the discussion.

Answer 12

DyNet: The Dynamic Neural Network Toolkit. From {1}:

We describe DyNet, a toolkit for implementing neural network models based on dynamic declaration of network structure. In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives. In DyNet's dynamic declaration strategy, computation graph construction is mostly transparent, being implicitly constructed by executing procedural code that computes the network outputs, and the user is free to use different network structures for each input. Dynamic declaration thus facilitates the implementation of more complicated network architectures, and DyNet is specifically designed to allow users to implement their models in a way that is idiomatic in their preferred programming language (C++ or Python). One challenge with dynamic declaration is that because the symbolic computation graph is defined anew for every training example, its construction must have low overhead. To achieve this, DyNet has an optimized C++ backend and lightweight graph representation. Experiments show that DyNet's speeds are faster than or comparable with static declaration toolkits, and significantly faster than Chainer, another dynamic declaration toolkit. DyNet is released open-source under the Apache 2.0 license and available at this http URL

It was previously known as cnn (whose python binding was named pycnn).

---

References:

• {1} Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, Pengcheng Yin. DyNet: The Dynamic Neural Network Toolkit. https://arxiv.org/abs/1701.03980

Answer 13

PyTorch

It is gaining a lot of support due to its ease of use and similarity with basic Python.

It runs "line-by-line" (via dynamic graphs), just like normal Python and can be easily debugged - even using standard print statements. It also integrates very well with NumPy and other well-known Python linbraries, like Scikit Learn.

Because it makes modelling easy, it is great for prototyping and exploring new ideas in general.

It supports multiple GPUs and does so in a really easy way.

Check out more features here.

---

Although many of the benefits above make PyTorch much nicer to use than other commonly used libraries, it is worth mentioning that the upcoming major release of Tensorflow will, by default, also use dynamic graph creation (a.k.a. eager mode). This will make it comparable to PyTorch in usage.

Answer 14

Microsoft Cognition Toolkit (previously known as CNTK) has a Python API. Amongst other things, it is supposed to be good for multi-GPU:

Examples and tutorials can be found on https://github.com/Microsoft/CNTK/tree/master/bindings/python

Answer 15

I recommend you to use tensorflow which is under strong development and supports deep learning. You can use the high-level neural networks API Keras that runs on top of tensorflow and is very simply to use, just try a tutorial and you are going to love it.

Answer 16

NeuPy is a Python library for Artificial Neural Networks. NeuPy supports many different types of Neural Networks from a simple perceptron to deep learning models.

Answer 17

If you are interested in performance comparisons, Soumith Chintala maintains a set of convnet benchmarks that cover several of the already mentioned python frameworks (TensorFlow, chainer, neon, Theano):

• https://github.com/soumith/convnet-benchmarks