6
$\begingroup$

I have built some implementations using NetworkX(graph Python module) native algorithms in which I output some attributes which I use them for classification purposes.

I want to scale it to a distributed environment. I have seen many approaches like neo4j, Graphx, GraphLab. However, I am quite new to this, thus I want to ask, which of them would be easy to locally apply graph algorithms (ex. node centrality measures), preferably using Python. To be more specific, which available option is closer related to NetworkX (easy installation, premade functions/algorithms, ML wise)?

$\endgroup$
  • $\begingroup$ I suggest looking into Spark's GraphFrames. It's Scala (not a bad thing!) but uses the familiar dataframe paradigm. $\endgroup$ – Emre Aug 16 '16 at 18:33
  • $\begingroup$ @Emre Yes indeed, I had a quick look. I can see that supports some Python, however, I suppose that the functionalities are less than using Scala. Btw, if you know, just to make sure, the licensing for GraphFrames is totally free? even for commercial purposes? $\endgroup$ – Grzegorz Aug 16 '16 at 18:46
  • $\begingroup$ It's published under the Apache License (FAQ) $\endgroup$ – Emre Aug 16 '16 at 19:37
  • $\begingroup$ @Emre Thank you very much and especially for you valuable answer, GraphFrames seems a legit solution. $\endgroup$ – Grzegorz Aug 16 '16 at 22:26
3
$\begingroup$

Good , old and unsolved question! Distributed processing of large graphs as far as I know (speaking as a graph guy) has 2 different approaches, with the knowledge of Big Data frameworks or without it.

SNAP library from Jure Leskovec group at Stanford which is originally in C++ but also has a Python API (please check if you need to use C++ API or Python does the job you want to do). Using snap you can do many things on massive networks without any special knowledge of Big Data technologies. So I would say the easiest one.

Using Apache Graphx is wonderful only if you have experience in Scala because there is no Python thing for that. It comes with a large stack of built in algorithms including centrality measures. So the second easiest in case you know Scala.

Long time ago when I looked at GraphLab it was commercial. Now I see it goes open source so maybe you know better than me but from my out-dated knowledge I remember that it does not support a wide range of algorithms and if you need an algorithm which is not there it might get complicated to implement. On the other hand it uses Python which is cool. After all please check it again as my knowledge is for 3 years ago.

If you are familiar with Big Data frameworks and working with them, Giraph and Gradoop are 2 great options. Both do fantastic jobs but you need to know some Big Data architecture e.g. working with a hadoop platform.

PS

1) I have used simple NetworkX and multiprocessing to distributedly process DBLP network with 400,000 nodes and it worked well, so you need to know HOW BIG your graph is.

2) After all, I think SNAP library is a handy thing.

$\endgroup$
  • 1
    $\begingroup$ Thank you very much, I did not know about the SNAP library. This seems a handy option. BTW would you now out of curiosity if SNAP scales with Spark? $\endgroup$ – Grzegorz Aug 16 '16 at 18:09
  • $\begingroup$ Also, about the graphLab me too, do not know if it is totally free, I read somewhere that you can get granted 1 year full, for academic purposes only. Btw I had found this interesting podcast about Dato if would be of interest O'reilly podcast $\endgroup$ – Grzegorz Aug 16 '16 at 18:28
  • 1
    $\begingroup$ I wouldn't invest in graphlab now that Apple bought it. $\endgroup$ – Emre Aug 16 '16 at 18:32
  • 1
    $\begingroup$ I actually used SNAP only for simple experiments on medium-size graphs so I do not know if it scales with Spark but they have a QA system to ask your question. I suggest to contact THEM directly. @PhilipC. $\endgroup$ – Kasra Manshaei Aug 17 '16 at 16:25
  • 1
    $\begingroup$ While GraphX isn't directly available in Python APIs, Apache Spark does now make that functionality available through the newer GraphFrames package. See graphframes.github.io and databricks.com/blog/2016/03/03/introducing-graphframes.html $\endgroup$ – Drew Dara-Abrams May 25 '17 at 4:56
1
$\begingroup$

In this present moment, Apache has develop a powerfull API called PySpark. And you can setup Graphframes directly from pyspark command line. Launch from you shell terminal:

pyspark --packages graphframes:graphframes:0.6.0-spark2.3-s_2.11

and you can develop your code entirely in python using graphframes API. Try the following example code

# Create a Vertex DataFrame with unique ID column "id"
v = sqlContext.createDataFrame([
    ("a", "Alice", 34),
    ("b", "Bob", 36),
    ("c", "Charlie", 30),
    ], ["id", "name", "age"])

# Create an Edge DataFrame with "src" and "dst" columns
e = sqlContext.createDataFrame([
    ("a", "b", "friend"),
    ("b", "c", "follow"),
    ("c", "b", "follow"),
    ], ["src", "dst", "relationship"])

# Create a GraphFrame
from graphframes import *
g = GraphFrame(v, e)

# Query: Get in-degree of each vertex.
g.inDegrees.show()

# Query: Count the number of "follow" connections in the graph.
g.edges.filter("relationship = 'follow'").count()

# Run PageRank algorithm, and show results.
results = g.pageRank(resetProbability=0.01, maxIter=20)
results.vertices.select("id", "pagerank").show()

Above we could calculate pageRank from graph g. There are several algorithms already implemented on PySpark with Graphframes. I hope I've helped.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.