9
$\begingroup$

Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks?

I'd be looking for something, which is conceptually similar to MATLAB's persistent variables. There it is possible to call a custom function / library from multiple individual editors (notebooks), and have that external function cache some result (or large table).

Mostly I would like to avoid reloading a heavily used table (which is loaded through a custom library that is is called from the notebooks ), since reading it takes around 2-3 minute whenever I start a new analysis.

$\endgroup$
  • 1
    $\begingroup$ This does not appear to be possible, and it could cause a lot of headaches if you're not careful. Is persisting the data to an efficient format like msgpack not an option? $\endgroup$ – Emre Jan 17 '17 at 23:55
  • $\begingroup$ @Emre Thank you. A tricky part with msgpack is that it does not solve the underlying problem of needing to read the table. Also it is a double-edged sword: While it saves around 40% of the time compared to the original format of the table, it also puts manual analysis one small step away from the original data (which is less clean) $\endgroup$ – tsttst Jan 18 '17 at 16:07
  • $\begingroup$ I think the best option is a cache like redis, which can be used in conjunction with msgpack. At least you can persist to memory instead of disk. $\endgroup$ – Emre Jan 18 '17 at 17:16
  • 1
    $\begingroup$ I would consider using Feather - it's very fast $\endgroup$ – MaxU Jan 19 '17 at 11:37
  • 1
    $\begingroup$ Would Spark and it's caching be an option? You'd essentially be limited to using Spark in your notebooks though to do your initial reading/processing $\endgroup$ – Dolan Antenucci Jan 23 '17 at 14:23
3
$\begingroup$

If it's important for your use cases, you could try switching to Apache Zeppelin. As all Spark notebooks there share the same Spark context, same Python running environment. https://zeppelin.apache.org/

So what you're asking happens natively in Zeppelin. Or to be complete, it is an option to share the same Spark context / same Python envrionment between all Spark notebooks (they're called 'notes' in Zeppelin):

Spark Interpreter Sharing options in Zeppelin

So you can choose to share context Globally (default Zeppelin's behavior), Per Note (the only possible Jupyter's behavior), or Per User.

If you can't / don't want to switch to Zeppelin, look at other options of sharing common dataframes between your notebooks using:

ps. You can't import ipynb files to Zeppelin currently as of now (it has its own notebook format stored as a json file), until https://issues.apache.org/jira/browse/ZEPPELIN-1793 is implemented; although it's not that hard to convert them manually in most cases.

$\endgroup$
  • 1
    $\begingroup$ Thank you. I will probably switch away from ipython / jupyter notebooks. Does zeppelin support the possibility to selectively only share the content of defined variables, but not of any identically named variable within different editors / notebooks / notes? (like MATLAB does) $\endgroup$ – tsttst Jan 26 '17 at 20:43
  • $\begingroup$ Unfortuntally - nope, it's controlled at a process level. So it's either all or nothing. If you choose Per Note, it'll be the same behavior as in Jupyter. If you choose Globally, they will share everything. We normally use Globally as it's less resource-intensive especially in multi-user environment. Haven't used Matlab for a while, but if you have to do sharing for only chosen variables - you could have a look at Apache Arrow or Feather, if it's Jupyter or Zeppelin. $\endgroup$ – Tagar Jan 26 '17 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.