10
$\begingroup$

Is there a way of keeping a variable (large table / data frame) in memory and share it across multiple ipython notebooks?

I'd be looking for something, which is conceptually similar to MATLAB's persistent variables. There it is possible to call a custom function / library from multiple individual editors (notebooks), and have that external function cache some result (or large table).

Mostly I would like to avoid reloading a heavily used table (which is loaded through a custom library that is is called from the notebooks ), since reading it takes around 2-3 minute whenever I start a new analysis.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ This does not appear to be possible, and it could cause a lot of headaches if you're not careful. Is persisting the data to an efficient format like msgpack not an option? $\endgroup$
    – Emre
    Jan 17, 2017 at 23:55
  • $\begingroup$ @Emre Thank you. A tricky part with msgpack is that it does not solve the underlying problem of needing to read the table. Also it is a double-edged sword: While it saves around 40% of the time compared to the original format of the table, it also puts manual analysis one small step away from the original data (which is less clean) $\endgroup$
    – tsttst
    Jan 18, 2017 at 16:07
  • $\begingroup$ I think the best option is a cache like redis, which can be used in conjunction with msgpack. At least you can persist to memory instead of disk. $\endgroup$
    – Emre
    Jan 18, 2017 at 17:16
  • 1
    $\begingroup$ I would consider using Feather - it's very fast $\endgroup$
    – MaxU - stand with Ukraine
    Jan 19, 2017 at 11:37
  • 1
    $\begingroup$ Would Spark and it's caching be an option? You'd essentially be limited to using Spark in your notebooks though to do your initial reading/processing $\endgroup$
    – Dolan Antenucci
    Jan 23, 2017 at 14:23

1 Answer 1

Reset to default
4
$\begingroup$

If it's important for your use cases, you could try switching to Apache Zeppelin. As all Spark notebooks there share the same Spark context, same Python running environment. https://zeppelin.apache.org/

So what you're asking happens natively in Zeppelin. Or to be complete, it is an option to share the same Spark context / same Python envrionment between all Spark notebooks (they're called 'notes' in Zeppelin):

Spark Interpreter Sharing options in Zeppelin

So you can choose to share context Globally (default Zeppelin's behavior), Per Note (the only possible Jupyter's behavior), or Per User.

If you can't / don't want to switch to Zeppelin, look at other options of sharing common dataframes between your notebooks using:

ps. You can't import ipynb files to Zeppelin currently as of now (it has its own notebook format stored as a json file), until https://issues.apache.org/jira/browse/ZEPPELIN-1793 is implemented; although it's not that hard to convert them manually in most cases.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you. I will probably switch away from ipython / jupyter notebooks. Does zeppelin support the possibility to selectively only share the content of defined variables, but not of any identically named variable within different editors / notebooks / notes? (like MATLAB does) $\endgroup$
    – tsttst
    Jan 26, 2017 at 20:43
  • $\begingroup$ Unfortuntally - nope, it's controlled at a process level. So it's either all or nothing. If you choose Per Note, it'll be the same behavior as in Jupyter. If you choose Globally, they will share everything. We normally use Globally as it's less resource-intensive especially in multi-user environment. Haven't used Matlab for a while, but if you have to do sharing for only chosen variables - you could have a look at Apache Arrow or Feather, if it's Jupyter or Zeppelin. $\endgroup$
    – Tagar
    Jan 26, 2017 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.