Productivity tips for Jupyter (Python)Michał KrassowskiBlockedUnblockFollowFollowingMar 17I’ve been very busy working on my MRes project in recent weeks, having very little sleep.
This made me seek ways to improve my workflow in the most important tool of my work: Jupyter Notebook/Jupyter Lab.
I collected all the hacks & tips in this piece, hoping that other researchers may find those useful:Play a sound once the computations have finished (or failed)Integrate the notifications with your OS (ready for GNOME shell)Jump to definition of a variable, function or classEnable auto-completion for rpy2 (great for ggplot2)Summarize dictionaries and other structures in a nice tableSelectively import from other notebooksScroll to the recently executed cell on error or when opening the notebookInteractive (following) tail for long outputsNote: To make it easy-to-use, I collected the snippets presented below into a Python3 package (jupyter-helpers).
You can get it with:pip3 install jupyter_helpersIt will work out of the box, but if you wish the best experience, these dependencies are highly recommended:pip3 install ipywidgetsjupyter labextension install @jupyter-widgets/jupyterlab-manager1.
Play a sound once the computations have finishedYou can configure your Jupyter to play a sound if the execution of the cell (or strand of cells) took more than a few seconds.
I have previously described two methods for that, a Python-based one, and a JavaScript one.
The code for the python one roughly goes like:However, the helpers package got an upgraded version which takes care of hiding the Audio player and many other things.
Use it like this:from jupyter_helpers.
notifications import NotificationsNotifications(Selectively import from other notebooks success_audio='path/to/beep-07.
wav', time_threshold=2)In the above case I use beep-07.
wav from soundjay.
com.
Play a honk sound on exceptionSimilarly, you can add a hook to play a different sound when an exception is raised.
Here is a very simple mechanism proposed by Kevin at SO:And of course a more advanced one is part of the jupyter_helpers package:from jupyter_helpers.
notifications import NotificationsNotifications(failure_audio='path/to/beep-05.
wav')2.
Integrate notifications with your OSWhen working from library I needed an alternative for beeps and honks.
Notify-send turned our to be a perfect tool for me as a primarily GNOME user (scroll down for instructions for other desktop environments).
To set things up use:from jupyter_helpers.
notifications import NotificationsNotifications( success_audio='path/to/beep-07.
wav', time_threshold=2, failure_audio='path/to/beep-05.
wav', integration='GNOME')This will work out of the box for GNOME users, though installing a drop-in replacement called notify-send.
sh will make the notifications go away once they are no longer needed.
This can be done with attached setup.
sh script.
Figure 1: Fully integrated notificationsThe OS integration is ready to be hooked for any other desktop environment, though it will require some scripting:from jupyter_helpers.
desktop_integration import DesktopIntegrationclass WindowsIntegration(DesktopIntegration): def notify(self, title, text, notify_id=None, **kwargs): pass # add your code here def notify_close(self, notify_id): pass # add your code hereNotifications( success_audio='path/to/beep-07.
wav', time_threshold=2, failure_audio='path/to/beep-05.
wav', integration=WindowsIntegration)Please consider sending a PR if you wish to integrate it with your OS.
3.
Jump to definition of a variable/function/classUse Alt + click to jump to a definition using your mouse, or Ctrl + Alt + B keyboard-only alternative with jupyterlab-go-to-definition extension:Jump-to-definition supports Python and R.
PRs to support other languages are welcomeFinally, use Alt+ o to jump back to your previous location:The ability to jump back is quite helpful for notebooks with longer outputsTo install the extension use:jupyter labextension install @krassowski/jupyterlab_go_to_definition4.
Enable auto-completion for rpy2 (ggplot2!)If your work is more about publications rather than interactive dashboards, there is a good chance you are familiar with ggplot2.
While there are some great projects like plotnine which attempt to port it to Python, I still find working with ggplot (especially the extensions) more feature-complete when using rpy2 R-Python interface.
However, auto-completion did not include the R objects (nor ggplot functions if loaded) in %%R cells so far.
I prepared a simple workaround:It may be improved in the future, as discussed in this GitHub issue.
Auto-completion now also includes R objects and ggplot functionsAgain, one simple import from jupyter_helpers will solve the issue:from jupyter_helpers import rpy2_autocompletion5.
Summarize dictionaries in a nice table viewThis is not a novel idea, though I hope that sharing my more than less advanced class may help others.
This is based on Python3 SimpleNamespace, but extends it with a pandas- and numpy- aware HTML representation for Jupyter:from jupyter_helpers.
namespace import NeatNamespaceNeatNamespace(your_dict)Long collections will be trimmed, so no need to worry about the space or running out of memory when browsers struggle to render accidentally printed dictionary.
Horizontal and vertical orientations are available for better space utilization.
Namespaces with HTML: when nested data needs to be looked at before converting to DataFrame6.
Selectively import from other notebooksFor some time I was trying to follow data/methods/results separation, having three Jupyter notebooks for each larger analysis: data.
ipynb, methods.
ipynb and results.
ipynb.
To save time on useless re-computation of some stuff I wanted to have selective import from data and methods notebooks for use in the results notebook.
It is now possible (building upon nbimporter) using one import and a magic:This was described in this SO thread, and I still hope to see some suggestions.
7.
Scroll to the recently executed cellYou may have noticed that previously shown Notifications class made the notebook scroll down on exception to the offending cell (Figure 1).
This can be disabled by passing scroll_to_exceptions=False.
If you conversely, want more auto-scrolling, you could use the underlying helper function to mark the cell that you ended working with at night to quickly open your notebook back on it in the morning:from jupyter_helpers.
utilities import scroll_to_current_cellscroll_to_current_cell(preserve=True)8.
Interactive tail for long outputsLastly, when running third-party applications (but not being at the point of building a fully-fledged pipeline), one may want to see only the tail of the currently running process.
In bash it is easily achieved using tail -f.
If you can relate to the need of watching the output and problem of it slowing down your computer when being flushed out to your notebook or just generating annoyingly long outputs, FollowingTail helper may be for you:Apply tail -f equivalent to keep the outputs at reasonable length!There is more to come!I only had time for so much now, but I will be sharing more of already written and new helpers and snippets.
This is a way of giving back to the amazing community of scientific computing (from both Python and R side) which enabled me to complete my thesis.
I have another project to come, which will likely require to overcome other challenges.
Once more is ready I will be sharing the news on Twitter (@krassowski_m).
Thank you for reading — hope it was worth writing for at least one researcher.
.. More details