It is a common problem that people want to import code from IPython Notebooks. This is made difficult by the fact that Notebooks are not plain Python files, and thus cannot be imported by the regular Python machinery.
Fortunately, Python provides some fairly sophisticated hooks into the import machinery, so we can actually make IPython notebooks importable without much difficulty, and only using public APIs.
import io, os, sys, types
from IPython.nbformat import currentfrom IPython.core.interactiveshell import InteractiveShell
Import hooks typically take the form of two objects:
a Module Loader, which takes a module name (e.g. 'IPython.display'), and returns a Module
a Module Finder, which figures out whether a module might exist, and tells Python what Loader to use
deffind_notebook(fullname,path=None):"""find a notebook, given its fully qualified name and an optional path This turns "foo.bar" into "foo/bar.ipynb" and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar does not exist. """ name = fullname.rsplit('.', 1)[-1]ifnot path: path = ['']for d in path: nb_path = os.path.join(d, name +".ipynb")if os.path.isfile(nb_path):return nb_path# let import Notebook_Name find "Notebook Name.ipynb" nb_path = nb_path.replace("_", " ")if os.path.isfile(nb_path):return nb_path
Notebook Loader
Here we have our Notebook Loader. It's actually quite simple - once we figure out the filename of the module, all it does is:
load the notebook document into memory
create an empty Module
execute every cell in the Module namespace
Since IPython cells can have extended syntax, the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them. If all of your notebook cells are pure-Python, this step is unnecessary.
classNotebookLoader(object):"""Module Loader for IPython Notebooks"""def__init__(self,path=None): self.shell = InteractiveShell.instance() self.path = pathdefload_module(self,fullname):"""import a notebook as a module""" path =find_notebook(fullname, self.path)print ("importing IPython notebook from %s"% path)# load the notebook objectwith io.open(path, 'r', encoding='utf-8')as f: nb = current.read(f, 'json')# create the module and add it to sys.modules# if name in sys.modules:# return sys.modules[name] mod = types.ModuleType(fullname) mod.__file__= path mod.__loader__ = self sys.modules[fullname]= mod# extra work to ensure that magics that would affect the user_ns# actually affect the notebook module's ns save_user_ns = self.shell.user_ns self.shell.user_ns = mod.__dict__try:for cell in nb.worksheets[0].cells:if cell.cell_type =='code'and cell.language =='python':# transform the input to executable Python code = self.shell.input_transformer_manager.transform_cell(cell.input)# run the code in themoduleexec(code, mod.__dict__)finally: self.shell.user_ns = save_user_nsreturn mod
The Module Finder
The finder is a simple object that tells you whether a name can be imported, and returns the appropriate loader. All this one does is check, when you do:
import mynotebook
it checks whether mynotebook.ipynb exists. If a notebook is found, then it returns a NotebookLoader.
Any extra logic is just for resolving paths within packages.
Now we register the NotebookFinder with sys.meta_path
sys.meta_path.append(NotebookFinder())
After this point, my notebooks should be importable.
Let's look at what we have in the CWD:
ls nbpackage
So I should be able to import nbimp.mynotebook.
Aside: displaying notebooks
Here is some simple code to display the contents of a notebook with syntax highlighting, etc.
from pygments import highlightfrom pygments.lexers import PythonLexerfrom pygments.formatters import HtmlFormatterfrom IPython.display import display, HTMLformatter =HtmlFormatter()lexer =PythonLexer()# publish the CSS for pygments highlightingdisplay(HTML("""<style type='text/css'>%s</style>"""% formatter.get_style_defs()))
defshow_notebook(fname):"""display a short summary of the cells of a notebook"""with io.open(fname, 'r', encoding='utf-8')as f: nb = current.read(f, 'json') html = []for cell in nb.worksheets[0].cells: html.append("<h4>%s cell</h4>"% cell.cell_type)if cell.cell_type =='code': html.append(highlight(cell.input, lexer, formatter))else: html.append("<pre>%s</pre>"% cell.source)display(HTML('\n'.join(html)))show_notebook(os.path.join("nbpackage", "mynotebook.ipynb"))
So my notebook has a heading cell and some code cells, one of which contains some IPython syntax.
Let's see what happens when we import it
from nbpackage import mynotebook
Hooray, it imported! Does it work?
mynotebook.foo()
Hooray again!
Even the function that contains IPython syntax works:
mynotebook.has_ip_syntax()
Notebooks in packages
We also have a notebook inside the nb package, so let's make sure that works as well.
ls nbpackage/nbs
Note that the __init__.py is necessary for nb to be considered a package, just like usual.