Converting HTML to a Jupyter Notebook
Last updated
Last updated
In this short post, we are going to learn how to turn the code from blog posts to Jupyter notebooks. Specifically, we will learn how to convert HTML to Jupyter Notebooks (.ipynb).
In this post, we are going to use the Python packages BeautifulSoup4, json, and urllib. We are going to use these packages to scrape the code from webpages putting their code within <code></code>.
Note, this code is not intended to steal other people’s code. I created this script to scrape my code and save it to Jupyter notebooks because I noticed that my code, sometimes did not work as intended.
Table of Contents
Now, we need to install BeautifulSoup4 before we continue converting HTML to Jupyter notebooks. Furthermore, we need to install lxml.
In this section, we are going to learn how to install the needed packages using the packages manager conda. First, open up the Anaconda Powershell Prompt
Now, we are ready to install BeautifulSoup4.
It is, of course, possible to install the packages using pip as well:
See the more recent post about installing packages in Python for more information.
Now, when we have installed the Python packages, we can continue with scraping the code from a web page. In the example, below, we will start by importing BeautifulSoup from bs4, json, and urllib. Next, we have the URL to the webpage that we want to convert to a Jupyter notebook (this).
In the next line of code, we create the dictionary headers.
This is because many websites (including the one you are reading now) will block web scrapers and this will prevent that from happening.
In the next code chunk, we are going to create a Request object. This object represents the HTTP request we are making.
Simply put, we create a Request object that specifies the URL we want to retrieve. Furthermore, we are calling urlopen using the Request object. This will, in turn, a response object for the requested URL.
Finally, we call .read() on the response:
We are now going to use BeautifulSoup4 to get make it easier to scrape the HTML:
Now we’re ready to convert HTML to a Jupyter Notebook (this code was inspired by this code example). First, we start by creating some metadata for the Jupyter Notebook
Jupyter notebook metadata
In the code below, we start by creating a dictionary in which we will, later, store our scraped code elements. This is going to be the metadata for the Jupyter notebook we will create. Note, .ipynb are simple JSON files, containing text, code, rich media output, and metadata. The metadata is not required but here we will add what language we are using (i.e., Python 3).
Example code cell from a Jupyter Notebook
More information bout the format of Jupyter notebooks can be found here.
Second, we are creating a Python function called get_code. This function will take two arguments. First, the beautifulsoup object, we earlier created, and the content_class to search for content in. In the case, of this particular WordPress, blog this will be post-content
Next, we are looping through all div tags in the soup object. Here, we only look for the post content. Next, we get all the code chunks searching for all code tags
In the final loop, we are going through each code chunk and creating a new dictionary (cell) in which we are going to store the code. The important part is where we add the text, using the get_text method. Here we are getting our code from the code chunk and add it to the dictionary.
Finally, we add this to the dictionary, nb_data, that will contain the data that we are going to save as a Jupyter notebook (i.e., the blog post we have scraped).
More about parsing JSON in Python
Note, we get the nb_data which is a dictionary from which we will create our notebook. In the final two rows, of the code chunk, we will open a file (i.e., test.ipynb) and write to this file using the json dump method.
Here’s a Jupyter notebook containing all code above.
In this post, we have learned how to convert HTML pages to Jupyter notebooks. Specifically, we used BeautifulSoup, lxml, and Python.
