---
description: Converting HTML to a Jupyter Notebook
---

# Converting HTML to Notebook

In this short post, we are going to learn how to turn the code from blog posts to Jupyter notebooks. Specifically, we will learn how to convert HTML to Jupyter Notebooks (.ipynb).

In this post, we are going to use the Python packages [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/), [json](https://docs.python.org/3/library/json.html), and [urllib](https://docs.python.org/3/library/urllib.html). We are going to use these packages to scrape the code from webpages putting their code within _\<code>\</code>_.

Note, this code is not intended to steal other people’s code. I created this script to scrape my code and save it to Jupyter notebooks because I noticed that my code, sometimes did not work as intended.

![](https://www.marsja.se/wp-content/uploads/2019/10/how_to_convert_a_website_to_jupyter_notebook-1024x361.png)

Table of Contents

* [Install the Needed Packages](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#Install_the_Needed_Packages)
  * [How to Install Python Packages using conda](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#How_to_Install_Python_Packages_using_conda)
  * [How to Install Python Packages using Pip](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#How_to_Install_Python_Packages_using_Pip)
* [How to Convert HTML to a Jupyter Notebook](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#How_to_Convert_HTML_to_a_Jupyter_Notebook)
  * [Setting a Custom User-Agent](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#Setting_a_Custom_User-Agent)
  * [Jupyter Notebook Metadata](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#Jupyter_Notebook_Metadata)
  * [Getting the Code Elements from the HTML](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#Getting_the_Code_Elements_from_the_HTML)
* [Conclusion: How to Convert HTML to Jupyter Noteboks (.ipynb)](https://www.marsja.se/converting-html-to-a-jupyter-notebook/#Conclusion_How_to_Convert_HTML_to_Jupyter_Noteboks_ipynb)

### Install the Needed Packages

Now, we need to install BeautifulSoup4 before we continue converting HTML\
to Jupyter notebooks. Furthermore, we need to install lxml.

#### How to Install Python Packages using conda

In this section, we are going to learn how to install the needed packages using the packages manager _conda_. First, open up the _Anaconda Powershell Prompt_

Now, we are ready to install BeautifulSoup4.

```
conda -c install anaconda beautifulsoup4 lxml
```

#### How to Install Python Packages using Pip

It is, of course, possible to install the packages using _pip_ as well:

```
install beautifulsoup4 lxml
```

See the more recent post about[ installing packages in Python](https://www.marsja.se/learn-all-about-installing-updating-packages-in-python/) for more information.

### How to Convert HTML to a Jupyter Notebook

Now, when we have installed the Python packages, we can continue with scraping the code from a web page. In the example, below, we will start by importing BeautifulSoup from bs4, json, and urllib. Next, we have the URL to the webpage that we want to convert to a Jupyter notebook (this).

```python
from bs4 import BeautifulSoup
  import json
  import urllib

  url = 'https://www.marsja.se/python-manova-made-easy-using-statsmodels/'
```

#### Setting a Custom User-Agent

In the next line of code, we create the dictionary _headers_.

This is because many websites (including the one you are reading now) will block web scrapers and this will prevent that from happening.

```
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11'\
             '(KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
         'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
         'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
         'Accept-Encoding': 'none',
         'Accept-Language': 'en-US,en;q=0.8',
         'Connection': 'keep-alive'}
```

In the next code chunk, we are going to create a Request object. This object represents the HTTP request we are making.

Simply put, we create a Request object that specifies the URL we want to retrieve. Furthermore, we are calling _urlopen_ using the Request object. This will, in turn, a response object for the requested URL.

Finally, we _call .read()_ on the response:

```python
req = urllib.request.Request(url,
    headers=headers)
  page = urllib.request.urlopen(req)
  text = page.read()
```

We are now going to use BeautifulSoup4 to get make it easier to scrape the HTML:

```python
soup = BeautifulSoup(text, 'lxml')
soup
```

#### Jupyter Notebook Metadata

Now we’re ready to convert HTML to a Jupyter Notebook (this code was inspired by this [code example](https://icecube.wisc.edu/~icecube-bootcamp/bootcamp2019/introduction/convert.py)). First, we start by creating some metadata for the Jupyter Notebook

<img src="https://www.marsja.se/wp-content/uploads/2019/10/convert_website_to_jupyter_notebook_code.png" alt="" data-size="original">

Jupyter notebook metadata

In the code below, we start by creating a dictionary in which we will, later, store our scraped code elements. This is going to be the metadata for the Jupyter notebook we will create. Note, .ipynb are simple JSON files, containing text, code, rich media output, and metadata. The metadata is not required but here we will add what language we are using (i.e., Python 3).

```
create_nb = {'nbformat': 4, 'nbformat_minor': 2, 
              'cells': [], 'metadata': 
             {"kernelspec": 
              {"display_name": "Python 3", 
               "language": "python", "name": "python3"
  }}}
```

Example code cell from a Jupyter Notebook

![](https://www.marsja.se/wp-content/uploads/2019/10/how_to_scrape_html_convert_python_code_to_jupyter_notebook-1.png)

More information bout the format of Jupyter notebooks can be found [here](https://ipython.org/ipython-doc/dev/notebook/nbformat.html).

#### Getting the Code Elements from the HTML

Second, we are creating a Python function called _get\_code_. This function will take two arguments. First, the beautifulsoup object, we earlier created, and the content\_class to search for content in. In the case, of this particular WordPress, blog this will be _post-content_

Next, we are looping through all _div_ tags in the soup object. Here, we only look for the post content. Next, we get all the code chunks searching for all _code_ tags

In the final loop, we are going through each code chunk and creating a new dictionary (_cell_) in which we are going to store the code. The important part is where we add the text, using the _get\_text_ method. Here we are getting our code from the code chunk and add it to the dictionary.

![](https://www.marsja.se/wp-content/uploads/2019/10/convert_html_to_ipynb.png)

Finally, we add this to the dictionary, _nb\_data_, that will contain the data that we are going to save as a Jupyter notebook (i.e., the blog post we have scraped).

```python
def get_data(soup, content_class):
    for div in soup.find_all('div', 
                             attrs={'class': content_class}):
        
        code_chunks = div.find_all('code')
        
        for chunk in code_chunks:
            cell_text = ' '
            cell = {}
            cell['metadata'] = {}
            cell['outputs'] = []
            cell['source'] = [chunk.get_text()]
            cell['execution_count'] = None
            cell['cell_type'] = 'code'
            create_nb['cells'].append(cell)

get_data(soup, 'post-content')

with open('Python_MANOVA.ipynb', 'w') as jynotebook:
    jynotebook.write(json.dumps(create_nb))
```

* More about [parsing JSON in Python](https://www.marsja.se/how-to-read-and-write-json-files-using-python-and-pandas/)

Note, we get the _nb\_data_ which is a dictionary from which we will create our notebook. In the final two rows, of the code chunk, we will open a file (i.e., test.ipynb) and write to this file using the json dump method.

[Here’s a Jupyter notebook ](https://github.com/marsja/jupyter/blob/master/convert_html_jupyter_notebook_tutorial.ipynb)containing all code above.

### Conclusion: How to Convert HTML to Jupyter Noteboks (.ipynb)

In this post, we have learned how to convert HTML pages to Jupyter notebooks. Specifically, we used BeautifulSoup, lxml, and Python.

[Source :](https://www.marsja.se/converting-html-to-a-jupyter-notebook/)
