nbdev is a system for exploratory programming. See the nbdev launch post for information about what that means. In practice, programming in this way can feel very different to the kind of programming many of you will be familiar with, since we've mainly be taught coding techniques that are (at least implicitly) tied to the underlying tools we have access to. I've found that programming in a "notebook first" way can make me 2-3x more productive than I was before (when we used vscode, Visual Studio, vim, PyCharm, and similar tools).

In this tutorial, I'll try to get you up and running with the basics of the nbdev system as quickly and easily as possible. You can also watch this video in which I take you through the tutorial, step by step (to view full screen, click the little square in the bottom right of the video; to view in a separate Youtube window, click the Youtube logo):

Set up Your Jupyter Server

Jupyter Environment

To complete this tutorial, you'll need a Jupyter Notebook Server configured on your machine. If you have not installed Jupyter before, you may find the Anaconda Individual Edition the simplest to install.

If you already have experience with Jupyter, please note that everything in this tutorial must be run using the same kernel.

Install `nbdev`

No matter how you installed Jupyter, you'll need to manually install nbdev. You can install nbdev with pip or conda from a terminal window:

pip install nbdev

or

conda install -c fastai nbdev

Jupyter notebook has a terminal window available, so we'll use that:

Start jupyter notebook
From the "New" dropdown on the right side, choose Terminal.
Enter "python -m pip install nbdev" (or use conda as specified above)

When the command completes, you're ready to go.

Set up Repo

Template

To create your new project repo, click here: nbdev template (you need to be logged in to GitHub for this link to work). Fill in the requested info and click Create repository from template.

NB: The name of your project will become the name of the Python package generated by nbdev. For that reason, it is a good idea to pick a short, all-lowercase name with no dashes between words (underscores are allowed). Now, open your terminal, and clone the repo you just created.

Alternatively, you can use a cli command nbdev_new to create a new nbdev project from an existing GitHub repo that you have cloned locally. To create a new GitHub repo locally, we recommend the gh cli tool, which allows you to create a new repo with the command gh repo create.

_There is also a GitLab template hosted here (you need to be logged in to GitLab for this link to work)._

GitHub pages

The nbdev system uses jekyll for documentation. Because GitHub Pages supports Jekyll, you can host your site for free on Github Pages without any additional setup, so this is the approach we recommend (but it's not required; any jekyll hosting will work fine).

To enable Github Pages in your project repo, click Settings in Github, then click Pages on the left side bar. Set "Source" to Master branch /docs folder. Once you've saved, if you refresh that page, Github will have a link to your new website. Copy that URL, and then go back to your main repo page, click "edit" next to the description and paste the URL into the "website" section. While you're there, go ahead and put in your project description too. NOTE: Don't expect your Pages to build & deploy properly yet; only after we edit settings.ini (below) and make some other changes will your GitHub Pages site be deployed.

Previewing Documents Locally

It is often desirable to preview the documentation locally before having it built and rendered by GitHub Pages. This requires you to run Jekyll locally, which requires installing Ruby and Jekyll. Instructions on how to install Jekyll are provided on Jekyll's site. You can run the command make docs_serve from the root of your repo to serve the documentation locally after calling nbdev_build_docs to generate the docs.

In order to allow you to run Jekyll locally nbdev_template contains manifest files, called Gem files, that specify all Ruby dependencies for Jekyll & nbdev. If you do not plan to preview documentation locally, you can choose to delete Gemfile and Gemfile.lock from your nbdev repository.

Edit settings.ini

Next, edit the settings.ini file in your cloned repo. This file contains all the necessary information for when you'll be ready to package your library. The basic structure (that can be personalized provided you change the relevant information in settings.ini) is that the root of the repo will contain your notebooks, the docs folder will contain your auto-generated docs, and a folder with a name you select will contain your auto-generated modules.

You'll see these commented out lines in settings.ini. Uncomment them, and set each value as needed.

# lib_name = your_project_name
# repo_name = name of github repo
# user = your_github_username
# description = A description of your project
# keywords = some keywords
# author = Your Name
# author_email = [email protected]
# copyright = Your Name or Company Name
# branch = The default branch of your GitHub repo (usually either master or main)

We'll see some other settings we can change later.

Install git hooks to avoid and handle conflicts

Jupyter Notebooks can cause challenges with git conflicts, but life becomes much easier when you use nbdev. As a first step, run nbdev_install_git_hooks in the terminal from your project folder. This will set up git hooks which will remove metadata from your notebooks when you commit, greatly reducing the chance you have a conflict.

But if you do get a conflict later, simply run nbdev_fix_merge filename.ipynb. This will replace any conflicts in cell outputs with your version, and if there are conflicts in input cells, then both cells will be included in the merged file, along with standard conflict markers (e.g. =====). Then you can open the notebook in Jupyter and choose which version to keep.

Edit 00_core.ipynb

Now, run jupyter notebook, and click 00_core.ipynb (you don't have to start your notebook names with a number like we do here; but we find it helpful to show the order you've created your project in). You'll see something that looks a bit like this:

#default_exp core

module name here

API details.

Let's explain what these special cells mean.

Module name and summary

The markdown cell uses special syntax to define the title and summary of your module. Feel free to replace "module name here" with a title and "API details." with a summary for your module.

This cell can optionally be used to define jekyll metadata, see get_metadata for more details.

Add a function

Let's add a function to this notebook, e.g.:

#|export
def say_hello(to):
    "Say hello to somebody"
    return f'Hello {to}!'

Notice how it includes #export at the top - this means it will be included in your module, and documentation. The documentation will look like this:

Add examples and tests

It's a good idea to give an example of your function in action. Just include regular code cells, and they'll appear (with output) in the docs, e.g.:

say_hello("Sylvain")

'Hello Sylvain!'

Examples can output plots, images, etc, and they'll all appear in your docs, e.g.:

from IPython.display import display,SVG

display(SVG('<svg height="100"><circle cx="50" cy="50" r="40"/></svg>'))

You can also include tests:

assert say_hello("Jeremy")=="Hello Jeremy!"

You should also add markdown headings as you create your notebook; one benefit of this is that a table of contents will be created in the documentation automatically.

Build lib

Now you can create your python module. To do so, just run nbdev_build_lib from the terminal when anywhere in your project folder.

$ nbdev_build_lib
Converted 00_core.ipynb.
Converted index.ipynb.

Note if using a subdirectory to contain .ipynb files instead of the project root

If you have set the parameter nbs_path to be anything other than the project root, you will not be able to import your generated modules without an extra step:

either install these modules locally, as their relative import will take you beyond the top-level package, which can be done by running pip install -e . in the project root, to install the modules to your environment in editable mode.
or make a simlink in your notebook folder to the library folder, which can be done by running ln -s lib_path lib_name (adjust lib_path and lib_name to your use case).

Edit index.ipynb

Now you're ready to create your documentation home page and README.md file; these are both generated automatically from index.ipynb. So click on that to open it now.

You'll see that there's already a line there to import your library - change it to use the name you selected in settings.ini. Then, add information about how to use your module, including some examples. Remember, these examples should be actual notebook code cells with real outputs.

Build docs

Now you can create your documentation. To do so, just run nbdev_build_docs from the terminal when anywhere in your project folder.

$ nbdev_build_docs
converting: /home/jhoward/git/nbdev/nbs/00_core.ipynb
converting: /home/jhoward/git/nbdev/nbs/index.ipynb

This will export HTML versions of your notebooks to the docs directory, and will create hyperlinks for any words in backticks (as long as they exist in your module). It will also create a menu for all notebooks you have created, and a table of contents for each.

Commit to Github

You can now check-in the generated files with git add, git commit and git push. (You can use git status to check which files have been generated.) The following command should be sufficient:

git add -A; git commit -m'check in files'; git push

Wait a minute or two for Github to process your commit, and then head over to the Github website to look at your results.

Continuous Integration (CI)

Back in your project's Github main page, click where it says 1 commit (or 2 commits or whatever). Hopefully, you'll see a green checkmark next to your latest commit. That means that your documentation site built correctly, and your module's tests all passed! This is checked for you using continuous integration (CI) with GitHub actions. This does the following:

Checks the notebooks are readable (with nbdev_read_nbs)
check the notebooks have been cleaned of needless metadata to avoid merge conflicts (with nbdev_clean_nbs)
check there is no diff between the notebooks and the exported library (with nbdev_diff_nbs)
run the tests in your notebooks (with nbdev_test_nbs)

The template contains a basic CI that uses the four points above, edit the file .github/workflows/main.yml to your liking and comment out the parts you don't want.

If you have a red cross, that means something failed. Click on the cross, then click Details, and you'll be able to see what failed.

View docs and readme

Once everything is passing, have a look at your readme in Github. You'll see that your index.ipynb file has been converted to a readme automatically.

Next, go to your documentation site (e.g. by clicking on the link next to the description that you created earlier). You should see that your index notebook has also been used here.

Congratulations, the basics are now all in place! Let's continue and use some more advanced functionality.

Add a class

Create a class in 00_core.ipynb as follows:

#|export
class HelloSayer:
    "Say hello to `to` using `say_hello`"
    def __init__(self, to): self.to = to

    def say(self):
        "Do the saying"
        return say_hello(self.to)

This will automatically appear in the docs like this:

Document with show_doc

However, methods aren't automatically documented. To add method docs, use show_doc:

show_doc(HelloSayer.say)

And add some examples and/or tests:

o = HelloSayer("Alexis")
o.say()

'Hello Alexis!'

Add links with backticks

Notice above there is a link from our new class documentation to our function. That's because we used backticks in the docstring:

"Say hello to `to` using `say_hello`"

These are automatically converted to hyperlinks wherever possible. For instance, here are hyperlinks to HelloSayer and say_hello created using backticks.

Set up autoreload

Since you'll be often updating your modules from one notebook, and using them in another, it's helpful if your notebook automatically reads in the new modules as soon as the python file changes. To make this happen, just add these lines to the top of your notebook:

%load_ext autoreload
%autoreload 2

Add in-notebook export cell

It's helpful to be able to export all your modules directly from a notebook, rather than going to the terminal to do it. All nbdev commands are available directly from a notebook in Python. Add this line to any cell and run it to export your modules (I normally make this the last cell of my notebooks).

from nbdev.export import notebook2script; notebook2script()

Run tests in parallel

Before you push to github or make a release, you might want to run all your tests. nbdev can run all your notebooks in parallel to check for errors. Just run nbdev_test_nbs in a terminal.

(base) jhoward@usf3:~/git/nbdev$ nbdev_test_nbs
testing: /home/jhoward/git/nbdev/nbs/00_core.ipynb
testing: /home/jhoward/git/nbdev/nbs/index.ipynb
All tests are passing!

View docs locally

If you want to look at your docs locally before you push to Github, you can do so by running a jekyll server. First, install Jekyll by following these steps. Then, install the modules needed for serving nbdev docs by cding to the docs directory, and typing bundle install. Finally, cd back to your repo root and type make docs_serve. This will launch a server on port 4000 (by default) which you can connect to with your browser to view your docs.

If Github pages fails to build your docs, running locally with Jekyll is the easiest way to find out what the problem is.

Set up prerequisites

If your module requires other modules as dependencies, you can add those prerequisites to your settings.ini in the requirements section. The requirements should be separated by a space and if the module requires at least or at most a specific version of the requirement this may be specified here, too.

For example if your module requires the fastcore module of at least version 1.0.5, the torchvision module of at most version 0.7 and any version of matplotlib, then the prerequisites would look like this:

requirements = fastcore>=1.0.5 torchvision<0.7 matplotlib

In addition to requirements you can specify dependencies with other keywords that have different scopes. Below is a list of all possible dependency keywords:

requirements: Passed to both pip and conda setup
pip_requirements: Passed to pip setup only
conda_requirements: Passed to conda setup only
dev_requirements: Passed to pip setup as a development requirement

For more information about the format of dependencies, see the pypi and conda docs on creating specifications in setup.py and meta.yaml, respectively.

Set up console scripts

Behind the scenes, nbdev uses that standard package setuptools for handling installation of modules. One very useful feature of setuptools is that it can automatically create cross-platform console scripts. nbdev surfaces this functionality; to use it, use the same format as setuptools, with whitespace between each script definition (if you have more than one).

console_scripts = nbdev_build_lib=nbdev.cli:nbdev_build_lib

Test with editable install

To test and use your modules in other projects, and use your console scripts (if you have any), the easiest approach is to use an editable install. To do this, cd to the root of your repo in the terminal, and type:

pip install -e .

(Note that the trailing period is important.) Your module changes will be automatically picked up without reinstalling. If you add any additional console scripts, you will need to run this command again.

Upload to pypi

If you want people to be able to install your project by just typing pip install your-project then you need to upload it to pypi. The good news is, we've already created a fully pypi compliant installer for your project! So all you need to do is register at pypi (click "Register" on pypi) if you haven't previously done so, and then create a file called ~/.pypirc with your login details. It should have these contents:

[pypi]
username = your_pypi_username
password = your_pypi_password

Another thing you will need is twine, so you should run once

pip install twine

To upload your project to pypi, just type make pypi in your project root directory. Once it's complete, a link to your project on pypi will be printed.

NB: if you are not using the make release command to upload to both conda and pypi (described below), you must increment the version number in settings.ini each time you want to push a new release to pypi.

Upload to pypi and conda (recommended)

The command make release from the root of your nbdev repo will bump the version of your module and upload your project to both conda and pypi. We recommend this approach as it is the easiest way to publish packages.

Install collapsible headings and toc2

There are two jupyter notebook extensions that I highly recommend when working with projects like this. They are:

Collapsible headings: This lets you fold and unfold each section in your notebook, based on its markdown headings. You can also hit left to go to the start of a section, and right to go to the end
TOC2: This adds a table of contents to your notebooks, which you can navigate either with the Navigate menu item it adds to your notebooks, or the TOC sidebar it adds. These can be modified and/or hidden using its settings.

Math equation support

nbdev supports equations (we use the excellent KaTeX library). Enable it with use_math: true in your _config.yml (it's enabled by default). You can include math in your notebook's documentation using the following methods.

Using $$, e.g.:


$$\sum_{i=1}^{k+1}i$$

Which is rendered as:

$$\sum_{i=1}^{k+1}i$$

Using $, e.g.:

This version is displayed inline: $\sum_{i=1}^{k+1}i$ . You can include text before and after.

Which is rendered as:

This version is displayed inline:$\sum_{i=1}^{k+1}i$ . You can include text before and after.

Custom search engine

For adding search to your docs site, nbdev supports Google Custom Search, including auto-completion as you type your search query. You can try it out by using the search box at the top of this page.

Although we can't fully automate the creation of the search engine (since you need to login to Google to do it) we have made it very easy. Here are the steps you need to follow: Setting up search.

Google Colab Badges

Because both the documentation and code for nbdev is written in notebooks, you can optionally view and run nbdev documentation in Google Colab. You can enable Google Colab badges that link to the appropriate notebook(s) in your GitHub repository.

You can toggle this feature on or off in your docs/_config.yml file:

# This specifies what badges are turned on by default for notebook docs.
default_badges:
  colab: true

Furthermore, if you want to hide a badge on an individual document but still show badges elsewhere, you can set the front matter hide_colab_badge: true. For example, if you wanted to hide the Colab badge from showing up on the notebook nbs/06_cli.ipynb, your front matter (in the form of a markdown cell at the top of the notebook) will look like this:

# Command line functions
> Console commands added by the nbdev library

- hide_colab_badge:true

Note how in the above example, the title Command line functions is formatted as a markdown heading and the summary Console commands added by the nbdev library is formatted as a markdown block quote. The additional option hide_colab_badge is a list item. It is important that this list item is separated from the summary by 2 newlines as shown above, in the same notebook markdown cell.

Look at nbdev "source" for more ideas

Don't forget that nbdev itself is written in nbdev! It's a good place to look to see how fast.ai uses it in practice, and get a few tips. You'll find the nbdev notebooks here in the nbs folder on Github.

nbdev tutorial