nbdev is a system for exploratory programming. See the nbdev launch post for information about what that means. In practice, programming in this way can feel very different to the kind of programming many of you will be familiar with, since we've mainly be taught coding techniques that are (at least implicitly) tied to the underlying tools we have access to. I've found that programming in a "notebook first" way can make me 2-3x more productive than I was before (when we used vscode, Visual Studio, vim, PyCharm, and similar tools).
In this tutorial, I'll try to get you up and running with the basics of the nbdev system as quickly and easily as possible. You can also watch this video in which I take you through the tutorial, step by step (to view full screen, click the little square in the bottom right of the video; to view in a separate Youtube window, click the Youtube logo):
To complete this tutorial, you'll need a google account (with access to a google drive) and a github account.
It's probaly a good idea for everyone to change github settings so that the default branch of new repos is called main.
If you do, you can skip all of the "Rename the default branch in github" section below.
To create your new project repo, click here: nbdev template Warning: This is not the fastai template, this is my fork of nbdev_template (as of Apr 2020) (you need to be logged in to GitHub for this link to work). Fill in the requested info and click Create repository from template.
NB: The name of your project will become the name of the Python package generated by nbdev. For that reason, it is a good idea to pick a short, all-lowercase name with no dashes between words (underscores are allowed).
Now, open your terminal, and clone the repo you just created.
If you need to rename the default branch of your new repo:
1) From your new projects main page in github, create a new branch called main:
2) Navigate to the branches of your new project:
3) Click the "Change default branch" button and make main your default branch:
4) Navigate to the branches of your new project and delete the master branch:
For more details, see https://github.com/github/renaming/.
The rest of this tutorial will assume you're using a default branch called main.
The nbdev system uses jekyll for documentation. Because GitHub Pages supports Jekyll, you can host your site for free on Github Pages without any additional setup, so this is the approach we recommend (but it's not required; any jekyll hosting will work fine).
To enable Github Pages in your project repo, click Settings in Github, then scroll down to Github Pages, and set "Source" to main branch /docs folder. Once you've saved, if you scroll back down to that section, Github will have a link to your new website. Copy that URL, and then go back to your main repo page, click "edit" next to the description and paste the URL into the "website" section. While you're there, go ahead and put in your project description too.
1) Open this colab helper notebook
2) Use "Mount Drive" button in the "Files" side-bar to mount google drive locally
3) Create a file called nbdev_colab_projects.ini in your /content/drive/My Drive folder
4) Update nbdev_colab_projects.ini to have a section for your project, as desribed here
5) Run the "Install and import the colab helper module" cell
6) Change project_name to match the project name you used in /content/drive/My Drive/nbdev_colab_projects.ini and run the "set-up project" cell
After completing these steps, you should abe able to find your cloned repo in your google drive, via
- The "Files" side-bar in colab,
- https://drive.google.com/ etc
Feel free to save your copy of the colab helper notebook in your google drive.
Next, edit the settings.ini file in your cloned repo. This file contains all the necessary information for when you'll be ready to package your library. The basic structure (that can be personalized provided you change the relevant information in settings.ini) is that the root of the repo will contain your notebooks, the docs folder will contain your auto-generated docs, and a folder with a name you select will contain your auto-generated modules.
You'll see these commented out lines in settings.ini. Uncomment them, and set each value as needed.
# lib_name = your_project_name
# user = your_github_username
# description = A description of your project
# keywords = some keywords
# author = Your Name
# author_email = email@example.com
# copyright = Your Name or Company NameWe'll see some other settings we can change later.
Jupyter Notebooks can cause challenges with git conflicts, but life becomes much easier when you use nbdev. git_push will run nbdev_install_git_hooks automatically. This will set up git hooks which will remove metadata from your notebooks when you commit, greatly reducing the chance you have a conflict.
But if you do get a conflict later, simply run nbdev_fix_merge filename.ipynb. This will replace any conflicts in cell outputs with your version, and if there are conflicts in input cells, then both cells will be included in the merged file, along with standard conflict markers (e.g. =====). Then you can open the notebook in Jupyter and choose which version to keep.
Now, open 00_core.ipynb from your cloned repo (you don't have to start your notebook names with a number like we do here; but we find it helpful to show the order you've created your project in). You'll see something that looks a bit like this:
from nbdev import *
%nbdev_default_export core
module name here
API details.
If it doesn't look like ↑ please update it so that it does (o:You'll need to run some colab specific setup in each notebook, if you want to use all nbdev features. Add a new code cell to the start of 00_core.ipynb and (after changing project_name to match the project name you used in nbdev_colab_projects.ini) run:
IN_COLAB = 'google.colab' in str(get_ipython())
if IN_COLAB:
!pip install git+https://github.com/pete88b/nbdev_colab_helper.git
from nbdev_colab_helper.core import *
project_name = 'nbdev_colab_helper'
init_notebook(project_name)
Let's explain what these special cells mean.
There are certain special magics that, when placed in a cell, provide important information to nbdev.
Before you can use the magic flags, you need to import them: from nbdev import *. This also imports the show_doc and notebook2script functions.
The flag %nbdev_default_export core, defines the name of the generated module (lib_name/core.py). For any cells that you want to be included in your python module, type %nbdev_export as the first line of the cell. Each of those cells will be added to your module.
You can explore the available flags and their documentation without leaving your notebook:
The markdown cell uses special syntax to define the title and summary of your module. Feel free to replace "module name here" with a title and "API details." with a summary for your module.
This cell can optionally be used to define jekyll metadata, see get_metadata for more details.
Let's add a function to this notebook, e.g.:
%nbdev_export
def say_hello(to):
"Say hello to somebody"
return f'Hello {to}!'
Notice how it includes %nbdev_export at the top - this means it will be included in your module, and documentation. The documentation will look like this:
It's a good idea to give an example of your function in action. Just include regular code cells, and they'll appear (with output) in the docs, e.g.:
say_hello("Sylvain")
Examples can output plots, images, etc, and they'll all appear in your docs, e.g.:
from IPython.display import display,SVG
display(SVG('<svg height="100"><circle cx="50" cy="50" r="40"/></svg>'))
You can also include tests:
assert say_hello("Jeremy")=="Hello Jeremy!"
You should also add markdown headings as you create your notebook; one benefit of this is that a table of contents will be created in the documentation automatically.
Now you can create your python module. To do so, go back to your colab helper notebook and run:
from nbdev.export import notebook2script
notebook2script()
Converted 00_core.ipynb.
Converted index.ipynb.Now you're ready to create your documentation home page and readme file; these are both generated automatically from index.ipynb. So click on that to open it now.
You'll see that there's already a line there to import your library - change it to use the name you selected in settings.ini. Then, add information about how to use your module, including some examples. Remember, these examples should be actual notebook code cells with real outputs.
Please remove permalink: pretty from docs/_config.yml - if you see "404: File not found" errors when navigating your docs, this could well be the problem. TODO: remove this when bug is fixed in nbdev/nbdev_template.
Now you can create your documentation. To do so, go back to your colab helper notebook and run:
from nbdev.cli import nbdev_build_docs
nbdev_build_docs()
converting: /content/drive/My Drive/Colab Notebooks/github/nbdev_colab_t1/00_core.ipynb
converting: /content/drive/My Drive/Colab Notebooks/github/nbdev_colab_t1/index.ipynb
converting /content/drive/My Drive/Colab Notebooks/github/nbdev_colab_t1/index.ipynb to README.mdYou can now git commit and git push. To do so, go back to your colab helper notebook and (after changing the commit message) run:
commit_message = '**changeme**'
git_push(project_config['git_branch'], commit_message)
nbdev_install_git_hooks
Executing: git config --local include.path ../.gitconfig
Success: hooks are installed and repo's .gitconfig is now trusted
nbdev_build_lib
Converted 00_core.ipynb.
Converted index.ipynb.
Converted tutorial.ipynb.
git add *
git commit -m "*wip* tutorial"
[master 69d5dec] *wip* tutorial
13 files changed, 126 insertions(+), 107 deletions(-)
git push origin masterWait a minute or two for Github to process your commit, and then head over to the Github website to look at your results.
Back in your project's Github main page, click where it says 1 commit (or 2 commits or whatever). Hopefully, you'll see a green checkmark next to your latest commit. That means that your documentation site built correctly, and your module's tests all passed! This is checked for you using continuous integration (CI) with GitHub actions. This does the following:
- Checks the notebooks are readable
- Checks the notebooks have been cleaned of needless metadata to avoid merge conflicts
- Checks there is no diff between the notebooks and the exported library
- Runs the tests in your notebooks
Edit the file .github/workflows/main.yml if you need to modify any of the CI steps.
If you have a red cross, that means something failed. Click on the cross, then click Details, and you'll be able to see what failed.
Once everything is passing, have a look at your readme in Github. You'll see that your index.ipynb file has been converted to a readme automatically.
Next, go to your documentation site (e.g. by clicking on the link next to the description that you created earlier). You should see that your index notebook has also been used here.
Congratulations, the basics are now all in place! Let's continue and use some more advanced functionality.
Create a class in 00_core.ipynb as follows:
%nbdev_export
class HelloSayer:
"Say hello to `to` using `say_hello`"
def __init__(self, to): self.to = to
def say(self):
"Do the saying"
return say_hello(self.to)
This will automatically appear in the docs like this:
However, methods aren't automatically documented. To add method docs, use show_doc:
show_doc(HelloSayer.say)
And add some examples and/or tests:
o = HelloSayer("Alexis")
o.say()
Notice above there is a link from our new class documentation to our function. That's because we used backticks in the docstring:
"Say hello to `to` using `say_hello`"
These are automatically converted to hyperlinks wherever possible. For instance, here are hyperlinks to HelloSayer and say_hello created using backticks.
TODO: xxx check autoreload in colab
Since you'll be often updating your modules from one notebook, and using them in another, it's helpful if your notebook automatically reads in the new modules as soon as the python file changes. To make this happen, just add these lines to the top of your notebook:
%load_ext autoreload
%autoreload 2It's helpful to be able to export all your modules directly from a notebook, rather than going to the terminal to do it. All nbdev commands are available directly from a notebook in Python. Add this line to any cell and run it to export your modules (I normally make this the last cell of my scripts).
from nbdev.export import notebook2script
notebook2script()
Before you push to github or make a release, you might want to run all your tests. nbdev can run all your notebooks in parallel to check for errors. To do so, go back to your colab helper notebook and run:
from nbdev.cli import nbdev_test_nbs
nbdev_test_nbs()
testing /content/drive/My Drive/Colab Notebooks/github/nbdev_colab_t1/00_core.ipynb
testing /content/drive/My Drive/Colab Notebooks/github/nbdev_colab_t1/index.ipynb
All tests are passing!I don't think it's possible to look at your docs via colab before you push to Github.
If your module requires other modules as dependencies, you can add those prerequisites to your settings.ini in the requirements section. This should be in the same format as install_requires in setuptools, with each requirement separated by a space.
Behind the scenes, nbdev uses that standard package setuptools for handling installation of modules. One very useful feature of setuptools is that it can automatically create cross-platform console scripts. nbdev surfaces this functionality; to use it, use the same format as setuptools, with whitespace between each script definition (if you have more than one).
console_scripts = nbdev_build_lib=nbdev.cli:nbdev_build_libTo test and use your modules in other projects, and use your console scripts (if you have any), the easiest approach is to use an editable install.
Once you've mounted your google drive, you can install editable versions of projects (if you have already git cloned the project into your google drive).
The following example installs a local copy of nbdev_colab_helper after running setup_project('nbdev_colab_helper'):
!pip install -e "/content/drive/My Drive/Colab Notebooks/github/nbdev_colab_helper"
(Note that the trailing period is important.) Your module changes will be automatically picked up without reinstalling. If you add any additional console scripts, you will need to run this command again.
TODO: Test this and add colab specific details
If you want people to be able to install your project by just typing pip install your-project then you need to upload it to pypi. The good news is, we've already created a fully pypi compliant installer for your project! So all you need to do is register at pypi (click "Register" on pypi) if you haven't previously done so, and then create a file called ~/.pypirc with your login details. It should have these contents:
[pypi]
username = your_pypi_username
password = your_pypi_passwordTo upload your project to pypi, just type make release in your project root directory. Once it's complete, a link to your project on pypi will be printed.
NB: make release will automatically increment the version number in settings.py before pushing a new release to pypi. If you don't want to do this, run make pypi instead.
Don't forget that nbdev itself is written in nbdev! It's a good place to look to see how fast.ai uses it in practice, and get a few tips. You'll find the nbdev notebooks here in the nbs folder on Github.