DP: A lightweight library for teaching differentiable programming

Abstract

Deep Learning has recently gained a lot of interest, as nowadays, many practical applications rely on it.
Typically, these applications are implemented with the help of special deep learning libraries, which inner implementations are
hard to understand. We developed such a library in a lightweight way with a focus on teaching. Our library DP (differentiable
programming) has the following properties which fit particular requirements for education: small code base, simple concepts,
and stable Application Programming Interface (API). Its core use case is to teach how deep learning libraries work in principle.
The library is divided into two layers. The low-level part allows programmatically building a computational graph based on
elementary operations. In machine learning, the computational graph is typically the cost function including a machine learning
model, e.g. a neural network. Built-in reverse mode automatic differentiation on the computational graph allows the training of
machine learning models. This is done by optimization algorithms, such as stochastic gradient descent. These algorithms use the
derivatives to minimize the cost by adapting the parameters of the model. In the case of neural networks, the parameters are the
neuron weights. The higher-level part of the library eases the implementation of neural networks by providing larger building
blocks, such as neuron layers and helper functions, e.g., implementation of the optimization algorithms (optimizers) for training
neural networks. Accompanied to the library, we provide exercises to learn the underlying principles of deep learning libraries
and fundamentals of neural networks. An additional benefit of the library is that the exercises and corresponding programming
assignments based on it do not need to be permanently refactored because of its stable API