DP: A lightweight library for teaching differentiable programming

Abstract

Deep Learning (DL) has recently gained a lot of interest, as nowadays, many practical applications rely on it. Typically, these applications are implemented with the help of special deep learning libraries, which inner implementations are hard to understand. We developed such a library in a lightweight way with a focus on teaching. Our library DP (differentiable programming) has the following properties which fit particular requirements for education: small code base, simple concepts, and stable Application Programming Interface (API). Its core use case is to teach how deep learning libraries work in principle. The library is divided into two layers. The low-level part allows programmatically building a computational graph based on elementary operations. In machine learning, the computational graph is typically the cost function including a machine learning model, e.g. a neural network. Built-in reverse mode automatic differentiation on the computational graph allows the training of machine learning models. This is done by optimization algorithms, such as stochastic gradient descent. These algorithms use the derivatives to minimize the cost by adapting the parameters of the model. In the case of neural networks, the parameters are the neuron weights. The higher-level part of the library eases the implementation of neural networks by providing larger building blocks, such as neuron layers and helper functions, e.g., implementation of the optimization algorithms (optimizers) for training neural networks. Accompanied to the library, we provide exercises to learn the underlying principles of deep learning libraries and fundamentals of neural networks. An additional benefit of the library is that the exercises and corresponding programming assignments based on it do not need to be permanently refactored because of its stable API.