The use of control theory methods in neural networks’ training based on a handwritten text

Abstract

The paper shows the importance of reducing the neural networks’ training time at present stage and the role of new optimization methods in neural networks’ training. The paper researches a modification of stochastic gradient descent, which is based on the idea of gradient descent representation as a discrete dynamical system. The connection between the extreme points, to which the gradient descent iterations tend, and the stationary points of the corresponding discrete dynamical system is a consequence of this representation. The further applied stabilizing scheme with predictive control, for which a theoretical apparatus was developed by means of geometric complex analysis together with solving optimization tasks in a set of polynomials with real coefficients, was able to train a multilevel perceptron for recognizing handwritten numbers many times faster. The new algorithm software implementation used the PyTorch library, created for researches in the field of neural networks. All experiments were run on NVidia graphical processing unit to check the processing unit’s resource consumption. The numerical experiments did not reveal any deviation in training time. There was a slight increase in the used video memory, which was expected as the new algorithm retains one additional copy of perceptron internal parameters. The importance of this result is associated with the growth in the use of deep neural network technology, which has grown three hundred thousand times from 2012 till 2018, and the associated resource consumption. This situation forces the industry to consider training optimization issues as well as their accuracy. Therefore, any training process acceleration that reduces the time or resources of the clusters is a desirable and important result, which was achieved in this article. The results obtained discover a new area of theoretical and practical research, since the stabilization used is only one of the methods of stabilization and search for cycles in control theory. Such good practical results confirm the need to add the lagging control and the additional experiments with both predictive and lagging control elements