Calculation of derivatives of cost function for Convolutional Neural Networks

Published in June 2019

This exercise was driven by pure curiosity rather than practical need. I was curious whether I can take a relatively simple CNN, calculate partial derivatives of cost function with respect to all learnable parameters analytically by hand, and then program it without using any ML framework or autograd library. I did exactly that, trained an image classifier, and got the same accuracy as in TensorFlow. I’m publishing the calculations here in case if anybody, for whatever reason, would be interested to follow.

Download PDF