I have always been bad at calculus. I think it all started with I was locked into a good grade in college and I stopped going to class. When my next calc class started I had a trash teacher and have been digging myself out ever since. Because of this, I haven’t been able to figure out what the heck is going on with gradients while working with TFv2.
I would see that we would have 4 x 4 equal to 16 but when we did a gradient on that same thing we would get 8. So, here is my attempt to write this out and explain what is going on. I also have a slightly longer Colab Notebook on GitHub.
First, we have the equation and . We then try and find the derivate .
If you know calculus, which it appears EVERY web site I go to assumes you do, you can see that to find this derivative you use the power rule. This states that you can convert to
Using that, we can see becomes and since we get
Now, to the second example that uses a cube. , , and is . This then gives use and using $x = 2$ we get
Hopefully, this clears up what is going one. If not, you can just call me a dummy like I am sure everyone else already does when I try and do calculus.