Page 153 - Data Science Algorithms in a Week

Regression

The gradient descent algorithm does it by updating the parameter p in the direction of (∂/∂
i
p ) E(p), in particular:
j

Here, learning_rate determines the speed of the convergence of the E(p) to the minimum.
Updating of the parameter p will result in the convergence of E(p) to a certain value
providing that learning_rate is sufficiently small. In the Python program, we use
learning_rate of 0.000001. However, the drawback of this update rule is that the minimum of
E(p) may be only a local minimum.
To update the parameter p programatically, we need to unfold the partial derivative on
E(p). Therefore, we update the parameter p as follows:

We will keep updating the parameter p until it changes only a very little, that is, the change
of both p0 and p is less than some constant acceptable_error. Once the parameter p stabilizes,
1
we can use it to estimate the weight from the height.
Implementation:

# source_code/6/regression.py
# Linear regression program to learn a basic linear model.
import math
import sys
sys.path.append('../common')
import common # noqa
# Calculate the gradient by which the parameter should be updated.
def linear_gradient(data, old_parameter):
gradient = [0.0, 0.0]
for (x, y) in data:
term = float(y) - old_parameter[0] - old_parameter[1] * float(x)
gradient[0] += term
gradient[1] += term * float(x)
return gradient

[ 141 ]