Vectorization Part 2

Introduction

When I first learned about vectorization, I was amazed at how much faster a vectorized algorithm could run compared to its non-vectorized counterpart. It almost felt like magic! Let’s dive into what makes vectorization so powerful.

Non-Vectorized Computation

Here’s how a for loop works without vectorization. If j ranges from 0 to 15, the loop performs operations one step at a time. At each timestamp, the algorithm computes a single value:

$$f(x) = \sum_{j=0}^{15} w_j x_j + b$$

This sequential process is slow, especially as n increases.

Vectorized Computation

With vectorization, the computer can fetch all the values of w and x, and compute their products all at once. It performs parallel computations, making the process much faster:

Vectorized operations allow the computer to calculate many values simultaneously, leading to much faster execution.

Multiple Linear Regression Example

Consider a linear regression problem with 16 features and 16 parameters. You compute the gradients of these parameters and update each one using gradient descent. Without vectorization, you would update the weights one by one:

In code, this looks like:

With vectorization, however, you can update all weights in parallel:

$$w = w - 0.1 \times d$$

This single line of code achieves the same result as the for loop, but much faster!

Benefits of Vectorized Code

Using a vectorized implementation makes linear regression more efficient, especially when dealing with large datasets or models with many features. The difference can be significant, reducing computation time from hours to minutes.

---

Conclusion

In this part, we explored the power of vectorization in practice. Using parallel processing, vectorized code significantly speeds up calculations, making it essential for scaling machine learning algorithms. In the next notebook, you'll explore the NumPy library in detail and see firsthand how vectorization impacts performance.