Training a Vision Model (II): Convolution, How an AI Learns to See

In the first article of this series we set up the skeleton: the project structure, the training script, and the `Net` class with its `conv1`, `pool`, `conv2`, `fc1`, and `fc2` layers. We said that `conv1` and `conv2` "detect patterns" — but we did not explain what that actually means.

In this second installment, we open the black box: what is a convolution, what is a filter (or kernel), and how all this looks in practice on a real image.

1. What is a Convolution?

A convolution is a mathematical operation applied to an image using a small window called a filter or kernel. This filter slides across the image, and at each position, performs a simple calculation that generates a new value. The final result is a new, smaller image where certain features (such as edges or textures) are highlighted.

To make it easy to understand:

• Imagine your image is a large grid of numbers (pixels).
• Your filter is a much smaller matrix, usually 3x3.
• At each position, you multiply each number in the filter by the image number underneath it, and sum all those results into a single value.

Photoshop: Filter > Other > Custom

Step-by-Step Numerical Example

Imagine we have a small image section (a 3x3 patch of a larger image) where there is a strong brightness contrast (a clean transition from white to black). In pixel values, it would look like this (high values represent white or light, low values represent black or shadow):

Image (3x3 section):
[  0,  0, 255 ]
[  0,  0, 255 ]
[  0,  0, 255 ]

And we will use this 3x3 filter specifically designed to detect vertical edges:

Filter (kernel):
[ -1,  0,  1 ]
[ -1,  0,  1 ]
[ -1,  0,  1 ]

To apply the convolution at this position: you multiply each filter number by the number directly below it in the image, and sum everything:

Calculation:
(0 * -1) + (0 * 0) + (255 * 1) +
(0 * -1) + (0 * 0) + (255 * 1) +
(0 * -1) + (0 * 0) + (255 * 1)
= 0 + 0 + 255 + 0 + 0 + 255 + 0 + 0 + 255
= 765

This resulting value `765` is a very high number (strong activation). This means the filter has successfully detected the pattern it was looking for in this position of the image.

What would happen in a flat color area? For example, a completely white section of the image:

Image (flat white 3x3 section):
[ 255, 255, 255 ]
[ 255, 255, 255 ]
[ 255, 255, 255 ]

If we apply the same filter on this flat section:

Calculation:
(255 * -1) + (255 * 0) + (255 * 1) +
(255 * -1) + (255 * 0) + (255 * 1) +
(255 * -1) + (255 * 0) + (255 * 1)
= -255 + 0 + 255 - 255 + 0 + 255 - 255 + 0 + 255
= 0

The result is `0` (no activation). The filter did not detect any light changes or edges.

2. Manual Filters vs. Learned Filters

The fundamental difference between classic computer vision and Deep Learning lies in how the numbers of this filter are decided:

1. A human designs them by hand (classic) — you decide to write `[-1, 0, 1]` because you know it will search for vertical edges.
2. The network learns them by itself during training (Deep Learning) — you do not tell it what to search for; it adjusts the filter numbers little by little, via *backpropagation*, until the filter becomes useful for classification.

Part 1: Manual Filters on a Real Image

We can write a quick Python script using OpenCV to load an image from our computer and apply hand-designed filters to check their effect:

import cv2                       # computer vision library, to read and process images
import numpy as np               # to work with matrices of numbers (arrays)
import matplotlib.pyplot as plt  # to draw results on screen
# We load an image and convert it to grayscale (a single channel instead of RGB,
# making it easier to see the filter effect without the distraction of color).
img = cv2.imread("images/entrenando-modelo-vision-cnn-parte2/fp-img.jpg", cv2.IMREAD_GRAYSCALE)
# We resize the image to 128x128 pixels, just so all examples
# have the same size and the calculation runs fast.
img = cv2.resize(img, (128, 128))
# We define three classic filters "by hand", each as a 3x3 matrix.
# These numbers were NOT learned through training: we wrote them ourselves
# on purpose, knowing what pattern we wanted them to detect.
filter_horiz = np.array([[-1, -2, -1],
                         [ 0,  0,  0],
                         [ 1,  2,  1]])   # reacts to top-to-bottom changes (horizontal edges)
filter_vert = np.array([[-1,  0,  1],
                        [-2,  0,  2],
                        [-1,  0,  1]])   # reacts to left-to-right changes (vertical edges)
filter_diag = np.array([[-2, -1,  0],
                        [-1,  4,  1],
                        [ 0,  1,  2]])   # reacts to diagonal changes
# cv2.filter2D applies the convolution: slides the filter over "img" and generates
# the resulting image (activation map) for each.
res_horiz = cv2.filter2D(img, -1, filter_horiz)
res_vert  = cv2.filter2D(img, -1, filter_vert)
res_diag  = cv2.filter2D(img, -1, filter_diag)
# We draw the 4 images side-by-side to compare the effect of each filter.
fig, axs = plt.subplots(1, 4, figsize=(15, 5))
axs[0].imshow(img, cmap='gray'); axs[0].set_title('Original (Grayscale)')
axs[1].imshow(res_horiz, cmap='gray'); axs[1].set_title('Horizontal Edges')
axs[2].imshow(res_vert, cmap='gray'); axs[2].set_title('Vertical Edges')
axs[3].imshow(res_diag, cmap='gray'); axs[3].set_title('Diagonal Edges')
plt.show()

What each filter type detects, in practice

3. Part 2: The Filters the CNN Learns on Its Own

Now let's look inside the network from the previous article. Recall that in `conv1` we defined 16 filters:

self.conv1 = nn.Conv2d(3, 16, 3, padding=1)

This literally means: *"I want 16 different filters, each 3x3, applied to a 3-channel (RGB) image"*. These 16 filters exist as numbers inside the model — and we can extract and draw them, just like we drew the manual filters before.

# net.conv1.weight contains the weights (numbers) of the 16 filters in conv1.
# .data detaches it from PyTorch's calculation graph (we only want to look at them, not train).
# .cpu() moves it to main computer memory, in case the model was on GPU.
filters = net.conv1.weight.data.cpu()
# Filter values can be negative, positive, widely dispersed...
# This line rescales them all between 0 and 1, to draw them as a normal image.
filters_norm = (filters - filters.min()) / (filters.max() - filters.min())
fig, axs = plt.subplots(2, 4, figsize=(10, 5))
for i in range(8):   # we only draw the first 8 out of 16, not to overcrowd the image
    ax = axs[i // 4, i % 4]
    filter_img = filtros_norm[i]
    # Each filter has shape [channels, height, width] (e.g., [3, 3, 3]).
    # permute(1, 2, 0) reorders the axes to [height, width, channels],
    # which is the format matplotlib expects to print a color image.
    filter_img = filter_img.permute(1, 2, 0).numpy()
    ax.imshow(filter_img)
    ax.set_title(f'Filter {i+1}')
    ax.axis('off')
plt.show()

An important detail: If you have just created the model and haven't trained it yet, these filters will have random values — that is how neural networks always start, like a blank page. You can already glimpse some structure (diagonal lines, brighter or darker areas), but they don't mean anything yet. It is during training, via *backpropagation*, when these numbers are adjusted little by little until they become truly useful detectors: one will end up looking like the vertical filter we designed by hand, another like the horizontal one, another like more complex textures — without anyone explicitly programming it to do so.

4. Classifying an Image with an Untrained Model

As a practical wrap-up, we will pass a real image through the model — even though it still has its factory random weights, without training. The goal here is not for it to get it right (it can't, it hasn't learned anything yet), but to verify that the entire flow works from start to finish: load image -> convert to tensor -> pass through the network -> get a prediction.

from PIL import Image
import torch
import torchvision.transforms as transforms
# We load the image from disk
pil_img = Image.open("images/entrenando-modelo-vision-cnn-parte2/fp-img.jpg")
# We apply the same transformations defined in the training script
transform = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])
# We add the "batch" dimension: PyTorch expects a group of images,
# so we convert a single image of shape [3, 32, 32] into a group of one image: [1, 3, 32, 32].
img_tensor = transform(pil_img).unsqueeze(0).to(device)
# We pass the image through the model
net.eval() # we set the model to evaluation mode (disables dropout, batchnorm, etc.)
with torch.no_grad(): # we disable gradient calculation, since we are not training now
    outputs = net(img_tensor)
# We simulate 3 possible classes, in the same order as in a real ImageFolder.
classes = ["Hen", "Duck", "Goose"]
# We load the image, pass it through the model (net), and check which output neuron
# has the highest value: that is "the chosen class" by the model.
# torch.max(outputs, 1) returns the index of the most activated neuron.
_, predicted = torch.max(outputs, 1)
print(f"The model classified the image as: {classes[predicted.item()]}")

With random weights, the result is indeed random — the model might say "Duck" just as it might say "Hen" or "Goose", no matter the image you place in front of it. But verifying that the complete pipeline runs (without shape, data type, or dimension errors...) is exactly what we need to verify before launching real training, which is what we will explore in a future article in this series.

5. What We Have Learned in This Installment

• What a convolution is, with a step-by-step hand-calculated numerical example.
• The difference between a filter you design by hand and a filter the network learns on its own.
• That filters in a newly created network start with random values and specialize during training.
• How to verify that the classification pipeline works end-to-end, even before training.

We have one last piece left to explain: what `pool` does exactly beyond "reducing size," and what goes on inside the dense layers (`fc1`, `fc2`) that receive that result and make the final decision. That is what we address in the third and final installment of this series.