Neural Networks from Scratch - P.2 Coding a Layer

TL;DR

A neuron’s forward computation is a weighted sum of its inputs plus a single bias scalar.

Briefing Cornell Notes

Briefing

Neural networks from scratch take shape by treating each neuron as a simple arithmetic unit: multiply each input by its own weight, add a single bias, and produce one output. The practical takeaway is that bias is tied to the neuron—not to each weight—so a neuron with three inputs still has one bias, while a neuron with four inputs has four weights but still only one bias. Using fixed “book” values (inputs 1, 2, 3; weights 0.2, 0.8, -0.5; bias 2), the computation lands on an output of 2.3, establishing the baseline formula that later training will refine.

From there, the build shifts from one neuron to more realistic layer behavior. Adding an extra input changes only the number of weights: the neuron now has four inputs and therefore four weights, but the bias remains a single scalar. With inputs 1, 2, 3, and 2.5 and a new weight set (1.0, 1.0, 1.0, 1.0) paired with the same single-bias structure, the updated output becomes 4.8. The transcript also clarifies what “inputs” can represent inside a network: they may be raw sensor-like features from an input layer, or they may be outputs emitted by earlier neurons in hidden layers. Either way, once those values feed a neuron, the neuron’s job stays the same—weighted sum plus bias.

The next step models a full layer by computing outputs for multiple neurons in parallel. When three neurons each receive the same four inputs, the layer output becomes a three-element vector: each neuron has its own weight set and its own bias, producing three distinct outputs. The code is expanded accordingly: weights are grouped into three separate sets (weight1, weight2, weight3), and biases are grouped into three separate scalars (bias1, bias2, bias3). Using the book’s specific numbers—weight1 (0.5, -0.9, 1.0, 0.2), weight2 (6, -0.5, -0.2, 0.6), and weight3 (-0.26, -0.27, 0.17, 0.87) with biases 2, 3, and 0.5—the layer outputs come out as approximately 4.8, 1.2, and 2.385.

The closing emphasis ties the arithmetic to learning: inputs are typically not directly adjustable because they come from data or earlier layers, but weights and biases can be tuned to change outputs. That tuning is the core challenge of deep learning, later handled through back propagation and gradient-based updates. For now, the focus stays on getting the shapes right—single neuron vs. neuron with more inputs vs. multiple neurons in a layer—so the math and the code remain consistent as the network grows more complex.

Cornell Notes

Each neuron computes a weighted sum of its inputs and adds exactly one bias, then outputs a single value. Increasing the number of inputs increases the number of weights, but the neuron still has only one bias. Inputs themselves can represent either raw feature values (like sensor readings) from an input layer or outputs produced by earlier neurons in hidden layers. When multiple neurons form a layer, each neuron keeps its own weight set and bias, producing a vector of outputs—one per neuron. Learning later focuses on adjusting weights and biases to change outputs, since inputs are usually fixed by data or earlier computations.

Why does a neuron have one bias even when it has multiple weights?

Bias is associated with the neuron as a whole, not with each connection. A neuron with three inputs has three weights (one per input) but only one bias scalar added after the weighted sum. The computation pattern stays: output = (inputs · weights) + bias. That’s why the same bias structure persists when moving from 3 inputs to 4 inputs.

What changes when a neuron gains an extra input?

Only the number of weights changes. Going from inputs (1, 2, 3) to (1, 2, 3, 2.5) increases the weight vector length from 3 to 4. The bias remains a single value. In the example, the neuron’s output updates from 2.3 to 4.8 after adding the extra input and using a new 4-weight set.

Where can neuron inputs come from inside a network?

Inputs can be either raw feature values from the input layer (e.g., sensor readings like heat and humidity) or outputs from neurons in earlier layers. In both cases, the neuron treats them as numeric values feeding its weighted sum. Later in the series, the distinction matters for how the network is wired, but the neuron math remains the same.

How does modeling three neurons in one layer differ from modeling one neuron?

A layer with three neurons produces three outputs, so it needs three separate weight sets and three separate biases. The inputs remain the same four values for all three neurons, but each neuron multiplies those inputs by its own four weights and adds its own bias. The result is a three-element output vector (one value per neuron).

Why can’t inputs usually be tuned during learning?

Inputs are typically fixed either by the dataset (sensor/feature values) or by outputs from earlier neurons. Changing them directly would break the network’s structure. Instead, learning adjusts weights and biases because those parameters control how fixed inputs are transformed into outputs.

What is the immediate learning objective before back propagation appears?

Before gradients enter the picture, the objective is to get the forward computation correct and consistent across shapes: single neuron with 3 inputs, single neuron with 4 inputs, and a layer of 3 neurons each with 4 inputs. Once those outputs are correct, back propagation can later compute how to update weights and biases to reduce error.

Review Questions

In the neuron formula, which term changes when the number of inputs increases, and which term stays constant?
When three neurons share the same four inputs, what parameters must differ across neurons to produce different outputs?
Why are weights and biases the primary targets for tuning during training rather than the inputs?

Key Points

1
A neuron’s forward computation is a weighted sum of its inputs plus a single bias scalar.
2
Adding inputs increases the number of weights but does not add additional biases per weight.
3
Neuron inputs can represent either raw feature values (input layer) or outputs from earlier neurons (hidden layers).
4
A layer with multiple neurons requires separate weight sets and separate biases for each neuron.
5
Layer outputs form a vector: one output value per neuron, computed independently using the same input vector.
6
Learning focuses on adjusting weights and biases because inputs are usually fixed by data or earlier computations.
7
Correctly matching tensor/vector shapes (single value vs. multi-neuron outputs) is essential before introducing back propagation.

Highlights

Bias belongs to the neuron, not to each connection: three inputs still mean one bias and three weights.

When inputs grow from 3 to 4, the neuron’s output changes only because the weight vector grows; the bias structure stays the same.

Modeling a layer means computing multiple neuron outputs in parallel, each with its own weights and bias, producing a multi-value output vector.

The path to learning runs through tuning weights and biases, since inputs typically can’t be directly changed.

Topics

Neural Network Layers
Neuron Arithmetic
Weights And Biases
Forward Pass
Back Propagation