A neural network is a way to model data by stacking layers of interconnected mathematical units called “neurons.” Unlike traditional algorithms that follow hand-written rules, neural networks learn patterns directly from example data.
In traditional programming, you write out logic: “If someone writes ‘great job,’ increase their rating by 1.” A neural network, instead, takes many labeled examples (such as messages marked positive or negative) and figures out, on its own, which patterns signal each label. This ability to learn directly from examples is what makes neural networks powerful, especially in cases where strict rules don’t capture all the subtleties of real data.
This is especially important in language tasks. Human language is subtle and context-dependent. Neural networks are good at picking up these nuances—patterns that are hard to describe with fixed rules but often appear in real conversations, reviews, or articles.
A neuron in a neural network is a small processing unit, inspired by how brain cells (also called neurons) send and receive signals. In biology, a neuron collects signals from others, processes them, and decides what to send next.
A neural network neuron works similarly. It takes input numbers, multiplies each by a “weight” (a number showing its importance), adds them up, tweaks the sum with a “bias,” then passes the result through a (usually nonlinear) mathematical function.
Neurons are arranged into layers. The first layer receives input data; each neuron in this layer processes the inputs in parallel. Their outputs form the inputs to the next layer, and so on, until the final layer produces the network’s output.
Think of each layer as a step on a factory assembly line. Raw materials (the input data) are transformed at each station, until you end up with a finished product (the network’s output).
Here’s what happens when you feed data into a basic neural network:
At every layer, the data is mixed, transformed, and prepared for the next step. This repeated transformation lets neural networks learn very complex relationships.
Let’s define some essential terms:
Weights are the adjustable numbers that decide how much each input affects a neuron’s output. A big weight means the input is important. A weight of zero means the input has no effect.
Biases are numbers added to the output of each neuron. They allow the neuron to shift its output independently of its inputs, making the model more flexible.
Activation functions are math formulas applied after the weighted sum and bias. They let neural networks model complicated relationships, not just straight lines. Without them, no matter how many layers you use, the network can only handle linear patterns.
Two common activation functions:
Neural networks learn through a process called training. Here’s a high-level view:
Don’t worry about the math for now. What matters is that neural networks gradually learn patterns in the data by repeatedly adjusting themselves based on feedback about their mistakes.
Neural networks have revolutionized natural language processing (commonly abbreviated as NLP):
Neural networks excel at all these tasks because they learn patterns directly from data—patterns too complex to be captured by simple rules.
Below is a minimal code example using PyTorch, a popular deep learning library. This code builds a one-layer (single linear layer) neural network that learns to map three input features to a single output. The example uses random data, so you can run it anywhere. This shows the core mechanics, not a production system.
import torch
import torch.nn as nn
# Random data: 100 examples, each with 3 features
X = torch.randn(100, 3)
# Random "labels" (what the network should learn to predict)
y = torch.randn(100, 1)
# Define a simple single-layer neural network
model = nn.Linear(3, 1)
# Mean squared error loss function
loss_fn = nn.MSELoss()
# Stochastic Gradient Descent optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
# Training loop
for epoch in range(100):
# Forward pass: compute prediction
y_pred = model(X)
# Compute loss
loss = loss_fn(y_pred, y)
# Zero gradients, backward pass, and update weights
optimizer.zero_grad()
loss.backward()
optimizer.step()
if (epoch+1) % 20 == 0:
print(f'Epoch {epoch+1}, Loss: {loss.item():.4f}')
# After training, model(X) gives outputs closer to y
This toy example walks through the training loop: predict, measure error, adjust, and repeat. For larger datasets and more complex models, the main idea stays the same.
Modify the provided PyTorch code to change the activation function in the network (e.g., use Sigmoid instead of no activation). Run the code, observe how the outputs differ before and after training, and write one sentence comparing the behaviors with the two activation functions.
Coming up on Day 18: Recurrent Neural Networks (RNNs) Explained