What Is Backpropagation? Gradient Descent Explained Simply

What is backpropagation?

Backpropagation is the algorithm that sends a neural network's prediction error backward through its layers and calculates how much each weight contributed to that error. As a result, gradient descent nudges every weight in the direction that lowers the error. In short, that loop is how an AI model learns from examples.

In short, backpropagation answers the question "who is to blame, and by how much?" The Google Machine Learning Crash Course calls it the most common training algorithm for neural networks. It also notes that backpropagation makes gradient descent feasible for networks with many layers.

This article stays on one question: what is backpropagation, and how does a network learn from its mistakes? If you want the bigger picture of how networks are built, read our deep learning guide first. Here we use words instead of formulas, because the idea is simpler than the math suggests.

Why is descending a foggy mountain a good analogy?

First, imagine you stand on a mountaintop in thick fog and need to reach the valley. You cannot see far, but you can still feel the slope under your feet. So you take a small step in the direction that goes downhill.

Put simply, gradient descent works exactly like this. For example, the height of the mountain is the model's error, and the valley is the point where the error is lowest. Meanwhile, your position is the current set of weights. Then, at every step, you move toward the direction in which the error falls fastest.

But what is backpropagation doing here, and how do you know the slope? So that is where backpropagation comes in. Instead of testing millions of weights one by one ("what happens to the error if I change this weight a little?"), it sends the error backward and computes every slope in a single efficient pass.

• The mountain: the shape of the error surface, also called the loss landscape.
• Your position: the network's current weights.
• The feel of the slope: the gradient, meaning each weight's effect on the error.
• Your step size: the learning rate.
• The valley: the low-error region where the model predicts well.

How does one round of neural network training work?

In other words, training is the same loop repeated many times. In each round, the network first looks at examples, makes a prediction, measures its error and adjusts its weights slightly. The loop has four parts, because each one depends on the one before it.

1. Forward pass: the input travels through the layers and becomes a prediction.
2. Loss calculation: the model compares the prediction with the true value and produces one error number.
3. Backward pass: the algorithm carries that error from the output back to the input and finds each weight's gradient.
4. Update: gradient descent changes each weight a small amount in the direction that reduces the loss.

Then the loop starts again with new examples. It continues until the error curve flattens, and then training stops. For example, a model that sorts product photos for an online store runs this loop thousands of times before it can tell a shoe from a bag.

This structure is most visible in supervised learning, because the correct label is the anchor of the error calculation.

What is a loss function and why does everything start there?

A loss function is a single number that tells you how far the model's prediction is from the truth. A large number means the model is badly wrong, while a small one means it is close. Training aims to push that number as low as possible.

Without a loss function, "better" has no meaning at all. So gradient descent chooses its direction by looking at this number. So picking the wrong loss is like walking confidently toward the wrong destination.

However, the choice depends on the problem. For example, when you predict a number, such as weekly sales, you use a loss that measures the size of the gap. Likewise, when you predict a category, such as the type of object in a photo, you use a loss that punishes confident wrong answers.

Here is the key point: the loss is the only feedback the model ever sees. Because of that, if you define it badly, the model can become excellent at a behavior you never wanted.

What is a gradient and why do we call it a slope?

A gradient tells you how much, and in which direction, the loss changes when you change a weight by a tiny amount. For example, on a one-dimensional hill it is an ordinary slope. However, in a model with millions of weights, every weight has its own slope, and together they form the gradient.

If the gradient is positive, raising the weight raises the loss, so you should lower the weight. If it is negative, then the opposite holds. In other words, the sign gives the direction, and the size shows how much that weight matters.

The whole update rule fits into one symbolic line: new weight = old weight - learning rate × gradient. Put simply, the minus sign means you walk against the slope, which is downhill.

Mathematicians call the gradient a derivative. However, you do not need to compute derivatives to follow this article. So think of the gradient as a measure of influence.

How does gradient descent work?

Gradient descent is an iterative optimization method that finds the weights with the lowest loss. The gradient descent page describes the loop like this: calculate the current loss, find the gradient, update the parameters in the opposite direction and repeat until the loss stops falling.

Once the loss stops falling, we say the model has converged. In practice, a flat loss curve means more training gives little extra benefit. Also, watching that curve is the most useful habit in training.

First, it helps to separate two roles here. Gradient descent decides the direction and the step, because it owns the update rule. Backpropagation, on the other hand, produces the gradients that gradient descent needs. In a multi-layer network, therefore, neither works without the other.

• Start: the weights begin as small random values.
• Measure: the model predicts on a group of examples and the loss is computed.
• Direction: backpropagation supplies the gradient for every weight.
• Step: each weight moves a small amount against its gradient.
• Repeat: the loop runs until the loss flattens.

What is backpropagation doing with the chain rule?

First, a neural network is a stack of many small operations. The error at the last layer depends on the output of the previous layer, which in turn depends on the layer before it. So backpropagation multiplies the effect of each link along this chain. Mathematicians call that the chain rule.

For example, think of a factory line where the final product has a defect. To find the cause, you then walk the line backward and weigh how much each station contributed. Backpropagation walks backward in the same way and computes each weight's share of the responsibility.

Moreover, the biggest advantage is speed. Because changing weights one at a time and watching the result is practically impossible in large models. Backpropagation produces all gradients in one backward pass at a cost similar to the forward pass. As a result, that efficiency is one reason today's transformer architecture and large language models can be trained at all.

Historically, Rumelhart, Hinton and Williams introduced the method in their Nature paper. They described a learning procedure that repeatedly adjusts connection weights to reduce the gap between the actual and the desired output. According to the paper, hidden units then come to represent important features of the task.

Can one weight make the idea concrete?

Yes, so let us shrink the network to a single weight. This is an illustrative calculation, and the numbers only show the logic. The model predicts by multiplying the input by its weight. Say the input is two and the correct answer is ten.

1. First, the weight starts at three. The model predicts six, but the target is ten, so the gap is four. The loss, the square of the gap, is sixteen.
2. Backpropagation finds the gradient: twice the gap times the input, which here equals minus sixteen. The minus sign says that raising the weight lowers the loss.
3. We pick a small learning rate, say one twentieth (0.05). Then gradient descent moves the weight from three to about three point eight.
4. The new prediction is seven point six, and the loss falls to about five point seven six.
5. One more step moves the weight to about four point two eight, and the loss drops to roughly two point zero seven.

The steps shrink because the gradient shrinks along with the error. In other words, the ideal weight is five. With a few more steps the weight approaches it, and the loss flattens near zero.

In practice, the same logic runs for millions of weights at once. Still, each weight's story stays the same: error, influence, small step.

Why does the learning rate matter so much?

First, the learning rate sets how big each update step is. The hyperparameters page describes it as the setting that influences how quickly the model converges. If it is too low, training crawls. If it is too high, however, the model may never converge.

So return to the foggy mountain. With tiny steps, for example, reaching the valley takes days. With giant steps, you leap over the valley and land on the opposite slope, then bounce back and forth.

• Too low: the loss falls very slowly, and training can take hours or days.
• Too high: the loss jumps around or even rises, and the model never settles.
• Well chosen: the loss falls quickly at first, then slowly, then flattens.
• Decaying over time: big steps early and fine adjustments at the end.

In practice, the learning rate is the most frequently tuned setting. The loss curve is also the fastest way to find a good value. If the curve slides down smoothly, you are doing well. If it jitters, then lower the rate.

What do epoch, batch and iteration mean?

First, people mix up these three words all the time. An epoch is one full pass of the training data through the model. Next, a batch is the group of examples the model sees before it updates its weights. Finally, an iteration is a single update step.

Google's course pages define an epoch as one complete pass through the whole dataset, and they add that models usually need several epochs to converge. Also, batch size affects both training speed and how noisy the gradient is.

Here is a simple example scenario. Suppose your training data has one thousand examples and your batch size is twenty. Then one epoch equals fifty iterations. Then, if you train for ten epochs, you make five hundred updates. These numbers are an illustrative calculation, not a benchmark.

However, these settings differ from the model's parameters. The training process finds the weights and the bias by itself, so you do not set them. You choose the learning rate, the batch size and the number of epochs, and we call those hyperparameters.

Why do most teams use stochastic gradient descent and mini-batches?

First, using the whole dataset for every step is slow. Stochastic gradient descent (SGD) therefore uses only one example per step. Google's course pages explain that this gives noisy but computationally cheap updates.

The mini-batch approach is the middle path. It averages the gradient over a small group of examples at each step. As a result, the noise of a single example shrinks, and you avoid the cost of processing everything.

However, noise is not always bad. Also, small random wobbles can help a model escape a poor dip. Moreover, modern hardware is built to process groups in parallel, so mini-batches became the practical standard.

In practice, when someone says "we trained with SGD," they usually mean mini-batch training. The term is used loosely, so always check the batch size in the documentation.

What do optimizers like Adam actually do?

First, an optimizer is the rule that turns a gradient into a weight update. For example, the simplest optimizer multiplies the gradient by the learning rate. More advanced ones also take past steps into account.

The idea of momentum resembles a ball gaining speed rolling downhill. It keeps a running average of recent gradients and then rolls past small bumps. Adaptive methods such as Adam adjust the step size for each weight on their own. They take smaller steps for weights that update often, and larger steps for rare ones.

Still, one distinction matters. An optimizer never finds the gradient itself, because that is the job of backpropagation. Backpropagation produces it, and the optimizer decides what to do with it. Check your library's official documentation for current defaults, because recommendations change over time.

Optimizer choice also affects training cost in very large models. That topic belongs in the wider discussion of AI scaling laws.

Are local minima and saddle points a real problem?

First, the error surface of a simple linear model has a single bowl, and gradient descent reliably reaches the bottom. Google's course pages state this property for linear regression. In deep networks, however, the surface is far bumpier.

For example, a local minimum is a valley that is lower than its surroundings but not the deepest one anywhere. Likewise, a saddle point looks like a hill in one direction and a dip in another. In both places the gradient approaches zero, so training seems to stall.

Still, the good news is that the solutions large networks reach usually work well enough. You do not need the perfect valley, because a good one is enough. The practical goal is a reasonable region that performs well on validation data.

• Train a few times from different random starts to avoid an unlucky beginning.
• Mini-batch noise can shake a model out of shallow dips.
• Momentum and adaptive optimizers speed up progress across flat regions.
• If validation loss looks good, missing the global minimum rarely causes trouble.

What are vanishing and exploding gradients?

First, in very deep networks, gradients travel backward by repeated multiplication. If the numbers are smaller than one, the result shrinks, and the lower layers receive almost nothing. So we call that a vanishing gradient. Google's course pages describe the effect as lower layers training very slowly or not at all.

The opposite also happens. When weights are very large, gradients grow layer by layer and training becomes unstable. In short, that is an exploding gradient. As a result, numbers overflow, and the loss suddenly spikes or becomes undefined.

The course pages list the same remedies. Specifically, ReLU helps against vanishing gradients, while batch normalization and a lower learning rate help against explosions. Also, dropout is a separate regularization technique that reduces overfitting.

How does the activation function affect backpropagation?

First, an activation function bends a layer's output into a nonlinear shape. Without it, a network with any number of layers would still behave like one linear operation. Therefore, it is essential for capturing complex patterns.

For backpropagation, what matters is that the function has a computable derivative. Then, during the backward pass, the error is multiplied by the slope of each layer's function. If that slope is almost zero, the signal dies at that layer.

However, older smooth curves flatten near their ends and shrink the gradient. ReLU keeps the slope constant for positive values, so it eases the vanishing gradient problem in deep networks. Google's course pages cite ReLU as a common fix.

Still, the best choice depends on the architecture and the task. For beginners, the safest path is to start with the default function that your library and architecture recommend.

What are automatic differentiation and a computation graph?

First, modern libraries run backpropagation for you. The idea behind it is automatic differentiation. Then, during the forward pass, the library records every operation in a kind of ledger called a computation graph. In the backward pass, it reads the ledger from end to start and produces the gradients.

As a result, you do not need to write derivative formulas by hand to try a new architecture. So you define the layers, and the library finds the gradient. The same machinery is also optimized to run fast on different hardware.

There is a cost, though. The backward pass needs intermediate values from the forward pass, so the library must keep them in memory. Specifically, that is one factor that drives memory needs in large training runs. Some methods save memory by recomputing values when needed instead of storing them.

In short, as an engineer you rarely write backpropagation. Instead, you manage it. Setting up the loss, the learning rate and the data correctly is your job.

What is backpropagation compared with gradient descent and the forward pass?

In short, these four ideas are chapters of the same story. People confuse them because they run back to back in the same training loop. Also, the table below compares the job of each one in a single line.

The most common mistake is to treat backpropagation and gradient descent as the same thing. Backpropagation only computes gradients, and that is all. The part that changes the weights is gradient descent or one of its optimizer variants.

These ideas also reappear in other fields. For a related example, see our guide to reinforcement learning.

Where does backpropagation show up in daily life and in business?

For example, face unlock on your phone, a spam filter in your inbox, translation apps and chatbots are all neural networks trained with backpropagation. The large language models behind today's foundation models follow the same training logic. So scale and data are the main differences.

Next, consider a business example. For instance, a support team wants to sort incoming tickets by topic automatically. This is an example scenario. First, the team labels past tickets, the model then trains on those labels, and every wrong classification raises the loss. Backpropagation and gradient descent find the weights that reduce those mistakes.

Most companies never need to write this algorithm from scratch. Instead, ready-made libraries handle it. What you do need, however, is the ability to read a loss curve and to check data quality. If you plan a corporate AI project, our AI automation services page shows where we can help.

• Image classification: sorting product photos into categories.
• Text classification: grouping support requests or reviews by topic.
• Forecasting: predicting numeric values such as demand or stock needs.
• Generative models: training systems that produce text and images.

What are the limits and risks of backpropagation?

What is backpropagation not? It is not magic, although it is powerful. It also uses every piece of information you give it to lower the loss. Therefore, if the data is poor, the model learns poorly. A model with an excellent training loss that fails on new data has memorized instead of learned. We cover that failure in our guide to overfitting.

• Data dependence: if labels are wrong or biased, the model absorbs that error too.
• Compute cost: training big models demands a lot of energy and hardware.
• Sensitive settings: the learning rate and batch size change results noticeably.
• Explainability: after training, the weights carry no story a human can read.
• Biology: no one claims the brain uses this exact method. The algorithm is an engineering solution.

Also, training loss alone is not a measure of success. So monitor a separate validation set as well. For applications with legal or ethical consequences, such as personal data or decision support, this article is not legal advice, so consult an expert.

Which checklist should you use while monitoring training?

The short answer to what is backpropagation is a backward flow of blame, and this is the checklist that works for us in the field. Each item rests on a concept from this article.

1. Check that the loss function matches your real goal.
2. Plot training loss and validation loss on the same chart.
3. If the curve jitters, then lower the learning rate.
4. Instead, if the loss never falls, review the rate, the initial weights and the data.
5. When training loss falls while validation loss rises, consider early stopping and regularization.
6. Write down your epoch and batch settings so that you can reproduce the result.
7. If the loss suddenly spikes, suspect an exploding gradient.

Of course, this list does not solve every problem. Still, most training issues trace back to one of these seven points. When something breaks, look at the data first, then the settings. Blaming the algorithm is usually the last step.

Who popularized backpropagation, and how?

The idea of learning from errors is old, but making it work in multi-layer networks took time. The Nature paper by Rumelhart, Hinton and Williams brought the method to a wide research audience. Its title translates roughly as learning representations by back-propagating errors.

The paper stressed that hidden layers can learn to represent features that the input and output never state directly. As a result, that insight became one of the foundations of deep learning.

Meanwhile, in the decades that followed, better hardware and more data let the same idea run at very large scale. Most of today's models still train on this backward-pass logic at their core.

Which common misconceptions about backpropagation should you drop?

• "Backpropagation is a learning algorithm." In fact, it computes gradients. An update rule such as gradient descent completes the learning.
• "A bigger learning rate means faster learning." A rate that is too large breaks training.
• "Zero loss means a perfect model." Training loss can reach zero while the model fails on new data.
• "Gradient descent always finds the best solution." The surface of a deep network is bumpy, and there is no guarantee.
• "The brain learns this way." Nothing proves that. Artificial networks are only inspired by biology.

Most wrong answers to what is backpropagation come from putting all these ideas into one box. So keeping the table above in mind saves time when you debug.

For hands-on practice, our Python roadmap for AI is a good next step. It also helps you try these concepts in your own code.

Conclusion: how does an AI learn?

So what is backpropagation in one sentence? It is the backward pass that tells each weight how much to change. In short, an AI learns by correcting its own mistakes. The forward pass makes a prediction, the loss function measures the error, backpropagation computes each weight's share, and gradient descent corrects the weights in small steps. Then the loop repeats millions of times.

Finally, the learning rate, the batch size and the optimizer set the speed and the stability of that journey. You can catch problems such as vanishing gradients, exploding gradients and overfitting early by watching the loss curves.

This article is educational and does not recommend any product or provider. For fast-changing details, check the official documentation of the library and the provider you use.