This seminal paper introduces a large, deep convolutional neural network (CNN) designed to classify 1.2 million high-resolution images into 1,000 distinct categories for the ImageNet LSVRC-2010 contest. The authors implemented a 60-million parameter architecture consisting of five convolutional layers followed by three fully-connected layers. To enable efficient training of such a large model, they utilized Rectified Linear Units (ReLUs) for non-saturating nonlinearity and a highly optimized GPU implementation to parallelize the workload across two GPUs. To combat significant overfitting, the researchers employed 'dropout' regularization and extensive data augmentation techniques including image translations and horizontal reflections. The final model achieved record-breaking performance, including a top-5 error rate of 15.3% in the ILSVRC-2012 competition, nearly halving the error rate of the next best non-CNN entry.
Key Takeaways
The CNN achieved a winning top-5 test error rate of 15.3% in the ILSVRC-2012 competition, outperforming the second-best entry's 26.2%.
The architecture consists of 60 million parameters and 650,000 neurons across eight learned layers (five convolutional and three fully-connected).
Replacing traditional tanh neurons with Rectified Linear Units (ReLUs) allowed the network to train six times faster on the CIFAR-10 dataset.
Dropout regularization, applied to the first two fully-connected layers, was essential for reducing complex co-adaptations and preventing overfitting.
Data augmentation via random patch extraction and horizontal reflections effectively increased the training set size by a factor of 2048.
The network's depth is critical for performance; removing even a single convolutional layer resulted in a top-1 performance loss of approximately 2%.
Learning Objectives
Explain the architectural components of a large-scale convolutional neural network.
Identify the advantages of non-saturating ReLUs over traditional saturating neurons.
Describe techniques for parallelizing neural network training across multiple GPUs.
Analyze common strategies for preventing overfitting in high-capacity models, such as dropout and data augmentation.
Evaluate the impact of dataset size and model depth on image classification performance.
Glossary
Convolutional Neural Network (CNN)
A class of deep neural networks most commonly applied to analyzing visual imagery, characterized by local pixel dependencies and stationary statistics.
ReLU (Rectified Linear Unit)
A non-saturating nonlinearity function f(x) = max(0, x) used in neurons to accelerate training time with gradient descent.
Dropout
A regularization technique that sets the output of each hidden neuron to zero with a probability (e.g., 0.5) during training to prevent co-adaptation.
Overlapping Pooling
A pooling scheme where the stride of the pooling units is smaller than the size of the neighborhood they summarize (s < z).
Top-5 Error Rate
The fraction of test images for which the correct label is not among the five labels considered most probable by the model.
Softmax
A final layer that produces a probability distribution over class labels; in this paper, it is a 1000-way layer.
Timeline
2010ILSVRC-2010 competition where the model achieved 17.0% top-5 error.
2012ILSVRC-2012 competition where a variant of the model won with a 15.3% top-5 error rate.
Mind Map
Everything is expanded by default. Use the − buttons to collapse a branch, or the controls below.
Deep CNN (AlexNet)
Architecture
5 Convolutional Layers
3 Fully-Connected Layers
ReLU Nonlinearity
Training Techniques
Multi-GPU Parallelization
Dropout Regularization
Data Augmentation
Dataset: ImageNet
Scaling Deep Learning for Image Recognition
Key metrics and innovations from the AlexNet paper
trending_down
15.3%
Top-5 Test Error (2012)
settings
60M
Model Parameters
speed
6x
Training Speedup with ReLU
memory
2
GTX 580 GPUs utilized
Combating Overfitting
Used dropout in fully-connected layers and added RGB channel intensity alterations to enhance robustness.
Efficient Architectures
Employed overlapping pooling (3x3 grid, stride 2) which reduced top-5 error by 0.3%.
Massive Scale
Trained on 1.2 million images across 1,000 classes for 5-6 days to reach convergence.
Why is the top-5 error rate a standard metric for ImageNet?
Since ImageNet has 1,000 categories, some images may naturally contain multiple objects or be ambiguous; the top-5 metric measures if the correct label is among the model's top 5 most confident guesses.
What hardware was used to train the network?
The models were trained for five to six days on two NVIDIA GTX 580 GPUs, each with 3GB of memory.
How does PCA-based data augmentation help?
It alters the intensities of RGB channels based on principal components of pixel values, capturing the property that object identity is invariant to changes in illumination color and intensity.
What was the significance of the 15.3% error rate?
It was a breakthrough because it was considerably lower than the second-best result (26.2%), demonstrating the superiority of deep learning over traditional computer vision methods at the time.
References
ImageNet Classification with Deep Convolutional Neural Networks