Lecture

Bias-Variance Tradeoff

Bias-Variance Tradeoff - Breaking down the learning performance into competing quantities. The learning curves.


Course Lectures
  • The Learning Problem
    Yaser Abu-Mostafa

    The Learning Problem - Introduction; supervised, unsupervised, and reinforcement learning. Components of the learning problem.

  • Error and Noise
    Yaser Abu-Mostafa

    Error and Noise - The principled choice of error measures. What happens when the target we want to learn is noisy.

  • Training Versus Testing
    Yaser Abu-Mostafa

    Training versus Testing - The difference between training and testing in mathematical terms. What makes a learning model able to generalize?

  • Theory of Generalization
    Yaser Abu-Mostafa

    Theory of Generalization - How an infinite model can learn from a finite sample. The most important theoretical result in machine learning.

  • The VC Dimension
    Yaser Abu-Mostafa

    The VC Dimension - A measure of what it takes a model to learn. Relationship to the number of parameters and degrees of freedom.

  • Bias-Variance Tradeoff
    Yaser Abu-Mostafa

    Bias-Variance Tradeoff - Breaking down the learning performance into competing quantities. The learning curves.

  • The Linear Model II
    Yaser Abu-Mostafa

    The Linear Model II - More about linear models. Logistic regression, maximum likelihood, and gradient descent.

  • Neural Networks
    Yaser Abu-Mostafa

    Neural Networks - A biologically inspired model. The efficient backpropagation learning algorithm. Hidden layers.

  • Overfitting
    Yaser Abu-Mostafa

    Overfitting - Fitting the data too well; fitting the noise. Deterministic noise versus stochastic noise.

  • Regularization
    Yaser Abu-Mostafa

    Regularization - Putting the brakes on fitting the noise. Hard and soft constraints. Augmented error and weight decay.

  • Validation
    Yaser Abu-Mostafa

    Validation - Taking a peek out of sample. Model selection and data contamination. Cross validation.

  • Support Vector Machines
    Yaser Abu-Mostafa

    Support Vector Machines - One of the most successful learning algorithms; getting a complex model at the price of a simple one.

  • Kernel Methods
    Yaser Abu-Mostafa

    Kernel Methods - Extending SVM to infinite-dimensional spaces using the kernel trick, and to non-separable data using soft margins.

  • Radial Basis Functions
    Yaser Abu-Mostafa

    Radial Basis Functions - An important learning model that connects several machine learning models and techniques.

  • Three Learning Principles
    Yaser Abu-Mostafa

    Three Learning Principles - Major pitfalls for machine learning practitioners; Occam's razor, sampling bias, and data snooping.

  • Epilogue
    Yaser Abu-Mostafa

    Epilogue - The map of machine learning. Brief views of Bayesian learning and aggregation methods.