Hello, and welcome to this video on hyperparameters. In this video, we will talk about hyperparameters, and we will introduce hyperparameter optimization. A deep learning model contains many hyperparameters at various levels. Let's look at them in detail. When we define a network, we can choose the number of layers, the number of nodes at each layer, the layer type, and the activation function for each layer. These already are many hyperparameters. Also, we can add regularization techniques and initializations to the network, and we have many choices for each of these. If we use dropout layer, we also have additional hyperparameters like the probability of dropping a node. If we perform data augmentation, we can choose which transformations to use, and those are also choices that can be treated as hyperparameters. Finally, we have to choose the optimizer. And for the optimizer, we have to choose the learning rate and possibly other parameters that are involved in that specific optimizer. 

Also, we need to decide which batch size we are going to pass to our training algorithm. Since we have all these possibilities, the question is how do we find the best combination of hyperparameters? Turns out that choosing the correct hyperparameters is very important in training large networks. So, we conduct what's called an experiment. An experiment sets the hyperparameters, trains for a certain number of epochs, and then checks the scores on the training and the test set. What we normally do is to conduct many experiments in parallel with a master process called coordinating the choice of parameters, and worker processes that conduct the experiments, and report back the results to the master. The master can choose several strategies to decide which hyperparameter combination to try next. Here are the three most common ones: random search, grid search, and Bayesian optimization. Let's start with grid search. Grid search works by assigning ranges to each parameter and then sampling regularly within the assigned range.

 A better way to sample within given ranges is to do random samples because this will give a better description of the probability distribution of the relevant parameters with respect to the irrelevant parameters. Finally, a more intelligent way to search is to use Bayesian optimization. Bayesian optimization works by assuming a prior probability distribution for the score over the hyperparameter space and then adopting such distribution each time a new score is obtained from a combination of hyperparameters. This sounds complicated, but luckily for us there are several types of packages implementing this search strategy, so we will use one of them in the exercise. In conclusion, in this video, we've introduced how to choose hyperparameters to tune a network, and we've talked about three different strategies to do that choice: grid search, random search, and Bayesian optimization. Thank you for watching, and see you in the next video.