Welcome to Troubleshooting Virtual Machine Networking Issues on GCP. To get the most from this course, you should already have some basic experience using virtual machines on Google Cloud Platform. If you don't have this experience, then you can go through one of our introductory GCP VM labs, such as starting a Linux virtual machine on Google compute engine. All right, let's get started. One relatively common issue is when you can't connect to a virtual machine instance using SSH. 

When you can't connect with SSH, then a feeling of helplessness can set in. But don't worry, there are quite a few things you can do to resolve the issue. Before we get into how to troubleshoot the Connection Failed error you see here, let's go over how to deal with the Permission Denied error. This error can occur for obvious reasons, such as not using the right flags on your SSH command when you try to connect. But there are also some potential issues that are specific to Google Cloud Platform.

To understand why, you need to know the different ways you can configure SSH connections on GCP instances. First up is OS login, which is the preferred method. It lets you control access by using IAM roles. The advantage of using IAM roles is that they're the standard way to control access to all other resources on GCP2. OS login can also manage your public SSH keys for you, and it even supports two-factor authentication. If you can't use the OS login method for some reason, then you can add public SSH keys to metadata. 

One easy way to do it is to add project-wide SSH keys by putting them in a project's metadata. Any user who has an SSH key in a project's metadata can connect to any instance in that project, with the exception of instances that specifically block project-wide SSH keys. The other way is to add SSH keys to the metadata of each instance. This gives you more fine grain control of who can access individual instances, but it takes a lot more effort to manage. So, how might these different configurations lead to a Permission Denied error? There are several different ways.

For example, OS login disables SSH keys and metadata. So, if you try to connect to an OS login-enabled VM using a key in metadata, you'll get an error. The opposite configuration also generates an error. That is, if you try to use a key stored in an OS login profile to connect to a VM that doesn't have OS login enabled, then it won't work. Another scenario is if you try to use a project-wide SSH key to connect to a VM that has project-wide keys disabled. Now let's get back to the Connection Failed error. Here's how to troubleshoot it. First, check your firewall rules. By default, your network will contain a firewall rule that allows SSH traffic. But you should make sure that the rule wasn't deleted or modified. The easiest way to check is to go into the Networking section of the Google Cloud Console and click on 'Firewall rules'. There should be a rule called default-allow-ssh that allows traffic on TCP Port 22. The source should be all zeros, meaning it will allow SSH requests from any IP address and the target should say apply to all targets. If you don't have this rule, then you can easily create it. Click 'CREATE FIREWALL RULE'.

You don't have to name it default-allow-ssh, but you probably should, just so it follows the convention of network name that is default-allow and protocol, that is SSH. Change the priority to 65534. Leave direction of traffic as Ingress and action on match as Allow. Change targets to All instances in the network. Leave the source filter as IP ranges. In the source IP ranges field, put 0.0.0.0/0. And put TCP:22 in the protocols in Ports field. Now click the 'Create' button and that's it. Let's try connecting with SSH again and see if that fixed it. Great, it did. 

If you didn't have a problem with the firewall rules but you still can't get in with SSH, then try connecting to port 22 manually and see if the SSH server responds. Use the NC command on the external IP address of your instance and then specify port 22. If you see the SSH banner, then you know that the network connection is okay and the SSH server is running. If you don't see the banner, then it could be a problem with either the network connection or the SSH server. Next, try accessing the serial console. Look at the serial port output to see if that will tell you why you can't connect using SSH. The next thing to check is if there's a problem with your account.

Try connecting with another username like this. You can put in whatever username you want and the gcloud tool will update the project's metadata to add the new user and allow SSH access. I'm going to call it new user. If that didn't work and your instance boots from a persistent disk, which is the case by default, then you can detach the persistent disk and attach it to a new instance. Of course, this will take down your existing instance. So, if the instance is serving production users properly and you don't want to cause an outage, then skip this procedure and go to the next one. First, delete the instance and be sure to use the Keep-disks option. Then create a new instance. I'm going to create it with the same name as the original one and attach the disk that we saved from the original instance. Also, add the auto-delete=no option, so the disk isn't automatically deleted when you delete this instance. Now SSH into the new instance. That worked. If your instance is serving production users properly and you don't want to cause an outage, then you can follow this procedure instead of the one I just did.

First, create an isolated network that only allows SSH connections. This is because you're going to clone your production instance and you don't want the clone to interfere with your production services. Now add a firewall rule to allow SSH connections to the network. Then create a snapshot of the boot disk. Now you can create a new disk with the snapshot you just created. Then create a new instance in the new network. Now attach the clone Disk. 

By the way, Google recommends that you don't give this instance an external IP address, but that makes it much more difficult to connect to it. Considering that the instance is sitting in a network that blocks everything except SSH requests, it shouldn't cause any disruption to your production services even though it has an external IP address. Now SSH into this instance. Although the clone disk is attached to this instance, it isn't mounted anywhere. So, you'll have to do that manually. First, create a mount point, then see what the device name of the disk is. The boot disk is disk-0, so the extra disk we attached has to be disk-1. Then mount the file system.

Now you can finally try to debug why you can't connect to the original instance using SSH. You can look through the logs for instance. That was a pretty complicated process, but that's the sort of thing you have to do if you don't want to disrupt your production service while you're debugging it. Here's a summary of what I just showed you. First, create an isolated network that doesn't allow any connections. Then add a firewall rule to allow SSH connections. Next, create a snapshot of the boot disk. After that, you can create a new disk from the snapshot. Then create a new instance in the new network and attach the clone disk to it. Finally, SSH into the instance and mount the disk so you can inspect it. If you still can't find the reason why your instance won't accept SSH connections and you're able to restart the instance at some point, then you can use a startup script to gather information.

If you're not sure what to put in the startup script, then you can use one provided by Google. The script will run the next time the instance boots. So, when you're ready, you can run gcloud compute instances reset instance-1. The script sends its output to the serial port. So, look there for the debugging info. And that's it for this lesson.