In this lesson, you'll learn how to create a small network of virtual machines that will simulate a company network. You'll also learn how to configure SSH key authentication and execute commands on remote systems via SSH. What we're going to do here is actually create three virtual machines. We're going to name the first one, admin01. The next one server01 and the third one, server02 and I may call those server one and server two to be brief. This type of setup is very common where you'll have one server acting as your main administration server that will act as the base of operations, if you will. And then several other servers on your network that are used to do the work for the business, they could be web servers, database servers, or what have you. So here I'm on my local system with my terminal open. I'm gonna go to shellclass and I'm gonna create a new directory for this project. I'm gonna create a directory called, multinet. So we're gonna be using multiple machines here on this miniature network. So I'm just going to call this multinet, change directory in there, and let's go ahead and initialize our Vagrant project, vagrant init and then supply a box name. Now I just need to edit the Vagrantfile and make some changes. I'm going to add some configuration right here below the config.vm box line and what I'm gonna do is say config.vm.define and we're going to define our first server of admin01 and put that same name here in pipes. Give this an IP address. And then close our statement for this particular host, so this will create one virtual machine. Now let's go ahead and create a stanza for our next virtual machine. We'll call this one server01. Give it the host name of server01. And then we're going to pretty much duplicate this, but make sure we change the ones to twos here and also make sure we change this IP address here. Okay, so now we have these three stanzas for our three different servers, we have admin01, server01 and server02. So now what we're going to do is run vagrant up. Now, when you're working with a Vagrant multi machine set up like this, when you run a command, it runs that command against all the systems, unless you specify which one of the virtual machines that you're working with. So if we run vagrant up, it's going to bring up all three of those systems. If we run vagrant up admin01, it's only going to bring up that system, for example. So we'll go ahead and run vagrant up and let Vagrant create all three of those systems. By the way, this is going to take an extra few minutes since we're creating three servers here at once. Okay, now that the vagrant up command has been executed, we should have three running virtual machines and we can check that with vagrant status. If you get an error after you run vagrant up or if you run vagrant status and not all three of your virtual machines are running, you need to check the contents of your Vagrantfile. And let me just jump in the Vagrantfile really quick and talk about a couple of things that I've seen in the past, some little gotchas, if you will and how to get around those. So let's look at our Vagrantfile here. And one thing I've seen happen is, for example, on the IP address line, I've seen people accidentally forget the closing double quote. So that is a common mistake. I see people miss a comma here, that is a mistake I've seen before. I've also seen some students do this. They'll use capital IP instead of lowercase ip. So every little thing you see here, every little bit of syntax, it's not like salt and pepper or some seasoning, it is exact and specific. It has to be exactly this way. It's not there for a random purpose. Also, another thing I've seen here, when someone gets to this third stanza, maybe they did like I did and they copied the first one, but they forget to change this. So they have this situation where they're really trying to define server02, but in the code block here, they're referencing something that does not correspond to that. So it needs to be the same here as well. And I've also seen this happen, which is, people leave off this first bit of information here, just use vm.hostname and so on like that and so that is also not gonna work. So again, if you run a vagrant status and not all three of your machines are up or the vagrant up command didn't complete successfully, these are some things that you can check. So I'm gonna jump out of here right now and now what I'm gonna do is run vagrant ssh. Now in the past, we've just been working with one system and when we run vagrant ssh, we get connected to that one virtual machine. Well, since we have three virtual machines, if we run vagrant ssh, Vagrant doesn't know which one we're trying to connect to, so we have to be very specific and tell it which machine we're going to connect to. And so what we're gonna do is connect here to admin01, our administration server and start working there. So I'm just gonna type in vagrant ssh admin01. Now that we're in this virtual machine, I'm gonna see if I can ping those other virtual machines on the network here, just to see if they're responding on our network. So I'm gonna do ping - C3, that means send a count of three packets to 10.9.8.11. Sure enough we're getting responses and that worked. I'm gonna do it to .12. Okay, three packets transmitted, no packet loss. That looks good. So this particular admin virtual machine can communicate over the network to server01 and server02. Now, when attempting to resolve a name to an IP address, for example, if we want to ping server01, what happens is that the /etc/hosts file is checked first by default and if no entry is found there, then DNS is queried. Since the main point of this class is Shell Scripting, we're not gonna spend the time to set up a DNS server for our three little virtual machines here. We're just gonna add our two servers to the /etc/hosts file so we can talk to them by name. Now, if we look at the /etc/hosts file, you can see that the format of the file is an IP address, followed by a name, and then any other additional names that you want to access those systems by. Since I'm a normal user, I'm going to actually use a command called tee in conjunction with sudo to append some lines to /etc/hosts. So let's look at the tee command really quick. And the tee command reads some standard input and writes to standard output in files. It pretty much passes anything that it receives out the other direction and so what's useful about this is the -a option, which allows you to append to existing files instead of overwriting. So how this comes in useful in our situation, is that we're going to append to an existing file of /etc/hosts and so we're gonna use the -a option. Now, if you don't append to a file, it just overwrites the file, it clobbers it, if you will. So the reason why this is important is, again, we're a normal user and what we want to do is do something like this. We're gonna echo this 10.9.8.11 server01, 'cause that's the IP address of server01. And we want to send that to the bottom of the /etc/hosts file. So what happens here, is when we execute the command tee, sends that to standard output and it also appends it to the /etc/hosts file. So let's do the same thing again here with our server02. Now, if we cat /etc/hosts, we can see those two entries are at the bottom of the /etc/hosts file. If you're trying to append some data to a file, you may think, "Oh, I can just run sudo echo." But it may not do what you expect it to do. So let's just try it out really quick. I'll do sudo echo test and then try to append that to /etc/hosts. And it says /etc/hosts permission denied and I get people saying, "Hey I ran that with root permissions. Why can't I update /etc/hosts?" Well what happens is, you're actually executing the echo command as root with sudo there, but the redirection takes place as your normal user and that is why you cannot append to a file, so sure you're running the echo command as a root, but you're not doing the redirection as root. So that's why you can use sudo in the way we did there, pipe that output to sudo tee -a and then that way you can do things like append data to files on the command line without an editor and also without switching to the root user. Okay, so that's maybe an interesting aside, maybe it was totally boring to you, but hopefully you got a little something out of that. Okay, so let's move on. So now we have entries in /etc/hosts file. So in theory, that means we can communicate to those systems by name. So we can do this ping -c 3 server01 and you can see 10.9.8.11 is the IP address that it's communicating with and of course, that is what is in the /etc/hosts file and that does correspond to server01 and we can do the same thing here. Let's just send one packet to server01 and sure enough, it sends it to 10.9.8.12. As I said earlier, we're going to be using this virtual machine as our main administration server. We'll be running commands from here that will execute on the other virtual machines on the network. So I want to SSH into those other two systems without a password and the first thing we need to do is create an SSH key pair if we would like to do that. So we'll just run ssh-keygen to generate a key and we're just going to accept the default here. Sure, that's a fine place for the key. We're not going to use a passphrase and what this will allow us to do, is a SSH without being prompted for a password every time we want to connect to a server. So we'll just hit ENTER a couple of times and now our key is created. So the next step is to put the public key on the remote system and there's a handy command that will help you do that and it's called ssh-copy-id. So we're going to ssh-copy-id to server01. Here we're connecting as the Vagrant user over SSH to server01 and it's saying, "Hey, here's the fingerprint of that host. Do you accept it and do you think it's where you're trying to go?" We, in theory, we could check this key on the other side and confirm, but we're on a test system. I'm sure this is gonna be safe, so we'll just hit yes here. So here it's prompting us for Vagrant's password and the password for Vagrant is vagrant. Okay, it says it added a key over there, so let's check it out, ssh server01 and you can see that our prompt changed to server01. So we're on the other VM over SSH. So now I'm just gonna exit out with exit. And if I wanted to, for example, run a command over there on server01, I could do this, ssh server01 and then specify the command. So let's run the host name command over on server01. So what happens is, we connect to server01, then execute the command and then we're disconnected and we're back to our initial system. So it's like we never connected over there, but we really did, it just wasn't an interactive login. So we need to do the same thing with server02. So ssh-copy-id server02. Accept the key and our password and we should be good to go. Let's just test it really quick, ssh server02. All right, we get the server a to prompt, we're over on that virtual machine and I'll just type exit here. So now this allows us to do things like this. Let's create a little for loop right here on the command line. So I'm gonna do for N in 1 2, do ssh to server0${N} and then run a command over there, hostname and done. So sure enough, what happens in this little simple for loop, is the first time around, N is assigned the value of one and of course server0${N} evaluates to server01 and then, so that's when we ssh to server01, run the host name command. Then the next time around the loop, N is set to two and we ssh to server0${N}, which is actually server02, execute the host name and then our loop is complete. Again, like I demonstrated above, you can just SSH into any server and run a command without having to start an interactive shell if you supply the command on the SSH command line there. That command you supply gets executed on the remote system and then you disconnect from the remote system and the output is displayed. Okay, so this allows us to do some things like this. Let's change into our shared folder of Vagrant and create a file here called servers. Notice I used a single redirect operator that created a new file with the first line there, if there was a service while that existed, it would have been overwritten and the second time I used double greater than signs, which appends, so I added server two to the servers file. So let's check the contents of that really quickly. Sure enough, that file link contains two lines, server01 and server02. So then we can do some things like this, for SERVER in $ , and so remember back, that the dollar sign, parenthesis, command, closing parenthesis syntax means, take the output of that command in those parentheses and return it or substitute the output of that command for the command itself here. So what's gonna happen is, the statement is really going to read for SERVER and server01, server02. So then we'll do this. We'll do ssh ${SERVER}, run the host name command again. We'll run a second command on that server and we'll just do an uptime. So first we see the host name output for server01, then we see the uptime output for server01, the loop repeats and we get the same output for server02. And by the way, if you wanna make sure that everything you specify after the host and an SSH command gets executed on the remote host, put it in quotation marks. So if we were to do this ssh server01 hostname ; hostname, what's gonna happen is, the semi-colon is going to be interpreted as the command separator, which it is, and the first host name is going to get executed on server01 via ssh and then it's going to terminate that command, because the colon is a command separator. Then the second host name on this command line will actually execute on the local host. So let's hit ENTER to prove that here. So sure enough, ssh server01 hostname evaluates to server01 and then hostname command itself returns admin01, because it gets executed on the local host. So if you wanna make sure that everything that occurs on that command line happens on the remote system, you need to put it in quotes. So let's do this. Here what happens is, the hostname ; hostname command or command group, everything in quotes there, gets executed on the remote system, on server01 in this particular example. So whether you're doing semi-colons or pipes or whatever, if you wanna make that happen on the remote system, just enclose it in quotes. Of course, normal expansion rules apply. So if you want variables to be expanded, you need to use double quotes like this. So let's say we're gonna use variables for commands. So we'll have command one variable be hostname and the command to variable be uptime. Now, what we can do is this, ssh server01 and then use double quotes, because we want CMD1 to be expanded to hostname and CMD2 to be expanded to uptime. The command to view the running processes or the process table is ps and a couple of options that I end up using often are ps -ef, which shows every process, the -e and -f represents the full listing. So it shows a full listing of every process. So let's take the output of the ps command and pipe it to head. So let's do this, ssh server01 'ps -ef, pipe that to head -3. I only look at the first couple of processes here and hit ENTER. So what happened with that command is ps -ef pipe head -3, all was executed on server01. Now we'll get the same output if we do this, ps -ef head -3, but the difference is this time, is that the head command is actually executed on this server and not the remote server. In this instance, there isn't a difference, but again, it's something to be aware of. Another thing to keep in mind is that SSH will exit with the exit status of the remote command that was executed or with 255 if an error occurred with SSH itself and of course, how do I know this? Well, I read the man page. So let me pull it up for you really quick. I'll do, man ssh and I'm looking for exit status here. So here it is, SSH exits with the exit status of the remote command or 255 if an error occurred. So let me get out of here with Q. So let's do something like this. Let's make SHH error out by giving it a server that doesn't exist. Here it says it can not resolve hostname server03 and if we check the exit status of the SSH command, we get 255. So if you're writing a script, you can check for exit status 255 to distinguish an SSH error from an error that's coming from a remote command that was just passed on by SSH. So if we do this ssh server02 that exists, run uptime and then check the exit status, we get a zero, which was actually the exit status returned by the uptime command. Okay, here's yet another important little thing to know when you're executing these remote commands with SSH. It's actually more accurate to say that SSH exits with the exit status of the last command executed remotely or 255, if an error occurred with SSH. So let me give you a couple of examples. So there is a command called true that always returns true or an exit status of zero and there is also a counterpart to that, which is called false, which always returns one, which is a non-success or a false here on the command line. So if we do this, ssh server01 will execute false and we'll pipe that to true, which doesn't do anything practically, but it's just to show us about these exit statuses. So if we do that and then we do this, echo $?, we get a return value of zero and that's because the last command that was executed true, return to zero exit status. So let's do this, ssh server01 true false and then when we look at that exit status, we get an exit status of one, because the false command returned that exit status of one. Let's say you want any non-zero exit statuses to be returned if you're using a pipe on the remote system. So the way to accomplish that is to use this. So when we check the exit status here, we get an exit status of one, because that occurred during the pipeline. Now let's look at this, let's look at the help for set, pipe that to less, 'cause there's a lot of information here. Just gonna go down to pipe here. So if we turn on this pipefail option, the return value of a pipeline is the exit status of the last command to exit with a non-zero exit status, or zero if no command exited with a non-zero exit status. So this is something to keep in mind if you wanna make sure that every command in a pipeline was successful when using SSH. Okay, enough about executing commands on remote systems. Let's write a simple script that checks to see if our servers are reachable via ping or not. So I'm gonna name this multinet-demo01. It goes without saying, we start with a shebang, add a little comment to our script here. So we're going to use a file that contains a list of our hosts here and we just created that command line, so we'll just reuse that. And let's put a check in here to make sure that it exists before we try to loop through it. In computer speak, this says, if not exist server file, then do something. Another way to say that is, if the file doesn't exist, then do something and the thing we're going to do here is say, cannot open ${SERVER_FILE} for whatever reason, and we are going to exit. Just gonna go back here and actually put this in quotation marks. So the next step here is just a loop through that file, so we'll do a for loop. This time, let's use a count of one, I'm kind of impatient. If it can ping in one, we'll consider it up and if we don't get a response with one packet, we'll be rather impatient and just assume it's down. Ping generates some output, as we know, I'm just gonna send all this output to /dev/null, because I really don't care about the output, what I care about is really the return status. I wanna know if that command succeeded or failed and of course, you know how to check that with this. If the status is not zero, then something went wrong. So we're just going to consider the server down. Now in the real world, if you can't ping a server that doesn't necessarily mean it's down. It could mean that it's using a firewall that's preventing you from pinging it. It could mean a lot of different things. It doesn't necessarily mean that server's down, but this simplistic example, we're going to assume that it's down or at least that's what we're going to report anyway. Okay, that will finish up our for loop. By the way, I checked the exit status and said not equal zero, then do down, you could've done the exact opposite. You could've said if the exit status equaled zero, then the server was up, else it was down, so you can do that anyway that makes sense in your own mind. That's what I've chosen to do, so I'm going to save my changes. Add some execute permissions on this, and then run this script. So it says server01 is up and server02 is up. Now let's stop one of the servers, make it go down and not be available over the network and then try our script again. There's a few different ways to do this, but I'm just going to exit out of admin01. Now I'm back on my local machine, on my physical computer, I'm not in a virtual machine here and then what I'm going to do, is just halt the server02 here, and by the way, if you just run vagrant halt, it's going to halt all the systems here that are defined, not just server02, so I had to specify that. So now I'm going to SSH back into admin01. Go to the Vagrant folder and execute our demo script here. So server01 came up right away. It's waiting on server02 and the ping is going to time out and get a non-zero exit status and our script reports that server02 is down. I have just a couple of notes here before we wrap things up. First, this relationship that we set up between the systems with the SSH authentication, it's a one-way relationship. We can SSH into server01 and server02 from admin01, but not the other way around. You could set it up that way, but it's just a common practice to have these one-way relationships, so that you can designate an administration server to do all your admin work from. Of course you can set things up to go the other way, but you'll have to do all the steps we did, like add host centuries, generate keys, copy the public keys to where you want to log into and so on. Also, I want to point out a critical distinction when using sudo. Let's look at this command, ssh server01 sudo id. The SSH command is executed as the current user and in our case, that current user is Vagrant. We are then connected to server01 as the Vagrant user and if we execute the command sudo id, then it's being executed as the Vagrant user. In this particular example, the command sudo id is executed as the Vagrant user on the server01 virtual machine. Of course, using sudo gives us root privileges and so our ID that we get back, is the root ID here. Now, if we do this command, sudo ssh server01 id, if we were to execute this particular command, it would execute SSH as the root user on our local system and we are running sudo ssh locally, then we're connecting to server01 as the root user and then executing the uptime command on the remote system. So let me hit ENTER here and see what happens. So here, it's asking us about, do you trust the server's key? And we can say yes. And then you may notice here at the bottom of your screen, we have route@server01. Again, we're executing sudo ssh on the local system, which makes us root first. Then we're connecting to server01, so it's as if we're running ssh server01 as the root user. So here, if we want to continue this process, we need to enter the root password for server01, which happens to be vagrant. So we get the same output as we did running the command the other way, but I just wanna make sure that you get the distinction. Running sudo ssh means we're connecting as root to server01 and to do that, we have to give root's password for server01. If you wanna do this without a password, then you have to generate SSH keys for the root user and copy the public key to the destination servers, like we did before with the Vagrant user. Now some Linux systems are configured such that root log-ins aren't allowed, so you have to connect as a normal user and then run sudo inside the virtual machine, instead of connecting as root. Anyway, I wanted to point out this very subtle, but very important difference. And with that, that brings us to the end of this lesson and now we have a private network of three virtual machines configured and running. If you followed along and built your own private network with me here today, that's great, and if not, well, that's fine too, because you'll get a chance to do that in the exercise that follows.