Linux Shell Scripting
The course is part of this learning path
In this course, we will explore how to set up a network of virtual machines and how to implement SSH key authentication and execute commands on remote systems. We'll look at how to install and remove software from local and remote systems. You'll learn about continue and break statement and their benefits and use cases. You learn how to automate processes on Linux through the use of cron jobs and examine running processes.
This course is part of the Linux Shell Scripting learning path. To follow along with this course, you can download all the necessary resources here.
- Learn how to create a network of virtual machines and how to configure SSH key authentication and execute commands on remote systems via SSH
- Learn how to install and remove software packages both on your local system as well as on remote systems
- Understand continue and break statements in loops and what they're used for
- Understand what cron is and how to use it to schedule the running of scripts in Linux at various intervals
- Learn how to examine running processes on a Linux system and how to determine their process IDs
- Anyone who wants to learn Linux shell scripting
- Linux system administrators, developers, or programmers
To get the most out of this course, you should have a basic understanding of the Linux command line.
In this lesson, you'll learn how to install and remove software packages both on your local system, as well as on remote systems. Plus, we'll be talking about how to control services, as well as making some network requests and testing those requests with the curl command. Now, as you can see here, I already have my terminal open and I've moved into our shellclass folder, and then also into the multinet sub-directory under there. I've ran vagrant up. And as you can see here with the vagrant status command I have three virtual machines already running. And as a matter of fact, I've already SSHed into the admin01 VM already. So I'm ready to go. Didn't want to make you wait through that entire process again this time. Anyway, we haven't explicitly covered how to install software yet in this course. I suspect the next couple of minutes are going to be a review for you. And if it is, just please bear with me. And if it isn't, then it's good to be you because you're about to learn something new. So on RPM-based Linux distributions such as Red Hat, CentOS, Oracle Linux, and Scientific Linux, you use the yum command to install software packages. The latest version of Fedora uses a yum replacement called DNF. Now, the good news is that it's really a drop-in replacement for yum. So if you're used to using yum search, then it's really easy for you just to simply type in dnf search, right? It's not a big deal. So as of Red Hat 7 and CentOS 7, et cetera, yum is still the command used to install software. Now that might change to DNF, for example, and if it does, again, you'll be covered because they work the same way. To find software installed, like I just mentioned, you can use yum, space, search followed by one or more search terms. For example, you could do this: yum search apache. What this does is looks through all the RPM package names, descriptions, and summaries for the keyword provided, which was apache in this example. You know, you can search for multiple packages by providing multiple keywords to search for. For example, you could use yum search apache, say mariadb, and php. By the way, MariaDB is a replacement for MySQL if you're familiar with that database, just in case if you were wondering. So let me just go ahead and hit Enter here on this command. And we get a lot of information that's scrolling by us on the screen. So let me just go ahead and pipe this to less. And you can see here, it says N/S, which is name and summary, matched apache, and it lists all these packages that the keyword apache matches for. And as I page down here, you'll see it searches for MariaDB. The same thing there. And then again, at the bottom here it's matched both apache and php. So that's kind of how this search works. Now, the yum search command is really useful for searching for packages that you don't know the exact name of but for which you know a related term. Now that you've found the name of the package you're looking for or a few names that might be what you need, you can find out more about those specific packages. To display information about one or more packages, run yum info followed by the package or package names that you want to know more about. For example, to look at the details of the HTTPD package you would do this: yum, space, info, space, httpd, and hit Enter. For example, my eyes go straight to the description section and it says the Apache HTTP server is a powerful, efficient, and extensible web server. So if we're looking to install the Apache web server there's a very good chance that the HTTPD package is the package that will do that for us. So once we've decided that we want to install this package we can do that and install all of its non-installed dependencies by running yum install followed by that package name. So to install the Apache web server, we would do this: sudo yum install httpd. By the way, you can search yum without root privileges. And that's why we haven't used sudo up until this point. But of course you need root privileges to actually install or remove software. So that's why we're using sudo here. Anyway, as you can see here on the screen, yum says it will install the package that you requested plus any other packages that it depends on. Yum says, is this okay? And if it is, you can continue by typing Y and hitting Enter. Now, I'm actually going to type N for no because I want to show you something here. If you want to automate the installation of software then you probably don't want to sit around answering yes to Yum's questions. You can answer yes automatically by using the dash Y option like this. Sudo yum install dash Y, that stands for yes, and then provide the package that you want to have installed. I want to take a very quick aside that will make some sense in just a bit. Now, while we have Apache installed I want to find out where it's configured to serve web pages from. Or set another way, I want to know where the document root is. I know that configuration files are typically stored in /etc. So I'm going to start looking for Apache's configuration there. So I'm gonna type cd, space, /etc, to move over into that directory. And now I'm just going to run LS with the dash L, A, R and T options. L is for a long-listing format, A is for all files. That includes "hidden" files if you will. R is for reverse, and T means sort by time. By using R and T together this means we'll get a reverse sorted list of files by time putting the most recent files last in the LS output. So let's execute this and see what happens. Here we see HTTPD very close to the bottom of our list which means it was recently added or updated. This makes sense that it would be the Apache configuration directory since we had to install the HTTPD package to install Apache. Now, I don't know what configuration file is going to contain the document root information, and honestly, I'm too lazy to find out. So I'm just going to recursively search in that directory for the thing I'm looking for. I'll use the grep command to perform the search, the dash option to grep, to make it perform a recursive search. And so I don't have to worry with capitalization whether something is capitalized or lowercase or whatever. I'll use the dash I option to ignore case. So I can do this: grep -ri documentroot. And I'm gonna look in the HTTP directory and recursively blow that since we have the dash R option. So here it lists three places in one file where the text "document root" occurs. I know that comments typically begin with a pound sign, so there are two lines that have a pound sign at the beginning, so we can ignore those. And so we're left with one line which is document root followed by /var/www/html. So that's where we can store the files we want Apache to serve up for us. Okay, sorry about that quick aside but we'll need this information in a bit. So to get back to where we were, we just installed Apache by executing sudo yum install -y httpd. Now, if we were to do it again, it wouldn't hurt anything. Yum would just tell us that it's already installed. So let's just test it out here and see what happens. It says, yep, there's nothing for me to do because you've already have that package installed, and it's the latest version. Just like you were able to install packages by name, you can remove them just as easily. To uninstall a package, just pass its name to the yum remove command. You can remove multiple packages at once by listing multiple packages on the command line. If you want to automatically answer yes, use the dash Y option here as well. So we can do this: sudo yum remove -y httpd. And hit Enter. You might have noticed the word erasing here in the output of yum. So another way to do the exact same thing is to use yum, space, erase. Yum remove and yum erase do 100% exactly the same thing. They are more or less aliases to one another, remove or erase. You could use those interchangeably with yum right now. So just use whichever one that makes sense to you or that you can remember. Some of the packages you will install are just commands or supporting files and they're not services. For example, there's a command I like to use when I'm looking at system performance called dstat. Now let's see if it's installed. So let me just type it here. Dstat, hit Enter. It says okay, command not found. So it doesn't appear that it's installed. Now, let's see if it's available as a package. So let me search for it. Yum search dstat. Okay, one package is returned named dstat. So let's get some information about it. Here it says Dstat is a versatile replacement for vmstat, iostat, netstat, and ifstat. So this does sound like the tool that I'm looking for. So let's go ahead and install it. And we can do that with sudo yum install -y dstat. So now that it's installed, I can use it if I want. It's just a command on the file system waiting to be run whenever I or someone else on the system needs it. So let me type it in, dstat, and sure enough we're getting some output from this command and I'm gonna hit Control+C to stop it. So that is one type of software that you'll be installing. However, you'll eventually install software that's meant to run all the time. This type of software is called a service. One example would be database software that constantly runs and allows applications or database clients to access the databases and the data that they provide. The database software provides the database service. Sometimes services are called servers. So you may hear something about a database service or a database server. Those are the same thing. Again, web server or web service. Those are probably the same thing. Now, getting back to our database example here, one popular database in use a lot today is MariaDB. So let's go ahead and get that installed. So let's search for it. Looks like we have several choices here. And we're looking for the service or server option. And so let's install the package that contains -server, mariadb-server. You can see that there are several dependencies and they're all being installed automatically or yum. So we didn't have to worry about sorting all that out ourselves, which is a great thing about using these awesome package managers. One way to connect to a MariaDB server is to use the MySQL command. And again, MariaDB is a drop-in replacement for MySQL or MySQL. So if you are used to using that and you install MariaDB instead, you're gonna be right at home. The commands and everything are just the same. So let me type mysql and hit Enter and see what happens. So here we see an error that states it cannot connect to the server. That means we have to start the database service and to do that, we'll use the systemctl start command. Sudo systemctl start followed by the service name which is mariadb.service. So now that that command has completed, let's go ahead and try it again. This time we don't get an error and we actually end up connecting to the database service and we're in and we can do whatever we want. We're done here. We're not really gonna do anything in the database. We're showing that you can actually connect to it. But the service, the database service, has to be running and available to do that. And of course, we started that with systemctl start. When you start a service using systemctl start it just starts the service. It doesn't do anything else. If I were to reboot the system right now the MariaDB service would not automatically start at this point. It wouldn't start at boot time. Now, to make sure that it does start at boot time, you need to enable it. Running systemctl enable mariadb.service will do the trick. So let's try that. Sudo systemctl enable mariadb.service. By the way, you can also use the shorthand version of the service name, which is just MariaDB. So we could have accomplished the same thing by doing this: sudo systemctl enable mariadb. It's already enabled so it's not giving us any extra output. So at this point, now that we've enabled the service, if we were to reboot the system, then the database would start at boot time. Now, if you want to prevent the service from starting on boot, you would use an opposite command here which is sudo systemctl disable. And I'm gonna use the shorthand version of the name mariadb instead of mariadb.service. Disabling the service just controls what happens at boot time. I don't want you to get things confused here. Enable and disable controls what happens at boot time. Now, starting and stopping controls what happens right now. So even after we've run disable, the MariaDB service is still running. And we can confirm this in a couple of ways. We could do this. Sudo systemctl status mariadb. And it says that the status is active and it's running and even gives us the PID and some other information as well. So we could use another command to check to see if it's running or active, and that is systemctl is-active, and then give the service name. Here it reports is active. Now, to stop the MariaDB service, you run systemctl stop. So we'll do this. Now, if we see if it's is active, it reports an unknown status. It's not active. If we look at the status output, it says that it's inactive. So we can see here that the stop command actually stops the service while the disable command just controls what happens at boot time. Let's go ahead and start the service. And we know how to do that. Now let's check its status. And now it says that it's active. Now, most of the times when you change the configuration associated with a service, you need to actually restart that service so that it can reread the updated configuration times. Again, a lot of services just read the configuration when they start and they retain that information in memory, and they're not constantly going back to the disk to check to see if their configuration is updated. So again, a lot of times you need to restart the service when updating configuration. Now, the easiest way to do that, to restart the service, is to simply use systemctl restart followed by the service name. And what this does is briefly stops the service and then starts it back up again. And then when it restarts, of course, then it will read any of the new configuration and act according to that new configuration. So let's say we've made an update to the MariaDB service, some of its configuration, and we want to restart it. Well, that's pretty easy. Sudo systemctl restart mariadb. So let me just highlight the workflow you'll be using when you install the service. So first, you figure out what package contains the service or the software, and then you use sudo yum install followed by that package name or service name. And then once you have the package installed you can perform any configurations or adjust any settings that the service will need. And sometimes you may or may not need to adjust any of these configurations or settings. And the next step, regardless of whether you made any changes to the default configuration or not, is to use systemctl start followed by the service name. And then the last step here is to enable that service with sudo systemctl enable followed by the service name. Now that ensures that the service will start on boot. So that's kind of the three-step process there. Yum install httpd, for example. Yum start httpd, and then yum enable httpd. Or as we've been using here, mariadb. Now let me ask you this question. How would you go about installing software on another system or on a remote system over the network? You already know how to run commands on remote systems using SSH, and installing software is nothing but running a command or a few commands, right? So you can use SSH to accomplish this. Now let's install the Dstat tool on server01 and test it out. First let me SSH over there. Server01. Type dstat and sure enough, command not found, it's not installed on server one. Let me exit out of server one and get back here to admin01. From admin01 I'm going to install dstat on server01 like this. So we'll just use SSH, give it the server name that we're connecting to, which is server01, then the command we need to install the software, which is sudo yum install -y dstat. And let me hit Enter and see what happens here. Okay, yum at the bottom there says it installed dstat and that it's complete. So it looks like everything is all right. Let's jump back on to server one and see if dstat works. So SSH server01. And run dstat here for a second or two. Okay, sure enough, looks good. Let me hit Control+C to stop out of that and then exit to get back to the admin 01 virtual machine. Now let me give you a hint here. You might want to remember how to do that because you just might need to write a shell script that installs software on remote systems soon. Now I'm not certain about that but you just might want to remember this, all right? Because this course is about shell scripting, I want to write a little script here to install the Apache web server and then put an index.html file into place on the document root. So let me go into the shared vagrant folder here and then create this file. We'll call it multinet-demo02.sh. I'm not gonna talk about this first line. You know what that is all about. Instead a quick comment here. So we know that installing software requires root or superuser privileges. So let's go ahead and put a check for that at the top of our script here. So if the user ID is not zero, that means you are not root. And let's go ahead and do an exit status of one. And we can actually send this to standard error. Now that we've tested that the script is running with root privileges we know we can go on to install the software. Let's also put an index.html file into the web server's document root. So you remember a little while ago we looked that up where that document root was. So let's do this. Let's put in "Hello World" and make that be the homepage if you will. The next step in the process is to go ahead and start Apache. And then we want to make sure that if the server ever reboots that Apache starts, so let's do this. Okay, we're all done here. Let's exit with an exit status of zero, and let's add executable permissions here, and let's give it a shot. Let's see what happens here. Okay, good, it says you need to run this with sudo or as root. Great, so let me do that. All right. It looks like it installed Apache and enabled it and everything. Let's see if it's running. Okay. Systemctl status reports that Apache is running. Let us see if it serves up the index.html file that we created and that had some of that "Hello World" text in it. Now, one way to test this would be to open up a web browser on your local system and type in the IP address of this VM. But another way to do this is to test it from the command line using the curl command. So what we do here is use curl, and then we specify a network protocol. Here we're using HTTP. And then we're going to give it a host name or an IP address, DNS name or so on. And so local host is a special name that just connects to the local system. And we'll go ahead and use this and see what happens. Okay, I wasn't expecting that. I was just expecting "Hello World" to be returned. And that's not what we got. So let's see why this is happening. Let me actually go back into the script here. Okay, I can see my typing mistake. It's right here. My keys look like they stuck here or I did a really bad job of typing. In any case, that needs to be corrected. So now that that's corrected, I'm gonna go ahead and actually execute this script again because there's nothing really in here that's gonna hurt anything if it gets executed again. So if Apache is already installed it will say it's installed. It'll create this file. And if it's started, that's fine. If it's enabled, that's fine as well. So let me go ahead and run it again and see what happens. So it says nothing to do cause it's already installed. Well, let's execute our curl command again, curl http. Now we get the information that I expected. So there's an example of some troubleshooting your shell scripts. And by the way, if we were to look here in that directory we have a file that doesn't need to be there. So let me go ahead and delete it. All right. I've cleaned up my little mess here. Now let's get back to curl really quickly. Now curl is just a tool that transfers data to or from a web server using various protocols. Here we specify the HTTP protocol, for example. Now, personally I use it mainly to make those types of requests, those web or HTTP or HTTPS requests, just like we did here. Now, curl isn't a command line web browser. It just is a simple tool that will send and/or receive data. Now, if you make an HTTP request it just returns the raw HTML from that response without doing anything to it. So let's look at another example here. Let's make a web request to a site that serves up a real site, real data, and let's see what happens here. So we'll use curl and we'll go to HTTPS. Let's visit google.com and see what happens. So once we do that, we just get a bunch of HTML back. You can see here like a closing body tag and a closing HTML tag. So curl didn't interpret any of that information. It just grabbed an information and displayed it to your screen. By the way, we can test the exit status of curl like we can for any other command. Now, let's quickly look at curl's man page to see what exit codes that it provides. So if we move down a little bit here, let's look at number seven, exit code seven, which is failed to connect to host. So that is something that we could test for. In any of these cases all these errors are obviously non-zero. So you could do just a general test, which is if the exit code is anything other than zero, then we have a problem. Or you could get really specific here and say, if the return code is seven, then that means we couldn't connect to the host. But if the return code was six, that means we couldn't resolve the host. It's a DNS or a naming issue, network name issue. Okay, so let me hit Q to get out of here. And let's stop the HTTP service here. Now let's use curl to connect to it. Hey, we get seven failed to connect to local host, and let's actually look at the return status here on the command line and sure enough, it is seven. So that's just something to keep in mind if you ever need to use curl to test a service. Another thing you can do with curl is actually talk to various web APIs. Rest APIs are fairly popular these days. By default, curl sends a get request. And if you want to use one of the other verbs or request methods, use dash capital X followed by the method. You know, put, post, delete or what have you. So you could do something like this. Curl -X delete. And let's say, you know, the URL is my.api/my/stuff thing one, and you want to delete one from stuff, for example. So that would be a way that you could use curl to interact with an API, either to send requests and/or receive those responses. Let's do a quick recap. In this lesson, you learned how to install and remove software packages both on a local as well as a remote system. You also learned how to control services with the systemctl command. Finally, you learned how to use the curl command to make network requests.
Jason is the founder of the Linux Training Academy as well as the author of "Linux for Beginners" and "Command Line Kung Fu." He has over 20 years of professional Linux experience, having worked for industry leaders such as Hewlett-Packard, Xerox, UPS, FireEye, and Amazon.com. Nothing gives him more satisfaction than knowing he has helped thousands of IT professionals level up their careers through his many books and courses.