In this next section, I will define what toil is and explain how it can be mitigated using automation. Earlier, I had mentioned that the third goal of DevOps is to leverage tooling and automation. A lot of traditional operations work was manual, repetitive and labor intensive. Common examples would include things like resetting passwords, responding to alerts, rolling out patches, restarting servers, and copying and pasting commands from a playbook. S.R.E calls this type of work toil.

Toil occurs every time an operator needs to manually touch a system during normal operations. Keep in mind that toil is not a synonym for boring or frustrating. Filling out an expense report may not be fun but that does not make it toil. Instead, toil is work that is tied to running a production service and tends to be manual repetitive, automatable, tactical, and devoid of long-term value. The amount of toil increases linearly as the service grows and if ignored can grow out of control until your entire team is consumed by it. Too much toil leads to career stagnation, boredom and burnout.

You can group S.R.E activities into four main categories: software engineering, systems engineering, overhead, and toil. Software engineer includes things like writing automation scripts, creating tools, or modifying infrastructure code to make it more robust. Systems engineering includes things like installing updates, server configuration, or load balancer setup. Overhead is administrative work that's not directly tied to running a service and includes things like conducting interviews, attending meetings and completing peer reviews. Toil is the work directly tied to running a service and it's repetitive, manual, et cetera.

The S.R.E discipline aims to reduce toil through automation. S.R.E is try to identify repeatable tasks and write programs to reproduce the work. This means creating things like scheduled jobs instead of manually running scripts, automated monitoring tools instead of manually monitoring and rebooting unresponsive servers, continuous integration and continuous deployment pipelines instead of manually testing and deploying new code. And auto-scaling infrastructure instead of manually provisioning new hardware.

While automation is extremely helpful, not every task is worth automating. But if a repetitive task can be automated, it probably should be. And once it has, you've successfully freed up more resources for future development efforts. Ideally, S.R.Es will spend about half their time or less on toil and the other half on reducing toil for themselves and others.