The course is part of this learning path
This course describes 6 different positions in the cloud computing industry, all of which are in high demand as established companies move to cloud technologies, companies expand IT departments, and new companies form. There’s great opportunity for fulfilling work with great pay. Overall it’s a great time to be in the cloud computing industry (which is rapidly becoming the IT industry).
Hello, and welcome back to the Careers in Cloud Computing course. I'm Adam Hawkins, and I'm your instructor for this lesson on the data engineer position. I'll cover the roles and responsibilities, technical skills, general salary range, and relevant certifications or trainings.
Data engineers are the gateway to business insight. They're responsible for building the data collection and analysis tools stakeholders use to understand the business. Don't expect to find data engineers in small teams. They're commonly present in growing teams with enough bandwidth to fill this position. So it's no surprise that data engineers focus on data. This requires different skills in a few different core areas. Programming skills are a must since data engineers need to communicate with a variety of different databases and software systems. Competence with common RDMS, the relational database management systems, and NoSQL data stores are a must. Strong SQL, or the structured query language skills are a must to answer stakeholder questions and provide business insight.
You may even encounter Kafka as a central player in data pipelines. Those skills cover interfacing with source data. Now it needs to go somewhere. This is where a host of skills come into the picture, like infrastructure engineering, using different databases designed for certain use cases, ETL, that's your extract, transform, load job, Hadoop, Storm, Spark, and even the R programming language. Data processing tends to happen in the cloud. So experience with a cloud provider and their specific data engineering or big data offerings will give you a boost as well.
Your experience level and technical skills have a huge impact on your end salary. There are entry, mid, and senior-level positions available. Motivated people can land entry-level positions through self-guided learning, formal learning, and a lot of hands-on practice. Now there are some certifications available in this area. The Google Cloud Platform Data Engineer is currently in beta. The general-purpose solution engineer certification helps you leverage the underlying cloud provider to build a data pipeline. I also recommend the "Designing Data Intensive Applications" book by Martin Kleppmann for existing tech workers looking to move into this position. It covers the high-level ideas, technical implementations, and organizational practices. Truth be told, it's a great read for anyone already in IT.
A data engineering job may be a great entry into IT or a great change of pace for an existing IT worker. You'll find it rewarding if you like understanding data, working with trends, answering questions, and discovering new insights. Now just like the last lesson, I prepared some questions to ask existing data engineers, or if you land a job interview, questions to ask to better understand the position. All right, that's enough for data engineer. The next lesson focuses on the system administrator.
Adam is backend/service engineer turned deployment and infrastructure engineer. His passion is building rock solid services and equally powerful deployment pipelines. He has been working with Docker for years and leads the SRE team at Saltside. Outside of work he's a traveller, beach bum, and trance addict.