GCP Data Lead

TechDigital Group; El Segundo (CA); On-site; USD 100,000 - 140,000; Full time

Job summary

A leading company in El Segundo is seeking a Data Engineer/Architect with expertise in Google Cloud Platform and BigQuery. The ideal candidate will have strong communication skills and hands-on experience in building data pipelines and managing the data engineering lifecycle.

Qualifications

• Prior experience in a data lead or architect role.
• Hands-on experience with Google Data Products.

Responsibilities

• Build data pipelines using Google BigQuery and GCP.
• Design and develop ETL/ELT frameworks.
• Manage end-to-end data engineering lifecycle.

Skills

Communication

Python

Data Engineering

Big Data Technologies

Tools

Google BigQuery

Google Cloud Platform

PySpark

Pandas

Airflow

Job description

Job Description:

The ideal candidate should possess excellent communication skills and have prior experience in a data lead or architect role, including hands-on work as an architect.

Key responsibilities include:

1. Professional experience in Data Engineering using Google BigQuery (GBQ) and Google Cloud Platform (GCP), focusing on building data pipelines.
2. Deep, hands-on experience with Google Data Products such as BigQuery, Dataflow, Dataproc, Dataprep, Cloud Composer, Airflow, and DAGs.
3. Expertise in Python programming, with proficiency in PySpark and Pandas.
4. Experience in Airflow, including creating DAGs, configuring variables, and scheduling.
5. Knowledge of Big Data technologies and solutions like Spark, Hadoop, Hive, MapReduce, and scripting languages such as YAML and Python.
6. Optional: Experience with DBT for creating lineage in GCP.
7. Experience working in a DevSecOps environment, particularly with CI/CD pipelines.
8. Design and development of ETL/ELT frameworks using BigQuery, with a strong understanding of BigQuery features like nested queries, clustering, and partitioning.
9. Experience with data integration, transformation, quality, and lineage tools.
10. Ability to automate data loads from BigQuery using APIs or scripting languages.
11. Managing end-to-end data engineering lifecycle, including non-functional requirements and operations.
12. Solution design skills, including prototyping, usability testing, and data visualization literacy.
13. Proficiency with SQL and NoSQL modern data stores.