Data Architect

TechDigital Group; El Segundo (CA); On-site; USD 80,000 - 130,000; Full time

Job summary

An innovative firm is seeking a skilled Data Engineer with expertise in Google Cloud Platform and BigQuery. This role involves designing and developing robust ETL frameworks, building data pipelines, and ensuring data quality and lineage. The ideal candidate will have hands-on experience with various Google Data Products and Big Data technologies, alongside strong Python programming skills. Join a dynamic team where your contributions will drive data-driven decisions and enhance operational efficiency in a fast-paced environment. If you are passionate about data engineering and looking for an exciting opportunity, this role is perfect for you.

Qualifications

• Professional experience in Data Engineering with Google Cloud Platform.
• Expertise in building data pipelines and working with BigQuery.

Responsibilities

• Design and develop ETL ELT frameworks using BigQuery.
• Automate data loads from BigQuery using APIs or scripting.

Skills

Python Programming

Data Engineering

Google Big Query

Google Cloud Platform

Airflow

Big Data Technologies

DBT

Dev-Sec-Ops

SQL

NoSQL

Tools

Google Data Products

Spark

Hadoop

Hive

MapReduce

Pandas

Job description

Has worked in the past as an Architect.

Professional experience in Data Engineering using Google Big Query (GBQ) and Google Cloud Platform (GCP) data sets and building data pipelines.

Hands on and deep experience working with Google Data Products (e.g. BigQuery, Dataflow, Dataproc, Dataprep, Cloud Composer, Airflow, DAG etc.).

Python Programming Expert PySpark, Pandas

Experience in Airflow (Create DAG, Configure the variables in Airflow, Scheduling).

Big Data technologies and solutions (Spark, Hadoop, Hive, MapReduce) and multiple scripting and languages (YAML, Python).

Experience in DBT to create the lineage in GCP – Optional.

Worked in Dev-Sec-Ops (CICD) environment.

Design and develop the ETL ELT framework using BigQuery, Expertise in Big Query concepts like Nested Queries, Clustering, Partitioning, etc.

Experience in Data Integration, Data Transformation, Data Quality and Data Lineage tools.

Should be able to automate the data load from Big Query using APIs or scripting language.

E2E Data Engineering and Lifecycle (including non-functional requirements and operations) management.

E2E Solution Design skills – Prototyping, Usability testing and data visualization literacy.

Experience with SQL and NoSQL modern data stores.