GCP Data Engineer - Large-Scale Pipelines & Azure Migration

Machine Learning Technologies, LLC; Austin (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Machine Learning Technologies, LLC is seeking Data Engineers with hands-on GCP data engineering expertise to design, build, and maintain production data pipelines. The project involves migrating from Azure to GCP and optimizing data processing at scale.

You will work with BigQuery, Dataflow, Cloud Storage, and ETL/ELT processes, using Python and SQL, PySpark, and Airflow/Cloud Composer to ensure data quality and performance across the enterprise data platform.

Qualifications

• 6+ years of experience in data engineering, data integration, and data pipeline development.
• Strong hands-on GCP data engineering experience is mandatory.
• Experience with BigQuery, Dataflow, Cloud Storage, and relevant GCP data services.
• Strong expertise in Python, SQL, ETL/ELT development, and data transformations.
• Experience building scalable batch and/or streaming data pipelines.
• Experience with PySpark/Spark and data processing frameworks.
• Strong understanding of data warehousing, data modeling, data quality, and performance optimization.
• Experience with data pipeline orchestration and scheduling tools such as Airflow/Cloud Composer.
• Strong troubleshooting, problem-solving, and communication skills.

Skills

GCP data engineering

BigQuery

Dataflow

Cloud Storage

Python

SQL

ETL/ELT

Data transformations

Airflow

PySpark

Data warehousing

Data modeling

Data quality

Tools

Airflow/Cloud Composer

Terraform

CI/CD

Job description

Machine Learning Technologies, LLC is seeking Data Engineers with hands-on GCP data engineering expertise to design, build, and maintain production data pipelines. The project involves migrating from Azure to GCP and optimizing data processing at scale.

You will work with BigQuery, Dataflow, Cloud Storage, and ETL/ELT processes, using Python and SQL, PySpark, and Airflow/Cloud Composer to ensure data quality and performance across the enterprise data platform.