GCP Data Engineer: BigQuery, Dataflow & Real-Time Pipelines

Machine Learning Technologies, LLC; Tampa (FL); On-site; USD 120,000 - 160,000; Full time

Job summary

Machine Learning Technologies, LLC in Tampa, FL, is seeking an experienced GCP Data Engineer to design and maintain scalable data pipelines on Google Cloud Platform, handling large datasets and supporting analytics initiatives.

You will optimize data warehouses, develop ETL/ELT workflows, and collaborate with BI and ML teams, using BigQuery, Dataflow, Beam, Airflow, Pub/Sub and Spark.

Qualifications

• Experience with GCP data services and BigQuery.
• Strong Python and SQL skills with optimization experience.
• Experience building ETL/ELT pipelines and data quality validation.
• Familiarity with Cloud Dataflow, Apache Beam, and Dataflow pipelines.
• Experience with Cloud Composer (Airflow) for workflow orchestration.
• Knowledge of IAM, encryption, service accounts, and cloud security best practices.

Responsibilities

• Design and implement scalable batch and streaming data pipelines on GCP.
• Build ingestion frameworks loading data from APIs, databases, files, and streaming sources into BigQuery.
• Develop data transformation logic using SQL, Python, Apache Beam, and Spark where appropriate.
• Create and maintain BigQuery datasets, tables, views, and optimized analytical data models.
• Orchestrate and monitor workflows using Cloud Composer and Airflow DAGs.
• Implement real-time processing with Pub/Sub and Dataflow.
• Optimize query performance, storage usage, and cloud costs.
• Build data quality checks, reconciliation processes, and failure recovery mechanisms.
• Collaborate with data architects, analysts, BI developers, and ML teams.
• Maintain technical documentation and support production releases.

Skills

BigQuery optimization

Python programming

Advanced SQL

ETL/ELT development

Data modeling

GCP data services

CI/CD & automation

Data warehousing

Cost management on cloud

Tools

Cloud Dataflow

Apache Beam

Cloud Composer (Airflow)

Pub/Sub

Cloud Storage

Dataproc

Spark

Git

Job description

Machine Learning Technologies, LLC in Tampa, FL, is seeking an experienced GCP Data Engineer to design and maintain scalable data pipelines on Google Cloud Platform, handling large datasets and supporting analytics initiatives.

You will optimize data warehouses, develop ETL/ELT workflows, and collaborate with BI and ML teams, using BigQuery, Dataflow, Beam, Airflow, Pub/Sub and Spark.