Hybrid Data Engineer – Cloud Pipelines & Data Quality

Metropolitan Transportation Authority; New York (NY); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Metropolitan Transportation Authority is seeking a Data Engineer to design, deploy, and maintain data solutions across on-prem and cloud environments. You will build data pipelines, implement ELT/ETL processes, and collaborate with analysts, scientists, and stakeholders to meet organizational data needs.

The role emphasizes secure coding, data quality, and governance, with exposure to Azure and CI/CD practices.

Qualifications

• Bachelor’s Degree in Arts/Sciences (BA/BS) or equivalent.
• Minimum 3 years of relevant experience.
• Strong knowledge of Big Data architectures and data lakes.

Responsibilities

• Design, develop, and implement data solutions to ensure timely data availability for analysis and decision-making.
• ELT/ETL data from diverse sources including on-premises and cloud-based systems, APIs, databases, and files.
• Write secure, well designed code; monitor and troubleshoot data pipelines to minimize disruptions.
• Implement data quality checks and validation processes; collaborate with the data team to resolve issues.
• Develop CI/CD processes to ensure high availability and agility.
• Create and maintain data catalog and data lineage; ensure privacy compliance.

Skills

Big Data architectures

Python

SQL

Pyspark

Airflow

DevOps

Terraform

Ansible

Power BI

Jira

Confluence

Microsoft Azure

AWS/GCP

Education

Bachelor’s Degree in Arts/Sciences or equivalent

Tools

Terraform

Ansible

Jira

Confluence

Power BI

Azure

Job description

Metropolitan Transportation Authority is seeking a Data Engineer to design, deploy, and maintain data solutions across on-prem and cloud environments. You will build data pipelines, implement ELT/ETL processes, and collaborate with analysts, scientists, and stakeholders to meet organizational data needs.

The role emphasizes secure coding, data quality, and governance, with exposure to Azure and CI/CD practices.