Specialist Data Engineer

Metropolitan Transportation Authority; New York (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

Metropolitan Transportation Authority is seeking a Data Engineer to design, deploy, and maintain data solutions across on-prem and cloud environments. You will build data pipelines, implement ELT/ETL processes, and collaborate with analysts, scientists, and stakeholders to meet organizational data needs.

The role emphasizes secure coding, data quality, and governance, with exposure to Azure and CI/CD practices.

Qualifications

• Bachelor’s Degree in Arts/Sciences (BA/BS) or equivalent.
• Minimum 3 years of relevant experience.
• Strong knowledge of Big Data architectures and data lakes.

Responsibilities

• Design, develop, and implement data solutions to ensure timely data availability for analysis and decision-making.
• ELT/ETL data from diverse sources including on-premises and cloud-based systems, APIs, databases, and files.
• Write secure, well designed code; monitor and troubleshoot data pipelines to minimize disruptions.
• Implement data quality checks and validation processes; collaborate with the data team to resolve issues.
• Develop CI/CD processes to ensure high availability and agility.
• Create and maintain data catalog and data lineage; ensure privacy compliance.

Skills

Big Data architectures

Python

SQL

Pyspark

Airflow

DevOps

Terraform

Ansible

Power BI

Jira

Confluence

Microsoft Azure

AWS/GCP

Education

Bachelor’s Degree in Arts/Sciences or equivalent

Tools

Terraform

Ansible

Jira

Confluence

Power BI

Azure

Job description

9:00 am - 5:30 pm (7.5 hours/day) or as required

This position is eligible for telewor k ing, which is currently 2 days per week . New hires are eligible to apply 30 days after their effective date of hire.

Opening :

The Metropolitan Transportation Authority is North America's largest transportation network, serving a population of 15.3 million people across a 5,000-square-mile travel area surrounding New York City, Long Island, southeastern New York State, and Connecticut. The MTA network comprises the nation’s largest bus fleet and more subway and commuter rail cars than all other U.S. transit systems combined. MTA strives to provide a safe and reliable commute, excellent customer service, and rewarding opportunities.

Summary :

This role is responsible for the design, deployment, development and maintenance of data solutions. The Data Engineer will participate in a variety of data related projects and work closely with the Data analysts, data scientists, Business users and other stakeholders to gather requirements and build data pipelines that meet the organizations' needs. The Data Engineer will deploy and support Data platforms that process and store data. They will contribute to software development methods, tools, and techniques and applies agreed standards and tools to achieve well-engineered outcomes.

Responsibilities :

• Design, develop, and implement data solutions to meet business requirements and data ingestion needs facilitating accurate and timely data availability for analysis and decision-making.
• Extract, load and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files.
• Write well designed, efficient code, that adheres to security standards.
• Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner to minimize disruptions in data processing for on premise and cloud environments.
• Implement data quality checks and validation processes to ensure accuracy and completeness of data. collaborate with the data team to resolve them.
• Writes complex queries and scripts to efficiently manipulate, transform, and process raw data.
• Creates and executes data validation processes to ensure the reliability and consistency of incoming data. Build processes to monitor data quality.
• Continuously optimize data pipelines for performance, scalability, and reliability.
• Create and maintain technical documentation. Contribute to building and maintaining data catalog and lineage.
• Design and develop CI/CD processes that ensure high availability and agility.
• Develop cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability.
• Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform.
• Stay updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization's data intelligence capabilities.
• Monitors and reports on supplier performance, customer satisfaction, adherence to security requirements and market intelligence.
• May mentor less experienced staff

Required Qualifications :

• Bachelor’s Degree in Arts/Sciences (BA/BS) . An equivalent combination of education and experience may be considered in lieu of a degree.
• Minimum 3 years of relevant experience.

Knowledge, Skills and Abilities :

• Strong knowledge of Big Data architectures, large data warehouse and Data Lake solutions.
• Experience designing and implementing Modern Data and Analytics solutions including Lakehouse architecture, Medallion architecture.
• Experience deploying and running cloud-based data solutions and infrastructure-as-code frameworks like Terraform
• Proficient in cloud deployments (MS Azure preferred) in an agile SDLC environment, leveraging modern programming languages, DevOps.
• Experience with major database platforms including Oracle, SQL Server as well as Cloud databases and NOSQL Databases.
• Strong understanding of data engineering concepts, ELT/ETL principles, and data modeling.
• Experience with data integration techniques for both structured and unstructured data.
• Solid programming skills in languages such as Python, Pyspark, SQL.
• Experience with Airflow.
• Experience in DevOps, Git Repos and CI/CD pipelines for code deployment.
• Experience in programming and infra-automation tools – Terraform, Ansible, python, etc.
• Experience with Microsoft Purview is a plus
• Strong knowledge of Microsoft Azure Cloud. AWS and GCP desirable.
• Functional knowledge of Microsoft Power BI
• Experience with Jira, Confluence.
• Dem o nstrated ability to work independently and navigate organizational ambiguity.
• Effective written and verbal communication skills
• Hands-on programming experience in a business setting.
• Proficiency in at least one software engineering methodology, including but not limited to Agile, Scrum, DevOps, Extreme Programming (XP), Kanban, Lean, Rapid Application Development (RAD).
• Experience applying a structured validation and testing method, including but not limited to: Unit Testing, Integration Testing, System Testing, Acceptance Testing, Regression Testing.
• Proficient collaboration skills to ensure design specifications are seamlessly implemented by the development team.
• Demonstrated ability to work independently and navigate organizational ambiguity.
• Effective written and verbal communication skills

Pref erred Technical Skills :

• Data Structures and Algorithms (Thorough Knowledge/Fully Proficient)
• Cloud Computing (Thorough Knowledge/Fully Proficient)

Core Competency

Proficiency Level

Competency Definition

Capable

Creating new and better ways for the organization to be successful

Customer Focus

Capable

Building strong customer relationships and delivering customer-centric solutions

Values Diversity

Adept

Recognizing the value that different perspectives and cultures bring to an organization

Communicates Effectively

Adept

Developing and delivering multi-mode communications that convey a clear understanding of the unique needs of different audiences

Tech Savvy

Capable

Anticipating and adopting innovations in business-building digital and technology applications

Technical Skills

Capable

Specialized knowledge and expertise on tools, programs, domains, platforms, and products used for specific tasks

Other Information :

Pursuant to the New York State Public Officers Law & the MTA Code of Ethics, all employees who hold a policymaking position must file an Annual Statement of Financial Disclosure (FDS) with the NYS Commission on Ethics and Lobbying in Government (the “Commission”).

MTA and its subsidiary and affiliated agencies are Equal Opportunity Employers, including with respect to veteran status and individuals with disabilities.

The MTA encourages qualified applicants from diverse backgrounds, experiences, and abilities, including military service members, to apply.