Senior Data Engineer: AWS ETL & Data Pipelines

Jobtailor; Columbia (MD); On-site; USD 110,000 - 160,000; Full time

Job summary

Jobtailor is seeking a Senior Data Engineer in Maryland to design, develop, and maintain robust data pipelines from external feeds into our data lake and downstream zones.

You will implement production-grade ETL workflows with AWS Glue, PySpark, Lambda, and MWAA, loading data into S3, Redshift, Oracle, and other stores while ensuring data quality and traceability.

Qualifications

• Bachelor's degree in a technical field or equivalent experience.
• 5+ years of data engineering experience deploying production pipelines.
• Hands-on AWS Glue, Spark/PySpark, Lambda, and S3.
• Strong Python (PEP 8) and SQL, with relational data integration.
• Experience with JSON/XML/CSV/Parquet data formats and Airflow/MWAA.
• Proficient with relational databases and data lake formats.
• Git/GitHub, Agile methodologies, and strong communication.
• Ability to obtain Moderate Risk Public Trust clearance; US residency.

Responsibilities

• Design, build, and maintain data pipelines from external feeds to data lake landing zones.
• Develop production-grade ETL workflows with AWS Glue, PySpark, and MWAA.
• Load data into S3, relational stores, NoSQL, and knowledge bases with quality checks.
• Enforce schema, validate data, and generate ETL metadata reports.
• Create semantic-layer objects and support XML parsing for regulatory filings.
• Build reusable components for self-service ETL and CI/CD deployments.
• Provide on-call support and collaborate with cross-functional teams.

Skills

AWS Glue ETL Development

PySpark Programming

Data Pipeline Design

SQL Proficiency

Agile Methodologies

Data Quality Checks

Education

Bachelor's degree in Data Science, Computer Science, Engineering, or related field

4 additional years of related experience in lieu of a degree

Tools

Git/GitHub

Airflow/MWAA

CloudFormation

PostgreSQL

Redshift

Oracle

S3

Iceberg/Parquet/ORC

Job description

• Design, develop, ingest, and maintain well-architected data pipelines that retrieve data from external feeds (APIs, SFTP, HTTPS, FTP, web scraping, Direct Connect) and internal agency sources into the data lake landing zone and downstream curated zones.
• Develop production-grade ETL workflows using AWS Glue, PySpark, Python, Lambda, and EMR, integrated with a shared ETL common library and orchestrated via Amazon Managed Workflows for Apache Airflow (MWAA).
• Load data accurately and optimally into S3 zones (Parquet, ORC, Iceberg), relational datastores (PostgreSQL, Redshift, Oracle), NoSQL databases, and knowledge bases/vector stores, preventing duplicate loads and maintaining data integrity and traceability across all lifecycle stages.
• Implement schema enforcement, XSD validation, data quality checks, error handling, and automated SNS notifications; ensure all production jobs populate ETL Load Reports and Gap Reports through static and dynamic ETL metadata.
• Develop semantic-layer objects (tables, views, materialized views) that ensure complete data coverage, optimized query performance, and consistent application of business logic.
• Develop XML parsing/shredding logic for high-volume regulatory filings using Glue PySpark, supporting schema evolution and batch processing per program standards.
• Design pipelines with query performance in mind and support rollback, reload, and date-range reprocessing capabilities without manual intervention.
• Support self-service ETL development by other agency teams through standardized, reusable components aligned with program standards, and facilitate the transition of externally developed ETL jobs into the Data Engineering team's production support.
• Create and maintain required engineering artifacts, including business requirements, ETL design documents, mapping documents, data models, data dictionaries, deployment references, operations and maintenance guides, and test plans.
• Deploy code through automated CI/CD pipelines using CloudFormation templates, following agency release, security, and governance processes.
• Provide operational support for production jobs, including rapid identification and resolution of failed jobs and performance issues; participate in on-call/after-hours support for production outages and emergencies as part of a team rotation.
• Collaborate with Data Officers, Data Stewards, SMEs, data providers, and IV&V teams to understand requirements and deliver user-accepted solutions; engage closely with the Product Owner and cross-functional teams to provide timely updates and resolve issues.
• Leverage AI-assisted development tools to accelerate coding, optimize workflows, and enhance code quality while adhering to security and performance standards.
• Work in Agile teams; drive iterative delivery, joint problem-solving, and continuous improvement, including participation in sprint planning and program increment planning.

Requirements

• Bachelor's degree in Data Science, Computer Science, Engineering, or a related technical discipline.
• In lieu of a degree, four additional years of related experience is required
• Minimum of 5 years of related data engineering experience developing and deploying data pipelines in production.
• Hands-on experience developing ETL pipelines in AWS using Glue, Spark/PySpark, Lambda, and S3.
• Proficiency in Python (adhering to PEP 8) and strong SQL skills, with experience integrating data from relational databases.
• Experience processing structured and semi-structured data formats, including JSON, XML, CSV, Avro, and Parquet.
• Experience with workflow orchestration (Airflow/MWAA preferred).
• Experience with relational databases (PostgreSQL, Redshift, or Oracle) and lake table/file formats (Iceberg, Parquet, ORC).
• Experience with Git/GitHub version control and Agile methodologies.
• Strong attention to detail with a commitment to delivering high-quality and accurate work; excellent written and verbal communication skills.
• Ability to obtain and maintain a Moderate Risk Public Trust clearance; residing in the United States

Core Competencies

Demonstrates expertise in designing and developing data pipelines using AWS Glue, PySpark, and Python, with a strong focus on ETL workflows and data integrity. Proficient in collaborating with cross-functional teams to deliver high-quality data solutions while adhering to Agile methodologies.

Highest-signal resume keywords

• AWS Glue ETL Development
• PySpark Programming
• Data Pipeline Design
• SQL Proficiency
• Agile Methodologies

ATS Optimization Keywords

Hard Skills

• ETL Development
• Data Pipeline Architecture
• Python Programming
• SQL Skills
• Data Quality Checks
• Schema Enforcement
• XML Parsing
• Workflow Orchestration
• Data Lake Management
• Version Control (Git/GitHub)

Soft Skills

• Attention to Detail
• Written Communication
• Verbal Communication
• Collaboration
• Problem-Solving

Industry Keywords

• Data Engineering
• Data Quality
• Data Integrity
• Regulatory Filings
• Agile Development

Tools & Technologies

• Amazon S3
• Amazon EMR
• Amazon Managed Workflows for Apache Airflow (MWAA)
• CloudFormation
• Data Lake Formats (Parquet, ORC, Iceberg)