Senior AWS Data Engineer – PySpark, Glue & Redshift

Delta Computer Consulting; Los Angeles (CA); On-site; USD 112,000 - 121,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

Life Insurance

401k

Continuing & On-Demand Education

Job summary

Delta Computer Consulting is seeking a Senior AWS Data Engineer in Torrance, CA, onsite with hybrid options. You will design, build, and optimize scalable cloud data pipelines using AWS Glue, EMR, Redshift, and MWAA to support data warehousing, analytics, and BI initiatives.

You will work with Python/PySpark, Spark, and CI/CD practices, ensuring data quality, security, and performance while collaborating with analysts and business stakeholders.

Qualifications

• 7+ years in data engineering, data warehousing, ETL/ELT
• 5+ years AWS experience: S3, EMR, Glue, Athena, Redshift, Airflow
• 5+ years programming with Python, PySpark, or Scala
• Strong Spark, Hadoop & distributed data processing expertise
• 2+ years CI/CD/DevOps for data pipelines
• 2+ years AWS IAM and Lake Formation governance experience
• Experience with Informatica IDMC/Cloud
• Experience applying Generative AI/Agentic AI to data workflows
• Solid data modeling, SQL, and data quality monitoring
• Strong cloud security, governance, and regulatory understanding

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines with AWS Glue, EMR, Lambda, Redshift, MWAA
• Build large-scale data processing solutions using Python, PySpark, Spark
• Develop data integration workflows for data warehouses and lakes
• Work with S3, Redshift, Athena, RDS, Redshift Spectrum and related services
• Implement data pipeline orchestration, scheduling, monitoring, error handling
• Optimize Redshift queries, performance, and cost efficiency
• Ensure data quality checks, validation, cleansing, and monitoring
• Develop secure data access using IAM, Lake Formation, roles and policies
• Support CI/CD deployments of data engineering solutions
• Collaborate with analysts and stakeholders to translate requirements into pipelines
• Maintain documentation and troubleshoot production issues

Skills

AWS data engineering

Python/PySpark

SQL

ETL/ELT design

Distributed data processing

Data modeling

CI/CD/DevOps

IAM & Lake Formation

Informatica IDMC/Cloud

Generative AI use cases

Tools

AWS Glue

AWS EMR

S3, Redshift, Athena

MWAA (Airflow)

Informatica IDMC/Cloud

Terraform / Jenkins / GitHub Actions / CodePipeline

Airflow

Job description

Work Type: Onsite/Hybrid schedule not specified

Pay Range: $81.00 – $88.00 per hour (based on experience & internal equity)

Benefits: Health, Dental, Vision, Life Insurance, 401k, Continuing & On-Demand Education

Must be willing to work in Torrance, CA

Sponsorship is available for this role

Position Summary

We are seeking an experienced Senior AWS Data Engineer to design, develop, and optimize scalable cloud-based data pipelines supporting enterprise data integration, warehousing, and analytics initiatives.

This role requires strong hands-on experience with AWS Glue, EMR, S3, Redshift, Athena, MWAA (Airflow), Python, PySpark, and Apache Spark, along with expertise in ETL/ELT development, distributed data processing, and cloud data architecture.

The ideal candidate has 7+ years of data engineering experience, including extensive AWS data engineering expertise, strong programming skills, and experience implementing secure, reliable, high-performance data pipelines.

Candidates should also have experience with Informatica IDMC/Informatica Cloud, AWS IAM, Lake Formation, CI/CD, and Generative AI or Agentic AI use cases in data engineering.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using AWS Glue, EMR, Lambda, Redshift, and MWAA (Apache Airflow).
• Build and optimize large-scale data processing solutions using Python, PySpark, Apache Spark, and distributed computing frameworks.
• Develop data integration workflows that efficiently extract, transform, and load data into enterprise data warehouses and data lakes.
• Work with Amazon S3, Redshift, Athena, RDS, Redshift Spectrum, and other AWS data services.
• Implement data pipeline orchestration, scheduling, monitoring, and automated error handling.
• Optimize Redshift queries, database performance, data transformations, and cloud resource utilization.
• Improve pipeline performance, scalability, reliability, and cost efficiency while meeting established SLAs.
• Implement data quality checks, validation rules, cleansing processes, and monitoring mechanisms.
• Develop secure data access solutions using AWS IAM, Lake Formation, IAM roles, policies, and access controls.
• Support enterprise data lake security, governance, and regulatory compliance requirements.
• Implement CI/CD pipelines and DevOps practices for automated deployment and release of data engineering solutions.
• Develop and maintain integrations using Informatica IDMC/Informatica Cloud.
• Support Generative AI and Agentic AI use cases involving data engineering workflows and data platforms.
• Collaborate with data analysts, business stakeholders, and technical teams to translate business requirements into production-ready pipelines.
• Ensure timely, accurate, and reliable data availability for business intelligence and analytics.
• Maintain technical documentation, troubleshoot production issues, and continuously improve data integration processes.

Required Skills & Experience

• 7+ years of experience in data engineering, database design, ETL/ELT development, and data warehousing.
• 5+ years of hands-on AWS experience, including S3, EMR, Glue, Athena, Redshift, RDS, Redshift Spectrum, and Airflow.
• 5+ years of programming experience using Python, PySpark, or Scala.
• Strong experience with Apache Spark, PySpark, Hadoop, and distributed data processing.
• Hands-on experience building and maintaining AWS Glue ETL jobs, EMR workloads, and Airflow/MWAA orchestration pipelines.
• Strong experience with Amazon Redshift, including SQL optimization, performance tuning, and large-scale data warehousing.
• Experience designing and maintaining scalable data lakes and cloud-based data integration solutions.
• 2+ years of CI/CD and DevOps experience supporting automated builds, deployments, and releases of data pipelines.
• 2+ years of hands-on AWS IAM and AWS Lake Formation experience, including roles, policies, access controls, and data lake governance.
• Experience with Informatica IDMC or Informatica Cloud.
• Experience with Generative AI or Agentic AI use cases in data engineering.
• Strong knowledge of data modeling, database design, SQL, and ETL/ELT processes.
• Experience implementing data validation, data cleansing, data quality monitoring, and error handling.
• Ability to troubleshoot pipeline failures, optimize data workflows, and improve performance and cost efficiency.
• Experience translating business requirements into technical specifications and production-ready data solutions.
• Strong understanding of cloud security, data governance, and regulatory compliance.
• Excellent analytical, troubleshooting, communication, and cross-functional collaboration skills.

Desired Skills

• Experience supporting large-scale enterprise AWS data platforms and cloud data modernization initiatives.
• Experience with AWS Lambda, AWS MWAA, Redshift Spectrum, and advanced Redshift optimization.
• Experience with infrastructure automation and CI/CD tools such as Terraform, Jenkins, GitHub Actions, or AWS CodePipeline.
• Knowledge of data lake architecture, lakehouse concepts, and modern cloud data platforms.
• Experience optimizing distributed Spark workloads and managing large datasets.
• Experience with Informatica Intelligent Data Management Cloud (IDMC) integrations and cloud data pipelines.
• Experience applying Generative AI, LLMs, or Agentic AI to data engineering workflows, automation, or data quality processes.
• Familiarity with data cataloging, metadata management, data lineage, and enterprise data governance.
• Experience supporting business intelligence, reporting, and advanced analytics platforms.
• Experience working in large enterprise, automotive, or manufacturing environments.