Senior AWS Data Engineer – Data Pipelines & Spark

Calance; Torrance (CA); Hybrid; USD 220,416,000 - 234,192,000; Full time

Benefits offered by this job

EPO/PPO Medical Plans

HMO/PPO Dental programs

Vision – VSP (Vision Plan Summary)

401K Retirement vesting program (VOYA)

Paid Bi-Weekly/Direct Deposit

Flex Spending Plan

Voluntary Life, AD&D, STD or LTD plans

Job summary

Calance is seeking a Sr. AWS Data Engineer for a 12-month hybrid contract in Torrance, CA. You will design and maintain data integration workflows using AWS Glue/EMR, AWS MWAA, Lambda and Redshift, with PySpark and Python for large datasets.

You will ensure data quality and integrity, optimize pipelines to meet SLAs and cost targets on AWS, and support BI analytics by translating business needs into data pipelines. The role requires onsite work Monday through Thursday.

Qualifications

• 5 years of experience in data engineering, database design, ETL processes, and data warehousing.
• 3 years of experience with AWS tools and technologies, such as S3, EMR, Glue, Athena, Redshift, RDS, Spectrum and Airflow.
• 2 years of experience with CI/CD tools.
• Strong knowledge of data storage and processing technologies, including databases and data lakes based distributed computing frameworks (e.g., Hadoop, Spark).
• 3 years programming in Python, Java, or Scala.
• Must have experience designing and developing Data Pipelines and Support them.
• Primary Skills : AWS EMR , GLUE, AIRFLOW, ICEBERG, REDSHIFT, RDS , IDMC, AMAZON KIRO and CI/CD

Responsibilities

• Design and implement data integration workflows using AWS Glue/EMR, AWS MWAA (Airflow), Lambda, and Redshift
• Demonstrate proficiency in PySpark, Apache Spark and Python for data processing large datasets
• Ensure data is accurately and efficiently extracted, transformed, and loaded into target systems.
• Validate and cleanse data to maintain high data quality.
• Ensure data quality and integrity by implementing monitoring, validation, and error handling mechanisms within data pipelines
• Enhance the performance, optimization of data workflows to meet SLAs, scalability of data integration processes and cost-efficiency on AWS cloud infrastructure.
• Identify and resolve performance bottlenecks, fine-tuning queries, and optimizing data processing to enhance Redshift performance
• Regularly review and refine integration processes to improve efficiency.
• Translate business requirements to technical specifications and coded data pipelines
• Ensure timely availability of integrated data for business intelligence and analytics.
• Collaborate with data analysts and business stakeholders to meet their data requirements.
• Maintain Documentation and Compliance: Document all data integration processes, workflows, and technical & system specifications.
• Ensure compliance with data governance policies, industry standards, and regulatory requirements.

Skills

Python/Java/Scala

Data engineering

CI/CD

Tools

AWS EMR

AWS Glue

Airflow

Iceberg

Redshift

RDS

IDMC

Amazon Kiro

CI/CD

Job description

Calance is seeking a Sr. AWS Data Engineer for a 12-month hybrid contract in Torrance, CA. You will design and maintain data integration workflows using AWS Glue/EMR, AWS MWAA, Lambda and Redshift, with PySpark and Python for large datasets.

You will ensure data quality and integrity, optimize pipelines to meet SLAs and cost targets on AWS, and support BI analytics by translating business needs into data pipelines. The role requires onsite work Monday through Thursday.