Senior Data Engineer: Spark, Databricks & Airflow

American IT Systems; Atlanta (GA); On-site; USD 83,000 - 90,000; Full time

Job summary

Elevance Health is seeking a Data Engineer in Atlanta, GA. The role focuses on designing, developing, and optimizing scalable data pipelines in a cloud environment using SQL, Python, SSIS, Airflow, Spark, Iceberg, and Databricks.

The candidate will migrate legacy SSIS workflows to modern cloud-based pipelines, implement data quality checks, and collaborate with architects, developers, and analysts to deliver reliable data solutions.

Qualifications

• Proficient in designing ETL/ELT pipelines and data integration.
• Hands-on experience with Spark-based processing and data lakehouse concepts.
• Strong SQL skills and ability to optimize queries and workloads.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines.
• Develop complex SQL queries, stored procedures, and data transformations.
• Build and manage data pipelines using Airflow, Spark, and Databricks.
• Migrate legacy ETL workflows to cloud-based Spark/Databricks pipelines.
• Collaborate with teams to ensure data quality, governance, and reliability.

Skills

SQL

Python

ETL/ELT

Data pipelines

Performance tuning

Tools

Apache Spark / PySpark

Databricks

Apache Airflow

SSIS

Apache Iceberg

Job description

Elevance Health is seeking a Data Engineer in Atlanta, GA. The role focuses on designing, developing, and optimizing scalable data pipelines in a cloud environment using SQL, Python, SSIS, Airflow, Spark, Iceberg, and Databricks.

The candidate will migrate legacy SSIS workflows to modern cloud-based pipelines, implement data quality checks, and collaborate with architects, developers, and analysts to deliver reliable data solutions.