Data Engineer -Iceberg

American IT Systems; Atlanta (GA); On-site; USD 83,000 - 90,000; Full time

Job summary

Elevance Health is seeking a Data Engineer in Atlanta, GA. The role focuses on designing, developing, and optimizing scalable data pipelines in a cloud environment using SQL, Python, SSIS, Airflow, Spark, Iceberg, and Databricks.

The candidate will migrate legacy SSIS workflows to modern cloud-based pipelines, implement data quality checks, and collaborate with architects, developers, and analysts to deliver reliable data solutions.

Qualifications

• Proficient in designing ETL/ELT pipelines and data integration.
• Hands-on experience with Spark-based processing and data lakehouse concepts.
• Strong SQL skills and ability to optimize queries and workloads.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines.
• Develop complex SQL queries, stored procedures, and data transformations.
• Build and manage data pipelines using Airflow, Spark, and Databricks.
• Migrate legacy ETL workflows to cloud-based Spark/Databricks pipelines.
• Collaborate with teams to ensure data quality, governance, and reliability.

Skills

SQL

Python

ETL/ELT

Data pipelines

Performance tuning

Tools

Apache Spark / PySpark

Databricks

Apache Airflow

SSIS

Apache Iceberg

Job description

Data Engineer

Client: Elevance Health
Location: Atlanta, GA
Rate: $60 65/hr on C2C
Experience: 8-12+ years

Job Description

We are seeking an experienced Data Engineer with strong hands‑on experience in SQL, Python, SSIS, Airflow, Apache Spark, Apache Iceberg, and Databricks. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions in a cloud-based environment.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, SSIS, Airflow, and Spark.
• Develop complex SQL queries, stored procedures, data transformations, and performance optimization solutions.
• Build and maintain data pipelines and workflows using Apache Airflow, including scheduling, dependencies, monitoring, retries, and failure handling.
• Develop and optimize Apache Spark/PySpark applications for large‑scale data processing.
• Work with Databricks to develop, deploy, and optimize data engineering workloads.
• Work with Apache Iceberg tables and modern data lake/lakehouse architectures.
• Migrate and modernize legacy SSIS ETL workflows into cloud‑based and Spark/Databricks‑based data pipelines.
• Implement data ingestion, transformation, validation, and integration processes from multiple sources.
• Optimize Spark jobs, SQL queries, data partitioning, and storage strategies for performance and scalability.
• Implement data quality checks, error handling, logging, monitoring, and pipeline recovery mechanisms.
• Collaborate with data architects, developers, analysts, and business teams to understand requirements and deliver reliable data solutions.
• Follow best practices for data security, governance, performance, and operational reliability.

Required Skills

• Strong hands‑on experience with SQL
• Strong programming experience with Python
• Hands‑on experience with Apache Spark / PySpark
• Experience with Databricks
• Strong experience with Apache Airflow
• Hands‑on experience with SSIS
• Experience with Apache Iceberg
• Strong understanding of ETL/ELT and data pipeline development
• Experience working with large‑scale datasets and distributed data processing
• Strong troubleshooting and performance‑tuning skills
• Good understanding of modern data lake/lakehouse architectures