Senior Data Engineer - Python / PySpark | Hybrid (Atlanta)

Quantum Technologies. LLC; Atlanta (GA); Hybrid; USD 93,000 - 133,000; Full time

Job summary

Quantum Technologies. LLC is seeking a Senior Data Engineer (Python/PySpark) in Atlanta, GA, on a hybrid, 12-month contract.

Responsibilities include designing scalable data pipelines, ETL/ELT workflows, and data integration across cloud platforms. The role requires 7+ years of data engineering experience, strong Python/PySpark skills, SQL proficiency, and familiarity with Airflow, Git, and CI/CD pipelines.

Qualifications

• 7+ years of data engineering experience.
• Strong Python and PySpark expertise.
• SQL and database development proficiency.
• Experience with AWS, Azure, or GCP.
• Familiarity with Airflow and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop complex SQL queries and optimize database performance.
• Process large datasets using Apache Spark.
• Integrate data from multiple sources into centralized data platforms.
• Develop data transformation and validation processes.
• Implement data quality checks and monitoring solutions.
• Work with cloud-based data storage and processing services.
• Optimize data processing workflows for performance and scalability.
• Collaborate with data analysts, architects, and application development teams.
• Troubleshoot data pipeline failures and resolve production issues.
• Maintain technical documentation and follow data engineering best practices.

Skills

Python

PySpark

SQL

Spark

AWS

Airflow

Git

CI/CD

Tools

Apache Airflow

Job description

Quantum Technologies. LLC is seeking a Senior Data Engineer (Python/PySpark) in Atlanta, GA, on a hybrid, 12-month contract.

Responsibilities include designing scalable data pipelines, ETL/ELT workflows, and data integration across cloud platforms. The role requires 7+ years of data engineering experience, strong Python/PySpark skills, SQL proficiency, and familiarity with Airflow, Git, and CI/CD pipelines.