Senior Data Engineer

Quantum Technologies. LLC; Atlanta (GA); Hybrid; USD 93,000 - 133,000; Full time

Job summary

Quantum Technologies. LLC is seeking a Senior Data Engineer (Python/PySpark) in Atlanta, GA, on a hybrid, 12-month contract.

Responsibilities include designing scalable data pipelines, ETL/ELT workflows, and data integration across cloud platforms. The role requires 7+ years of data engineering experience, strong Python/PySpark skills, SQL proficiency, and familiarity with Airflow, Git, and CI/CD pipelines.

Qualifications

• 7+ years of data engineering experience.
• Strong Python and PySpark expertise.
• SQL and database development proficiency.
• Experience with AWS, Azure, or GCP.
• Familiarity with Airflow and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop complex SQL queries and optimize database performance.
• Process large datasets using Apache Spark.
• Integrate data from multiple sources into centralized data platforms.
• Develop data transformation and validation processes.
• Implement data quality checks and monitoring solutions.
• Work with cloud-based data storage and processing services.
• Optimize data processing workflows for performance and scalability.
• Collaborate with data analysts, architects, and application development teams.
• Troubleshoot data pipeline failures and resolve production issues.
• Maintain technical documentation and follow data engineering best practices.

Skills

Python

PySpark

SQL

Spark

AWS

Airflow

Git

CI/CD

Tools

Apache Airflow

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Job Title: Senior Data Engineer (Python / PySpark)
Location: Atlanta, GA (Hybrid)
Duration: 12 Months Contract
Bill Rate: $82/hr
Job Type: W-2, 1099, C2C
Client: To be Disclosed
Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

We are seeking an experienced Senior Data Engineer with strong expertise in Python, PySpark, SQL, and cloud-based data engineering technologies. The ideal candidate will be responsible for designing and implementing scalable data pipelines, processing large datasets, and developing reliable data integration solutions.

Roles and Responsibilities:

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and optimize ETL and ELT workflows for structured and unstructured data.
• Develop complex SQL queries and optimize database performance.
• Process large datasets using Apache Spark.
• Integrate data from multiple sources into centralized data platforms.
• Develop data transformation and validation processes.
• Implement data quality checks and monitoring solutions.
• Work with cloud-based data storage and processing services.
• Optimize data processing workflows for performance and scalability.
• Collaborate with data analysts, architects, and application development teams.
• Troubleshoot data pipeline failures and resolve production issues.
• Maintain technical documentation and follow data engineering best practices.

Required Skills:

• 7+ years of experience in data engineering.
• Strong hands-on experience with Python and PySpark.
• Expertise in SQL and database development.
• Experience with Apache Spark and distributed data processing.
• Strong knowledge of ETL/ELT development.
• Experience with AWS, Azure, or GCP.
• Knowledge of data warehousing concepts and data modeling.
• Experience with Apache Airflow or similar orchestration tools.
• Familiarity with Git and CI/CD pipelines.
• Strong analytical and problem-solving skills.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.