Senior Data Engineer — Scalable AI Data Pipelines

Quantiphi; Atlanta (GA); On-site; USD 110,000 - 170,000; Full time

Job summary

Quantiphi is seeking a Senior Data Engineer in Atlanta to design and build scalable, production-grade data platforms powering AI and analytics use cases. You will work hands-on at the intersection of data engineering, AI enablement, and cloud platforms, collaborating with clients and internal stakeholders to create modern data ecosystems.

The role requires 5+ years in data engineering and strong experience with Azure Databricks, PySpark, Python, SQL, and Pandas.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, IT, or related field.
• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks, PySpark, Python, SQL, and Pandas.
• Experience building and maintaining scalable ETL/ELT data pipelines.
• Experience working with large datasets, data transformation, validation, and quality checks.
• Proficiency with Git/version control and software development best practices.

Responsibilities

• Design, develop, test, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop data processing solutions using Azure Databricks and PySpark.
• Write, optimize, and troubleshoot complex SQL queries for data extraction and transformation.
• Use Python and Pandas for data manipulation, transformation, validation, and analysis.
• Integrate data from multiple sources and transform it into usable formats for downstream applications and analytics.
• Develop and maintain data workflows within the Microsoft Azure ecosystem.
• Perform data cleansing, validation, and quality checks to ensure data accuracy and consistency.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Optimize PySpark and Databricks jobs for performance, scalability, and cost efficiency.
• Work with large datasets and implement efficient data processing techniques.
• Collaborate with Data Scientists, Data Analysts, Software Engineers, and business stakeholders to understand data requirements.
• Document data pipelines, processes, technical solutions, and data flows.
• Follow established best practices for coding, testing, security, version control, and data governance.
• Participate in code reviews and contribute to continuous improvement of data engineering processes.

Skills

Data pipeline design

Team collaboration

Code reviews

Education

Bachelor's degree in CS/Engineering/IT

Tools

Azure Databricks

PySpark

Python

SQL

Pandas

Git

Job description

Quantiphi is seeking a Senior Data Engineer in Atlanta to design and build scalable, production-grade data platforms powering AI and analytics use cases. You will work hands-on at the intersection of data engineering, AI enablement, and cloud platforms, collaborating with clients and internal stakeholders to create modern data ecosystems.

The role requires 5+ years in data engineering and strong experience with Azure Databricks, PySpark, Python, SQL, and Pandas.