Data Engineer

Quantiphi; Atlanta (GA); On-site; USD 110,000 - 170,000; Full time

Job summary

Quantiphi is seeking a Senior Data Engineer in Atlanta to design and build scalable, production-grade data platforms powering AI and analytics use cases. You will work hands-on at the intersection of data engineering, AI enablement, and cloud platforms, collaborating with clients and internal stakeholders to create modern data ecosystems.

The role requires 5+ years in data engineering and strong experience with Azure Databricks, PySpark, Python, SQL, and Pandas.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, IT, or related field.
• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks, PySpark, Python, SQL, and Pandas.
• Experience building and maintaining scalable ETL/ELT data pipelines.
• Experience working with large datasets, data transformation, validation, and quality checks.
• Proficiency with Git/version control and software development best practices.

Responsibilities

• Design, develop, test, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop data processing solutions using Azure Databricks and PySpark.
• Write, optimize, and troubleshoot complex SQL queries for data extraction and transformation.
• Use Python and Pandas for data manipulation, transformation, validation, and analysis.
• Integrate data from multiple sources and transform it into usable formats for downstream applications and analytics.
• Develop and maintain data workflows within the Microsoft Azure ecosystem.
• Perform data cleansing, validation, and quality checks to ensure data accuracy and consistency.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Optimize PySpark and Databricks jobs for performance, scalability, and cost efficiency.
• Work with large datasets and implement efficient data processing techniques.
• Collaborate with Data Scientists, Data Analysts, Software Engineers, and business stakeholders to understand data requirements.
• Document data pipelines, processes, technical solutions, and data flows.
• Follow established best practices for coding, testing, security, version control, and data governance.
• Participate in code reviews and contribute to continuous improvement of data engineering processes.

Skills

Data pipeline design

Team collaboration

Code reviews

Education

Bachelor's degree in CS/Engineering/IT

Tools

Azure Databricks

PySpark

Python

SQL

Pandas

Git

Job description

Quantiphi is an award-winning, AI-First digital engineering and consulting company focused on delivering high-impact Services and Solutions that help organizations solve what truly matters. We partner with enterprises to reimagine their businesses through intelligent, scalable, and transformative AI driving measurable outcomes at the very core of their operations.

Since our founding in 2013, Quantiphi has tackled some of the world’s most complex business challenges by combining deep industry expertise, disciplined cloud and data engineering practices, and cutting-edge applied AI research. Our work is rooted in delivering accelerated, quantifiable business value, not just technology for technology’s sake.

Headquartered in Boston, Quantiphi is a global organization with 4,000+ professionals serving clients across key industry verticals, including BFSI, Healthcare & Life Sciences, CPG, MFG, TME etc. As an Elite and Premier partner to leading cloud and AI platforms such as NVIDIA, Google Cloud, AWS, and Snowflake, we build and deliver enterprise-grade AI services and solutions that create real-world impact.

We’ve been recognized with:

• 21x Google Cloud Partner of the Year awards in the last 8 years.
• 3x AWS AI/ML award wins.
• 3x NVIDIA Partner of the Year titles.
• 2x Snowflake Partner of the Year awards.
• We have also garnered top analyst recognitions from Gartner, ISG, and Everest Group.
• We offer first-in-class industry solutions across Healthcare, Financial Services, Consumer Goods, Manufacturing, and more, powered by cutting-edge Generative AI and Agentic AI accelerators.
• We have been certified as a Great Place to Work for the third year in a row- 2021, 2022, 2023.

Be part of a trailblazing team that’s shaping the future of AI, ML, and cloud innovation.

Your next big opportunity starts here!

Experience Level: 5+ yrs

Work Location: Atlanta, GA

Role Overview

As a Senior Data Engineer, you will design and build scalable, production-grade data platforms and pipelines that power AI and analytics use cases. This role is highly hands-on and sits at the intersection of data engineering, AI enablement, and cloud platforms.

You will work closely with clients and internal stakeholders to build modern data ecosystems.

Key Responsibilities:

• Design, develop, test, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop data processing solutions using Azure Databricks and PySpark.
• Write, optimize, and troubleshoot complex SQL queries for data extraction and transformation.
• Use Python and Pandas for data manipulation, transformation, validation, and analysis.
• Integrate data from multiple sources and transform it into usable formats for downstream applications and analytics.
• Develop and maintain data workflows within the Microsoft Azure ecosystem.
• Perform data cleansing, validation, and quality checks to ensure data accuracy and consistency.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Optimize PySpark and Databricks jobs for performance, scalability, and cost efficiency.
• Work with large datasets and implement efficient data processing techniques.
• Collaborate with Data Scientists, Data Analysts, Software Engineers, and business stakeholders to understand data requirements.
• Document data pipelines, processes, technical solutions, and data flows.
• Follow established best practices for coding, testing, security, version control, and data governance.
• Participate in code reviews and contribute to continuous improvement of data engineering processes.

Basic Qualifications:

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.
• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks, PySpark, Python, SQL, and Pandas.
• Experience building and maintaining scalable ETL/ELT data pipelines.
• Experience working with large datasets, data transformation, validation, and quality checks.
• Proficiency with Git/version control and software development best practices.

Other Qualifications:

• Experience with Azure Data Lake, Azure Data Factory, Synapse, or related Azure services.
• Experience with Delta Lake and Spark optimization.
• Knowledge of data modeling, data warehousing, and data governance.
• Experience with CI/CD, testing, and production support.
• Strong problem-solving, communication, and collaboration skills.
• Experience in a client-facing or consulting environment is a plus.
• Azure/Databricks certification is preferred.

What’s in it for you?

• Industry Influence: Shape the future of AI and analytics by advising senior executives and influencing transformation strategies across industries.
• Executive Visibility: Work directly with C-suite leaders, industry executives, and strategic decision-makers on high-impact initiatives.
• Thought Leadership: Represent a leading AI-first organization in industry forums, conferences, and executive engagements.
• Professional Legacy: Help define how organizations adopt, scale, and operationalize AI while mentoring the next generation of technical and business leaders.