Senior Data Engineer, DataOps & Pipelines Lead

Insight Global; Atlanta (GA); On-site; USD 120,000 - 170,000; Full time

Job summary

Insight Global is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines that support analytics, reporting, AI/ML, and platform operations. You will own pipeline reliability, lead root-cause analyses, and drive automation to reduce recurring issues across enterprise data platforms.

You will work with Databricks, Delta Lake, Azure Data Factory, and PySpark, collaborating with stakeholders to deliver trusted data and actionable insights that empower business

Qualifications

• 5-7 years of experience in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
• Strong expertise in advanced SQL development, query optimization, data analysis, and troubleshooting.
• Proficiency in Python and PySpark for large-scale data transformation, pipeline development, and automation.
• Hands-on experience with Databricks, Delta Lake, Azure Data Factory, and Azure data services.
• Experience designing and supporting batch, near real-time, and streaming data solutions.
• Strong understanding of CI/CD pipelines, Git-based source control, pipeline testing, and DevOps/DataOps practices.
• Experience with monitoring, observability, logging, alerting, and production support processes.
• Knowledge of data modeling, data warehousing concepts, and dimensional design principles

Responsibilities

• Design, develop, and enhance scalable data pipelines supporting analytics, reporting, AI/ML, and operational business processes.
• Own the reliability and stability of enterprise data platforms by proactively monitoring pipelines, troubleshooting production issues, and ensuring adherence to SLAs and operational standards.
• Lead root cause analysis and incident resolution efforts, driving permanent fixes, automation opportunities, and preventative controls to reduce recurring issues.
• Develop and maintain data solutions using Databricks, Azure Data Factory, SQL, and PySpark, following enterprise architecture and engineering standards.
• Drive DataOps best practices, including CI/CD implementation, automated testing, code reviews, deployment automation, release management, and environment governance.
• Improve data quality, performance, observability, and operational efficiency through monitoring, alerting, lineage tracking, logging, and automation initiatives.
• Work with structured, semi-structured, and streaming data, building scalable ingestion and transformation frameworks.
• Collaborate with business stakeholders, analytics teams, architects, product owners, and application teams to deliver reliable and business-aligned data solutions.
• Identify opportunities for continuous improvement in processes, architecture, governance, security, cost optimization, and platform scalability.

Skills

SQL development

Data analysis

Python

PySpark

Databricks

Delta Lake

Azure Data Factory

Azure data services

Batch, near real-time, streaming data

CI/CD

Git

DevOps/DataOps

Monitoring & observability

Data modeling & warehousing

Tools

Databricks

Delta Lake

Azure Data Factory

Azure data services

Job description

Insight Global is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines that support analytics, reporting, AI/ML, and platform operations. You will own pipeline reliability, lead root-cause analyses, and drive automation to reduce recurring issues across enterprise data platforms.

You will work with Databricks, Delta Lake, Azure Data Factory, and PySpark, collaborating with stakeholders to deliver trusted data and actionable insights that empower business