Data Engineer, Sr

Insight Global; Atlanta (GA); On-site; USD 120,000 - 170,000; Full time

Job summary

Insight Global is seeking a Senior Data Engineer to design, build, and optimize scalable data pipelines that support analytics, reporting, AI/ML, and platform operations. You will own pipeline reliability, lead root-cause analyses, and drive automation to reduce recurring issues across enterprise data platforms.

You will work with Databricks, Delta Lake, Azure Data Factory, and PySpark, collaborating with stakeholders to deliver trusted data and actionable insights that empower business

Qualifications

• 5-7 years of experience in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
• Strong expertise in advanced SQL development, query optimization, data analysis, and troubleshooting.
• Proficiency in Python and PySpark for large-scale data transformation, pipeline development, and automation.
• Hands-on experience with Databricks, Delta Lake, Azure Data Factory, and Azure data services.
• Experience designing and supporting batch, near real-time, and streaming data solutions.
• Strong understanding of CI/CD pipelines, Git-based source control, pipeline testing, and DevOps/DataOps practices.
• Experience with monitoring, observability, logging, alerting, and production support processes.
• Knowledge of data modeling, data warehousing concepts, and dimensional design principles

Responsibilities

• Design, develop, and enhance scalable data pipelines supporting analytics, reporting, AI/ML, and operational business processes.
• Own the reliability and stability of enterprise data platforms by proactively monitoring pipelines, troubleshooting production issues, and ensuring adherence to SLAs and operational standards.
• Lead root cause analysis and incident resolution efforts, driving permanent fixes, automation opportunities, and preventative controls to reduce recurring issues.
• Develop and maintain data solutions using Databricks, Azure Data Factory, SQL, and PySpark, following enterprise architecture and engineering standards.
• Drive DataOps best practices, including CI/CD implementation, automated testing, code reviews, deployment automation, release management, and environment governance.
• Improve data quality, performance, observability, and operational efficiency through monitoring, alerting, lineage tracking, logging, and automation initiatives.
• Work with structured, semi-structured, and streaming data, building scalable ingestion and transformation frameworks.
• Collaborate with business stakeholders, analytics teams, architects, product owners, and application teams to deliver reliable and business-aligned data solutions.
• Identify opportunities for continuous improvement in processes, architecture, governance, security, cost optimization, and platform scalability.

Skills

SQL development

Data analysis

Python

PySpark

Databricks

Delta Lake

Azure Data Factory

Azure data services

Batch, near real-time, streaming data

CI/CD

Git

DevOps/DataOps

Monitoring & observability

Data modeling & warehousing

Tools

Databricks

Delta Lake

Azure Data Factory

Azure data services

Job description

Overview

TheData & Insights organization at Insight Global is responsible for deliveringTrusted Data & Verified Insights to drive business decisions across the company. As aSenior Data Engineer, you will be part of the Data Engineering & Operations team responsible for enhancing and maintaining scalable data pipelines, supporting and improving platform operations processes, and enabling analytics, data science, and AI use cases.

This is a hands-on role reporting to theData Operations Manager with direct impact on Insight Global’s ability to deliver high-quality, actionable data to the business.

Responsibilities

• Design, develop, and enhance scalable data pipelines supporting analytics, reporting, AI/ML, and operational business processes.
• Own the reliability and stability of enterprise data platforms by proactively monitoring pipelines, troubleshooting production issues, and ensuring adherence to SLAs and operational standards.
• Lead root cause analysis and incident resolution efforts, driving permanent fixes, automation opportunities, and preventative controls to reduce recurring issues.
• Develop and maintain data solutions using Databricks, Azure Data Factory, SQL, and PySpark, following enterprise architecture and engineering standards.
• Drive DataOps best practices, including CI/CD implementation, automated testing, code reviews, deployment automation, release management, and environment governance.
• Improve data quality, performance, observability, and operational efficiency through monitoring, alerting, lineage tracking, logging, and automation initiatives.
• Work with structured, semi-structured, and streaming data, building scalable ingestion and transformation frameworks.
• Collaborate with business stakeholders, analytics teams, architects, product owners, and application teams to deliver reliable and business-aligned data solutions.
• Identify opportunities for continuous improvement in processes, architecture, governance, security, cost optimization, and platform scalability.

Qualifications

Required Skills

• 5-7 years of experience in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
• Strong expertise in advanced SQL development, query optimization, data analysis, and troubleshooting.
• Proficiency in Python and PySpark for large-scale data transformation, pipeline development, and automation.
• Hands-on experience with Databricks, Delta Lake, Azure Data Factory, and Azure data services.
• Experience designing and supporting batch, near real-time, and streaming data solutions.
• Strong understanding of CI/CD pipelines, Git-based source control, pipeline testing, and DevOps/DataOps practices.
• Experience with monitoring, observability, logging, alerting, and production support processes.
• Knowledge of data modeling, data warehousing concepts, and dimensional design principles

Preferred Skills

• Experience with Databricks Unity Catalog.
• Knowledge of data governance, data quality frameworks, security, and compliance requirements.
• Proven experience managing production incidents, performing root cause analysis, and driving long-term remediation plans.
• Ability to prioritize operational stability while balancing enhancement and project delivery commitments.
• Experience working in Agile, Scrum, or Scrumban environments.

This Role Is Ideal If You:

• Are excited to be part of a data team supporting data pipelines in Databricks.
• Thrive in Agile environments and enjoy working across disciplines.
• Take ownership of operational responsibilities and ability to fix data issues.
• Are passionate about delivering reliable, high-quality data solutions.
• Embody Insight Global’s shared values and contribute to a collaborative, high-performance culture.