Senior Data Engineer

Insight Global; Atlanta (GA); Hybride; USD 130 000 - 170 000; Plein temps

Résumé du poste

Insight Global is seeking a Senior Data Engineer/DataOps lead to design and scale data pipelines supporting analytics, AI/ML, and operations. You will own platform reliability, drive CI/CD, and mentor junior engineers.

You will collaborate with analytics teams and architects to deliver business-aligned data solutions while advancing modernization and cloud adoption.

Qualifications

• 5–7 years in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
• Strong SQL development, data analysis, and troubleshooting skills.
• Proficient in Python and PySpark for large-scale data workflows.
• Hands-on with Databricks, Delta Lake, Azure Data Factory, and Azure data services.

Responsabilités

• Design, develop, and enhance scalable data pipelines for analytics, AI/ML, and operations.
• Own reliability and stability of enterprise data platforms; monitor pipelines and SLAs.
• Lead root cause analysis and permanent fixes to reduce recurring issues.
• Drive DataOps with CI/CD, automated testing, and deployment automation.
• Improve data quality, observability, logging, and operational efficiency.
• Mentor junior engineers and provide technical leadership.
• Collaborate with stakeholders to deliver reliable data solutions.
• Participate in production support, deployments, and readiness reviews.

Connaissances

SQL development

Python

PySpark

Databricks

Delta Lake

Azure Data Factory

CI/CD

DevOps practices

Monitoring observability

Data modeling

Cloud architectures

Outils

Databricks

Delta Lake

Azure Data Factory

Azure data services

Description du poste

Hybrid in office Mon-Thurs, Friday remote

Direct hire!

Primary Responsibilities

• Design, develop, and enhance scalable data pipelines supporting analytics, reporting, AI/ML, and operational business processes.
• Own the reliability and stability of enterprise data platforms by proactively monitoring pipelines, troubleshooting production issues, and ensuring adherence to SLAs and operational standards.
• Lead root cause analysis and incident resolution efforts, driving permanent fixes, automation opportunities, and preventative controls to reduce recurring issues.
• Develop and maintain data solutions using Databricks, Azure Data Factory, SQL, and PySpark, following enterprise architecture and engineering standards.
• Drive DataOps best practices, including CI/CD implementation, automated testing, code reviews, deployment automation, release management, and environment governance.
• Improve data quality, performance, observability, and operational efficiency through monitoring, alerting, lineage tracking, logging, and automation initiatives.
• Support modernization and migration efforts, including adoption of new data engineering technologies.
• Work with structured, semi-structured, and streaming data, building scalable ingestion and transformation frameworks.
• Collaborate with business stakeholders, analytics teams, architects, product owners, and application teams to deliver reliable and business-aligned data solutions.
• Mentor junior engineers and provide technical leadership through design guidance, troubleshooting support, knowledge sharing, and engineering best practice adoption.
• Participate in production support, deployment activities, and operational readiness reviews, ensuring platform availability and rapid issue recovery.
• Identify opportunities for continuous improvement in processes, architecture, governance, security, cost optimization, and platform scalability.

Required Skills & Qualifications

Technical Skills

• 5-7 years of experience in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
• Strong expertise in advanced SQL development, query optimization, data analysis, and troubleshooting.
• Proficiency in Python and PySpark for large-scale data transformation, pipeline development, and automation.
• Hands-on experience with Databricks, Delta Lake, Azure Data Factory, and Azure data services.
• Experience designing and supporting batch, near real-time, and streaming data solutions.
• Strong understanding of CI/CD pipelines, Git-based source control, pipeline testing, and DevOps/DataOps practices.
• Experience with monitoring, observability, logging, alerting, and production support processes.
• Knowledge of data modeling, data warehousing concepts, and dimensional design principles
• Experience with cloud-native architectures and enterprise-scale data platforms.
• Familiarity with deployment automation and platform governance practices.

Operational Excellence

• Proven experience managing production incidents, performing root cause analysis, and driving long-term remediation plans.
• Ability to prioritize operational stability while balancing enhancement and project delivery commitments.
• Strong communication skills with the ability to translate technical solutions into business outcomes.
• Ability to work effectively with business stakeholders, product owners, architects, and leadership teams.

Preferred Qualifications

• Experience supporting enterprise data modernization or cloud migration programs.
• Knowledge of data governance, data quality frameworks, security, and compliance requirements.
• Experience working in Agile, Scrum, or Scrumban environments.
• Relevant certifications in Azure Data Engineering, Databricks, or Cloud Architecture.

What Success Looks Like in IG DataOps

A successful senior team member is not only delivering data pipelines but also:

• Acts as a trusted technical leader for the team.
• Improves platform stability and reduces operational risk.
• Automates repetitive support activities.
• Drives faster incident resolution and permanent fixes.
• Expands business understanding to proactively solve problems.
• Mentors team members and elevates overall engineering maturity.
• Contributes to Data Platform modernization and strategic initiatives.
• Balances project execution with operational excellence.