Senior Data Engineer: AI-Driven Pipelines & Data Lakes

LABUR; Bedford (MA); On-site; USD 103,000 - 110,000; Full time

Job summary

LABUR is hiring a Senior Data Engineer with 8+ years of experience to build and optimize data lakes, pipelines, and AI-enabled data engineering practices. You will work across Databricks, Azure Data Factory, and Informatica IDMC to deliver scalable, governance-driven solutions for a highly regulated environment.

The role emphasizes independent work, strong communication with business users, and hands-on migration work from PowerCenter to IDMC as needed.

Qualifications

• 8+ years of hands-on experience in data engineering, analytics engineering, or related disciplines.
• Strong proficiency with Databricks, including Delta Lake, PySpark, and Unity Catalog.
• Strong experience with Informatica; hands-on IDMC implementation experience strongly preferred.
• Proficiency in SQL and Python.
• Solid experience with Azure Data Factory for pipeline orchestration and scheduling.
• Experience in life sciences or other highly regulated industries strongly preferred.
• Demonstrated ability to leverage AI to solve business problems and optimize the data lifecycle.
• Excellent communication skills with the ability to explain technical issues to business stakeholders in clear, accessible terms.

Responsibilities

• Design, develop, document, and maintain scalable data ingestion and transformation pipelines using Azure Data Factory, Databricks, and PySpark
• Monitor daily scheduled jobs, troubleshoot failures, and resolve data issues promptly
• Enforce data governance, data quality standards, and coding best practices across all deliverables
• Collaborate with business and technical stakeholders to translate requirements into scalable, production-ready solutions
• Identify and implement opportunities to optimize existing pipelines and data lake infrastructure by leveraging AI
• Implement and manage data catalog solutions using Unity Catalog to create reliable, discoverable data products
• Lead hands-on integration work in Informatica IDMC, including migrations from PowerCenter to IDMC
• Lead data analysis and solution design efforts, aligning cross-functional stakeholders around a shared technical vision
• Embed governance principles into every stage of the data delivery lifecycle

Skills

Databricks

Unity Catalog

PySpark

SQL

Python

Azure Data Factory

Informatica IDMC

Data governance

AI in data engineering

Communication

Tools

Databricks

Azure Data Factory

Informatica IDMC

Job description

LABUR is hiring a Senior Data Engineer with 8+ years of experience to build and optimize data lakes, pipelines, and AI-enabled data engineering practices. You will work across Databricks, Azure Data Factory, and Informatica IDMC to deliver scalable, governance-driven solutions for a highly regulated environment.

The role emphasizes independent work, strong communication with business users, and hands-on migration work from PowerCenter to IDMC as needed.