Senior Data Engineer - Hybrid (Azure Databricks)

Publicis Sapient; Arlington (VA); Hybrid; USD 109.000 - 130.000; Vollzeit

Benefits dieser Stelle

Inclusive workplace

Learning & development opportunities

Competitive compensation and benefits

Flexibility to support work-life BAL

Comprehensive health benefits for you

Generous paid leave and holidays

Wellness program and employee assit

Zusammenfassung

Publicis Sapient in Arlington, VA is seeking a Senior Data Engineer to design and enhance scalable data pipelines using Azure Data Factory, Delta Lake, and Azure Synapse. You will own testing, data quality, performance tuning, and production support for enterprise data solutions.

You will work with PySpark, Python, SQL, and Databricks to build reliable data products and collaborate with data architects, analysts, and business partners.

Qualifikationen

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.
• 4+ years of data engineering experience, including hands-on delivery and support of enterprise data pipelines.
• 2+ years of hands-on Azure Databricks data engineering experience.
• Strong experience with Databricks, Delta Live Tables (DLT), Delta Lake, PySpark, Python, SQL, and Spark SQL.
• Strong experience orchestrating data pipelines with Azure Data Factory.
• Experience loading and serving data through Enterprise Data Lake, Delta Lake, and Azure Synapse Analytics dedicated SQL pools.
• Experience configuring Databricks compute, data quality rules, monitoring, maintenance processes, and workload optimization.
• Experience performing unit testing and coordinating integration testing and UAT.
• Experience creating HLDs, detailed designs, data pipeline documentation, and operational runbooks.
• Experience with Azure DevOps, Git, CI/CD pipelines, and production deployment practices.
• Demonstrated experience supporting production data platforms and resolving data pipeline incidents.

Aufgaben

• Create new batch and streaming data pipelines by leveraging and extending existing data ingestion frameworks and tools.
• Orchestrate, schedule, and monitor data pipelines using Azure Data Factory.
• Develop and enhance data transformations based on business and technical requirements using PySpark, Python, and SQL.
• Parse, transform, and load data into the Enterprise Data Lake, Delta Lake, and Azure Synapse Analytics dedicated SQL pools.
• Build and maintain Databricks data engineering solutions, including Delta Live Tables (DLT), compute configuration, and pipeline maintenance.
• Configure and maintain data quality rules, validation controls, reconciliation processes, monitoring, and automated alerts.
• Perform unit testing and coordinate integration testing and user acceptance testing (UAT) with business and technology partners.
• Create and maintain high-level designs (HLDs), detailed designs (DDs), data mappings, operational runbooks, and support documentation.
• Tune and optimize Databricks, Spark, SQL, and Synapse workloads for performance, reliability, scalability, and cost efficiency.
• Provide production support, including incident troubleshooting, root-cause analysis, defect resolution, and operational maintenance.
• Use Azure DevOps and Git to support source control, CI/CD, release management, and controlled production deployments.
• Collaborate with data architects, analysts, business stakeholders, and support teams to deliver reliable enterprise data products.

Kenntnisse

Azure Databricks

Delta Lake

PySpark

Python

SQL

Azure Data Factory

Delta Live Tables

Ausbildung

Bachelor's degree in Computer Science, Engineering, Information Systems

Tools

Azure Synapse

Databricks

Git

Azure DevOps

Jobbeschreibung

Publicis Sapient in Arlington, VA is seeking a Senior Data Engineer to design and enhance scalable data pipelines using Azure Data Factory, Delta Lake, and Azure Synapse. You will own testing, data quality, performance tuning, and production support for enterprise data solutions.

You will work with PySpark, Python, SQL, and Databricks to build reliable data products and collaborate with data architects, analysts, and business partners.