Sr. Data Engineer

Sansar Tec, LLC; Haddon Heights (NJ); On-site; USD 110,000 - 140,000; Full time

Job summary

A leading data solutions company is seeking an experienced Senior Data Engineer specializing in Azure and Databricks to drive data engineering initiatives. The ideal candidate will create scalable data pipelines and support the Enterprise Data Lake and Data Warehouse. Responsibilities include implementing complex ETL processes, orchestrating data flows, and providing production support. Candidates should have at least 5 years of Azure-focused data engineering experience, with strong skills in SQL, Python, and PySpark.

Qualifications

• 5+ years of experience in data engineering with Azure-focused solutions.
• Proven track record in building and orchestrating data pipelines in a cloud environment.
• Strong skills in SQL, Python, and PySpark for data transformations.

Responsibilities

• Build new data pipelines and enhance existing ones using Databricks and Azure Data Factory.
• Implement complex data transformations according to business requirements.
• Utilize Azure Data Factory to orchestrate data flows and automate processes.
• Conduct unit testing and manage User Acceptance Testing (UAT).
• Develop and maintain design documents and runbooks.
• Configure and optimize compute resources.
• Identify and implement performance improvements for data pipelines.
• Provide ongoing support for production environments.

Skills

Azure Data Factory

Databricks

PySpark

SQL

Python

Education

Bachelor's or Master's degree in Computer Science or related field

Job description

Overview

Haddon Heights, New Jersey, United States

Job Overview: Were seeking an experienced Senior Data Engineer specializing in Azure and Databricks to drive data engineering initiatives, create scalable data pipelines, and support our Enterprise Data Lake and Data Warehouse (Synapse). The ideal candidate will be well-versed in Azure Data Factory, Databricks (DLT), and have hands-on experience with PySpark, SQL, and Azure DevOps. This role requires deep technical knowledge in data pipeline orchestration, data transformation, and optimizing performance within Azure cloud services.

You must reside in the US; No 3rd parties.

Responsibilities

• Data Pipeline Development: Build new data pipelines and enhance existing ones using Databricks (Data Engineering, Delta Lake), Azure Data Factory, and Synapse to load data into the Enterprise Data Lake, Delta Lake, and Enterprise Data Warehouse.
• Data Transformation and Loading (ETL/ELT): Implement complex data transformations to parse, cleanse, and load data according to business requirements and best practices.
• Orchestration and Automation: Utilize Azure Data Factory to orchestrate data flows and automate data ingestion processes.
• Testing and Quality Assurance: Perform unit testing, coordinate integration testing, and manage User Acceptance Testing (UAT) to ensure high-quality data.
• Documentation and Runbooks: Develop and maintain high-level design (HLD), detailed design (DD), and runbooks to support production operations and improve data pipeline transparency.
• Compute Configuration and Maintenance: Configure and optimize compute resources, apply Data Quality (DQ) rules, and manage regular maintenance schedules.
• Performance Tuning: Identify and implement performance improvements for data pipelines, ensuring efficiency and scalability.
• Production Support: Provide ongoing support for production environments, troubleshoot issues, and ensure optimal system performance.

Technical Skills

• Core Technologies: Databricks (Data Engineering, DLT), Azure Data Factory, Synapse (Dedicated SQL Pool), SQL, PySpark, Python
• Nice to Have: Experience with Precisely for data integration and governance

Qualifications

• Bachelors or Masters degree in Computer Science, Data Engineering, or related field.
• 5+ years of experience in data engineering with Azure-focused data solutions.
• Proven track record in building and orchestrating data pipelines in a cloud environment, with expertise in Databricks and Azure Data Factory.
• Strong skills in SQL, Python, and PySpark for data transformations.
• Excellent problem-solving skills with a focus on performance optimization and production support.