Job Summary
The Senior Data Engineer is responsible for designing, building, and managing scalable data pipelines and data platform solutions within the Enterprise Data Platform (EDP) using Azure and Databricks. The role focuses on enabling reliable, governed, and high-performance data integration and transformation across multiple source systems to support analytics, reporting, and advanced data use cases.
Duties and Responsibilities
This position combines hands-on data engineering, platform optimization, and stakeholder coordination, ensuring the delivery of robust, secure, and business-aligned data solutions while supporting data governance and best practices.
DataEngineering & Pipeline Development
- Design, develop, and maintain scalable data pipelines using Azure Databricks, Azure Data Factory, and related services
- Implement ETL/ELT processes to ingest, transform, and load data into the Enterprise Data Platform
- Develop batch and near real-time data processing solutions
- Apply medallion architecture (bronze, silver, gold layers) for structured data processing
- Ensure data quality, consistency, and integrity across pipelines and datasets
PlatformOperations & Optimization
- Monitor, maintain, and optimize data pipelines, workflows, and jobs in Databricks and Azure
- Troubleshoot pipeline failures, performance issues, and data inconsistencies
- Optimize Spark jobs, query performance, and storage utilization (e.g., Delta Lake optimization)
- Ensure high availability, scalability, and reliability of data services
- Maintain documentation and ensure compliance with data governance, security, and operational standards
TeamManagement
- Provide guidance to data engineers and developers on best practices in Azure and Databricks
- Review code, pipelines, and technical designs to ensure quality and standards compliance
- Support the team in resolving technical issues and improving delivery efficiency
- Promote reusable frameworks, templates, and automation practices
- Mentor team members on data engineering concepts and cloud-based architectures
StakeholderEngagement
- Collaborate with business units, analytics teams, and platform engineers to gather and validate data requirements
- Translate business needs into scalable data models and pipelines
- Act as liaison between source system owners, data consumers, and platform teams
- Communicate pipeline status, issues, and improvements to stakeholders
- Support analytics and BI teams by ensuring availability of clean and reliable datasets
Technical Competencies and Skills
Data Engineering & Databricks, Azure Data Platform, Data Modeling & Performance Optimization, Data Governance & Security, DevOps & Automation
Education, Trainings and Licenses Required
- Bachelor's degree in Computer Science, Information Technology