Group Data Engineer I

DP World; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

DP World in Bengaluru seeks a Senior Data Architect to define target architectures, design patterns and standards for batch and streaming data, lakehouse design, data modelling, sharing and serving. You will lead reviews and critical technical decisions to ensure scalability, security, resilience and reuse.

You will build production-grade pipelines using SQL, Python, Spark and cloud data platform technologies, establish reusable components, CI/CD and infrastructure-as-code, and mentor engineers

Qualifications

• Bachelor’s degree in CS/Engineering/IT; Master’s desirable.
• 7+ years in data architecture/engineering with large-scale data platforms.
• Advanced SQL, Python, Spark, distributed processing and data modelling.
• Cloud platforms (Azure, AWS, GCP); Databricks/lakehouse experience preferred.
• Batch and streaming ingestion, CDC, APIs, data lakes/warehouses and modern architecture patterns.
• Git, CI/CD, infrastructure-as-code, automated testing, observability and production practices.
• Governance, security, privacy, lineage, metadata management and data quality controls.
• Ability to lead architecture/design reviews and mentor engineers.

Responsibilities

• Define target architectures, engineering patterns and standards for batch/streaming integration, lakehouse design, data modelling, sharing and serving.
• Lead design reviews and technical decisions for complex/high-impact initiatives ensuring scalability and reuse.
• Design and build production-grade pipelines and reusable frameworks using SQL, Python, Spark and cloud data platform technologies.
• Establish reusable ingestion/transformation components, CI/CD and infrastructure-as-code to accelerate onboarding and reduce delivery risk.
• Set standards for observability, SLAs, performance tuning, disaster recovery and root-cause resolution.
• Optimize compute, storage and workload design to improve platform performance, reliability and unit cost.
• Embed automated data quality, lineage, metadata, access control, privacy and retention requirements into the lifecycle.
• Partner with Governance and Security teams to ensure trusted, auditable data products.
• Drive automated testing, code quality, version control, deployment automation, coding standards and debt reduction.
• Evaluate emerging technologies and convert proofs of concept into scalable enterprise standards.
• Mentor engineers and provide hands-on support for complex troubleshooting and decisions.
• Collaborate with product, analytics, AI/ML, platform and source-system teams to deliver reusable data capabilities.

Skills

Leadership

Collaboration

Communication

Azure

Airflow

Azure Data Factory

Databricks

Terraform

CloudFormation

Tableau

Power BI

CI/CD

Agile

Education

Bachelor's degree in Computer Science, Engineering, Information Technology or related discipline

Master's degree desirable

Tools

Apache Airflow

Azure Data Factory

Databricks

Terraform

CloudFormation

Docker

Job description

• Define target architectures, engineering patterns and standards for batch/streaming integration, lakehouse design, data modelling, sharing and serving.
• Lead design reviews and technical decisions for complex or high-impact initiatives, ensuring scalability, security, resilience and reuse.
• Design and build production-grade pipelines and reusable frameworks using SQL, Python, Spark and cloud data platform technologies.
• Establish reusable ingestion/transformation components, CI/CD and infrastructure-as-code to accelerate onboarding and reduce delivery risk.
• Reliability, Performance & Cost:
• Set standards for observability, SLAs, performance tuning, disaster recovery, incident prevention and root-cause resolution.
• Optimize compute, storage and workload design to improve platform performance, reliability and unit cost.
• Data Quality, Governance & Security:
• Embed automated data quality, lineage, metadata, access control, privacy and retention requirements into the engineering lifecycle.
• Partner with Governance and Security teams to ensure critical data products are trusted, auditable and compliant.
• Engineering Excellence & Automation:
• Drive automated testing, code quality, version control, deployment automation, coding standards and technical debt reduction.
• Evaluate emerging technologies, lead proofs of concept and convert proven capabilities into scalable enterprise standards.
• Mentor engineers, raise technical capability and provide hands‑on support for complex troubleshooting and engineering decisions.
• Collaborate with product, analytics, AI/ML, platform and source-system teams to deliver reusable, trusted data capabilities.

Job Description

• Define target architectures, engineering patterns and standards for batch/streaming integration, lakehouse design, data modelling, sharing and serving.
• Lead design reviews and technical decisions for complex or high-impact initiatives, ensuring scalability, security, resilience and reuse.
• Design and build production-grade pipelines and reusable frameworks using SQL, Python, Spark and cloud data platform technologies.
• Establish reusable ingestion/transformation components, CI/CD and infrastructure-as-code to accelerate onboarding and reduce delivery risk.
• Reliability, Performance & Cost:
• Set standards for observability, SLAs, performance tuning, disaster recovery, incident prevention and root-cause resolution.
• Optimize compute, storage and workload design to improve platform performance, reliability and unit cost.
• Data Quality, Governance & Security:
• Embed automated data quality, lineage, metadata, access control, privacy and retention requirements into the engineering lifecycle.
• Partner with Governance and Security teams to ensure critical data products are trusted, auditable and compliant.
• Engineering Excellence & Automation:
• Drive automated testing, code quality, version control, deployment automation, coding standards and technical debt reduction.
• Evaluate emerging technologies, lead proofs of concept and convert proven capabilities into scalable enterprise standards.
• Mentor engineers, raise technical capability and provide hands‑on support for complex troubleshooting and engineering decisions.
• Collaborate with product, analytics, AI/ML, platform and source-system teams to deliver reusable, trusted data capabilities.

Qualifications

• Bachelor's degree in Computer Science , Engineering, Information Technology or a related discipline; a Master's degree is desirable.
• Minimum of 7+ years of experience in data architecture, data engineering, or a similar role, with a strong focus on designing large-scale data platforms.
• Advanced hands‑on expertise in SQL, Python, Spark, distributed data processing and data modelling.
• Strong experience with cloud data platforms (Azure, AWS or GCP); Databricks/ lakehouse experience is preferred.
• Deep knowledge of batch and streaming ingestion, CDC, orchestration, APIs, data lakes/warehouses and modern data architecture patterns.
• Strong experience with Git, CI/CD, infrastructure-as-code, automated testing, observability and production engineering practices.
• Working knowledge of data governance, security, privacy, lineage, metadata management and data quality controls.
• Demonstrated ability to lead architecture/design reviews, resolve complex technical issues, mentor engineers and influence senior stakeholders.

Key Skills

• Strong leadership, collaboration, and communication skills.
• Expertise in cloud platforms and services (Azure preferred).
• Proficiency in data pipeline orchestration tools (e.g., Apache Airflow, Azure Data Factory).
• Knowledge of containerization and microservices architecture.
• Familiarity with data visualization and BI tools (e.g., Power BI, Tableau).
• Experience with infrastructure-as-code tools (e.g., Terraform, CloudFormation).
• Ability to think strategically while balancing business needs and technical solutions.
• Experience with Agile methodologies and working in a fast‑paced, collaborative environment.

Desirable Qualifications

• Certifications such as Microsoft Certified: Azure Solutions Architect Expert or Google Cloud Professional Data Engineer.
• Experience with machine learning and AI workloads on data platforms.
• Knowledge of DevOps practices and CI/CD for data pipelines.