IT engineer data lakehouse M&E/QM

Continental; India; On-site; INR 1,500,000 - 2,000,000; Full time

Benefits offered by this job

Training opportunities

Mobile and flexible working models

Sabbaticals

Job summary

Continental is seeking a Data Engineer to design and develop scalable data pipelines in the Azure Databricks environment. The role includes implementing data products, applying rigorous software engineering practices, and ensuring high data quality and reliability.

The ideal candidate will have a degree in Computer Science or related fields and 3–6 years of relevant experience. Benefits include training opportunities and flexible working models.

Qualifications

• Degree in Computer Science, Data Engineering, or related field.
• 3–6 years of hands-on experience in data engineering roles.
• Experience in implementing complex data pipelines.

Responsibilities

• Design scalable data pipelines in Azure Databricks.
• Build data layers and ensure compatibility with reporting tools.
• Develop automated data validation tests.

Skills

Data Engineering

PySpark

Scala

Azure

CI/CD

Agile Methodologies

Data Quality

Education

Degree in Computer Science

Databricks DE Associate

Azure Data Engineer

Tools

IDE

Power BI

Job description

Job Description

Design, develop, and operate scalable and maintainable data pipelines in the Azure Databricks environment. Develop all technical artefacts as code, implemented in professional IDEs, with full version control and CI/CD automation. Enable data‑driven decision‑making in Manufacturing & Engineering (M&E) and Quality by ensuring high data availability, quality, and reliability. Implement data products and analytical assets using software engineering principles in close alignment with business domains and functional IT. Apply rigorous software engineering practices such as modular design, test‑driven development, and artifact reuse in all implementations.

Global delivery footprint; cross‑functional data engineering support across Manufacturing & Engineering domains. Collaboration with business stakeholders, functional IT partners, product owners, architects, ML/AI engineers, and Power BI developers. Agile, product‑team structure embedded in an enterprise‑scale Azure environment.

Main Tasks

• Design scalable batch and streaming pipelines in Azure Databricks using PySpark and/or Scala.
• Implement ingestion from structured and semi‑structured sources (e.g., SAP, APIs, flat files).
• Build bronze/silver/gold data layers following the defined lakehouse layering architecture and governance.
• Implement use‑case driven dimensional models (star/snowflake schema) tailored to M&E and Quality needs.
• Ensure compatibility with reporting tools (e.g., Power BI) via curated data marts and semantic models.
• Implement enterprise‑level data warehouse models (domain‑driven 3NF models) for M&E and Quality data, aligned with other business domains.
• Develop and apply master data management strategies (e.g., Slowly Changing Dimensions).
• Develop automated data validation tests using frameworks.
• Monitor pipeline health, identify anomalies, and implement quality thresholds.
• Define and implement meaningful data quality rules with source systems and business stakeholders, including related reports.
• Develop and structure pipelines using modular, reusable code in a professional IDE.
• Apply TDD principles with automated unit, integration, and validation tests.
• Integrate tests into CI/CD pipelines to enable fail‑fast deployment strategies.
• Commit all artifacts to version control with peer review and CI/CD integration.
• Work closely with Product Owners to refine user stories and define acceptance criteria.
• Translate business requirements into data contracts and technical specifications.
• Participate in agile events such as sprint planning, reviews, and retrospectives.
• Document pipeline logic, data contracts, and technical decisions in markdown or auto‑generated docs from code.
• Align designs with governance and metadata standards (e.g., Unity Catalog).
• Track lineage and audit trails through integrated tooling.
• Profile and tune data transformation performance.
• Reduce job execution times and optimize cluster resource usage.
• Refactor legacy pipelines or inefficient transformations to improve scalability.

Benefits

• Training opportunities
• Mobile and flexible working models
• Sabbaticals

Equal Opportunity Employer

We offer equal opportunities to everyone – regardless of age, gender, nationality, cultural background, disability, religion, ideology or sexual orientation.

Qualifications

• Degree in Computer Science, Data Engineering, Information Systems, or related discipline.
• Certifications in software development and data engineering (e.g., Databricks DE Associate, Azure Data Engineer, or relevant DevOps certifications).
• 3–6 years of hands‑on experience in data engineering roles in enterprise environments.
• Demonstrated experience building production‑grade codebases in IDEs, with test coverage and version control.
• Proven experience in implementing complex data pipelines and contributing to full lifecycle data projects (development to deployment).
• Experience in at least one business domain: Manufacturing & Engineering (M&E) and Quality or a comparable field.
• Experience working in international teams across multiple time zones and cultures, preferably with teams in India, Germany, and the Philippines.