Data Engineer

Deutsche Telekom Digital Labs; Gurugram District; Presencial; INR 1.400.000 - 2.100.000; Jornada completa

Descripción de la vacante

Deutsche Telekom Digital Labs in Gurugram (India) is seeking a seasoned Data Engineer to design, build, and optimize production data pipelines on Google Cloud Platform. You will create scalable batch and streaming solutions, implement robust data models, and drive data quality and governance across teams.

You will work with BigQuery, Dataflow/Beam, Spark, and tools like Apache Airflow, dbt, and Dataform to deliver reliable data products.

Formación

• 4+ years hands-on Data Engineering experience building production data solutions.
• Strong practical experience with Google Cloud Platform (GCP).
• Strong hands-on experience with BigQuery.
• Advanced SQL skills including complex joins, CTEs, window functions and query tuning.
• Strong Python development skills for data processing, automation and testing.
• Experience developing production ETL / ELT pipelines.
• Hands-on experience with Cloud Composer / Apache Airflow.
• Experience with Dataform and/or dbt.
• Experience designing batch and incremental processing pipelines.
• Experience with Dataflow / Apache Beam or Dataproc / Spark / PySpark.
• Strong understanding of data modelling, including normalization, denormalization and dimensional modelling.
• Experience working with large-scale datasets.
• Experience with incremental and idempotent pipeline patterns.
• Understanding of data contracts and schema evolution.
• Experience implementing data-quality validation and reconciliation.
• Understanding of metadata and data lineage.
• Experience with Git, automated testing and CI/CD.
• Experience troubleshooting and supporting production data pipelines.
• Ability to independently design solutions rather than only implement predefined specifications.

Responsabilidades

• Design, develop and maintain production-grade data pipelines on Google Cloud Platform.
• Build scalable batch and incremental data-processing solutions.
• Develop complex transformations using BigQuery, SQL and Dataform / dbt.
• Build and maintain data-processing workflows using Cloud Composer / Apache Airflow.
• Develop reusable transformation components, orchestration patterns, libraries and templates.
• Implement metadata-driven and configuration-driven processing where appropriate.
• Build distributed data-processing solutions using Dataflow / Apache Beam and/or Dataproc / Spark.
• Design data models for analytical and downstream data-product requirements.
• Implement incremental and idempotent processing patterns.
• Define and maintain schemas and data contracts.
• Implement automated data-quality checks, validation and reconciliation.
• Capture and integrate metadata and lineage into data-processing workflows.
• Optimise BigQuery queries and pipelines for performance and cloud cost.
• Implement automated testing and integrate data workloads with CI/CD pipelines.
• Build monitoring and operational controls for production pipelines.
• Troubleshoot production issues and perform root-cause analysis.
• Contribute to reusable data-platform capabilities and engineering standards.
• Collaborate with Data Engineers, Platform Engineers, DevOps, Architects and business stakeholders.

Conocimientos

BigQuery

SQL

Python

ETL/ELT

Airflow

dbt/Dataform

Dataflow/Beam

Dataproc/Spark

Data modelling

CI/CD

Herramientas

dbt

Dataform

Git

CI/CD tooling

Descripción del empleo

Responsibilities:

• Design, develop and maintain production-grade data pipelines on Google Cloud Platform.
• Build scalable batch and incremental data-processing solutions.
• Develop complex transformations using BigQuery, SQL and Dataform / dbt.
• Build and maintain data-processing workflows using Cloud Composer / Apache Airflow.
• Develop reusable transformation components, orchestration patterns, libraries and templates.
• Implement metadata-driven and configuration-driven processing where appropriate.
• Build distributed data-processing solutions using Dataflow / Apache Beam and/or Dataproc / Spark.
• Design data models for analytical and downstream data-product requirements.
• Implement incremental and idempotent processing patterns.
• Define and maintain schemas and data contracts.
• Implement automated data-quality checks, validation and reconciliation.
• Capture and integrate metadata and lineage into data-processing workflows.
• Optimise BigQuery queries and pipelines for performance and cloud cost.
• Implement automated testing and integrate data workloads with CI/CD pipelines.
• Build monitoring and operational controls for production pipelines.
• Troubleshoot production issues and perform root-cause analysis.
• Contribute to reusable data-platform capabilities and engineering standards.
• Collaborate with Data Engineers, Platform Engineers, DevOps, Architects and business stakeholders.

Requirements:

• 4+ years of hands-on Data Engineering experience building production data solutions.
• Strong practical experience with Google Cloud Platform (GCP).
• Strong hands-on experience with BigQuery.
• Advanced SQL skills including complex joins, CTEs, window functions and query tuning.
• Strong Python development skills for data processing, automation and testing.
• Experience developing production ETL / ELT pipelines.
• Hands-on experience with Cloud Composer / Apache Airflow.
• Experience with Dataform and/or dbt.
• Experience designing batch and incremental processing pipelines.
• Experience with Dataflow / Apache Beam or Dataproc / Spark / PySpark.
• Strong understanding of data modelling, including normalization, denormalization and dimensional modelling.
• Experience working with large-scale datasets.
• Experience with incremental and idempotent pipeline patterns.
• Understanding of data contracts and schema evolution.
• Experience implementing data-quality validation and reconciliation.
• Understanding of metadata and data lineage.
• Experience with Git, automated testing and CI/CD.
• Experience troubleshooting and supporting production data pipelines.
• Ability to independently design solutions rather than only implement predefined specifications.

Required skills:

• Cloud and Storage: Google Cloud Platform, BigQuery, and Google Cloud Storage.
• Transformation and Orchestration: Advanced SQL, Dataform / dbt, and Cloud Composer / Apache Airflow.
• Data Processing: Dataflow / Apache Beam, Dataproc / Spark / PySpark.
• Programming and Engineering: Python, Git, Automated testing, CI/CD, Monitoring and troubleshooting.
• Data Engineering Capabilities: ETL / ELT, Batch processing, Incremental and idempotent pipelines, Data modelling, Data contracts, Data quality and reconciliation, metadata and lineage, Query performance optimisation, Cloud-cost optimisation, and Reusable data-platform components.

Good to Have:

• Experience with Apache Iceberg or modern lakehouse table formats.
• Experience with streaming or event-driven processing.
• Practical understanding of Data Product / Data Mesh principles.
• Experience with metadata-driven or configuration-driven processing.
• Experience with Change Data Capture.
• Familiarity with Terraform / Infrastructure as Code.
• Experience building reusable components used by multiple engineering teams.

Strong Interpersonal Skills:

• Strong ownership mindset and ability to take engineering work through production.
• Strong analytical, debugging, and problem-solving skills.
• Ability to communicate technical decisions clearly.
• Comfortable participating in design and code reviews.
• Ability to collaborate with engineering and business stakeholders.
• Ability to work independently while seeking guidance for complex architectural decisions.
• Comfortable working within distributed and multicultural teams.