Data Engineer Advisor

Mployee.me

Hyderabad

Presencial

INR 400.000 - 700.000

Jornada completa

Hace 9 días

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

Mployee.me in Hyderabad, India seeks an experienced Hands-On Data Tech Lead to drive modernization of our enterprise data platform using Azure Databricks, PySpark, and Delta Lake. You will lead the migration of legacy SSIS packages and architect end-to-end ELT/ETL pipelines with cloud-native scalability.

The role blends deep coding with technical leadership as you mentor engineers, collaborate with data architects, BI analysts, and DevOps, and implement CI/CD for data pipelines.

Formación

  • 8–10 years of experience with SSIS, SQL Server and modern cloud data stack.
  • Hands-on data tech lead with mentoring and architectural skills.
  • Experience migrating ETL pipelines to Azure Databricks/PySpark Lakehouse.
  • Strong knowledge of MDW/Medallion architectures in Azure.
  • Familiar with Delta Lake, ADF, and data governance practices.
  • Proficient in T-SQL, stored procedures, and performance tuning.
  • Experience with CI/CD for data pipelines (Azure DevOps or GitHub Actions).

Responsabilidades

  • Migrate legacy SSIS to cloud-native Databricks/PySpark Lakehouse.
  • Architect End-to-End Modern Data Warehouse with Bronze/Silver/Gold.
  • Mentor engineers; enforce design patterns and coding standards.
  • Collaborate with Data Architects, BAs and DevOps to translate requirements.
  • Develop and optimize batch and real-time ETL/ELT with PySpark on Databricks.
  • Refactor on-prem SSIS and T-SQL into scalable cloud pipelines.
  • Implement Delta Lake features, schema evolution, and time travel.
  • Integrate ADF with on-prem and ADLS Gen2 for workflows.
  • Tune Spark configurations to reduce compute overhead and runtimes.
  • Enforce data quality with automated validation, reconciliation, logging.
  • Set up CI/CD automation for deployment pipelines.

Conocimientos

Azure Databricks
PySpark
Delta Lake
Unity Catalog
ADLS Gen2
Python
SQL Server
SSIS
Azure Data Factory
Data Modeling
Git / CI-CD
Azure DevOps
Data Governance
Time-travel

Herramientas

Azure Databricks
PySpark
SQL Server
SSIS
Azure Data Factory
Delta Lake
ADLS Gen2
Git
Azure DevOps
CI/CD

Descripción del empleo

India T&M Special Bonus
Job Summary

A highly skilled 8 to 10 years of experience on SSIS. SQL Server, Azure Databricks, Pyspark, Hands-On Data Tech Lead to drive the modernization and scaling of our enterprise data platform to provide technical leadership while remaining deeply technical and actively writing code. Also lead the migration, integration, and development of data pipelines across our legacy SQLServer/SSIS environments and our modern cloud infrastructure built on Azure Databricks and PySpark and act as the principal architect and developer for end-to-end ELT/ETL pipelines, ensuring optimal performance, data governance, and scalability.

  • Lead the migration of legacy SQL Server Integration Services (SSIS) packages to modern, cloud-native Azure Databricks / PySpark Lakehouse environments.
  • Architect end-to-end Modern Data Warehouse (MDW) and Medallion architecture systems (Bronze, Silver, Gold layers) within Azure.
  • Mentor and guide a team of data engineers, conducting stringent code reviews, enforcing design patterns, and establishing development standards.
  • Collaborate with stakeholders including Data Architects, Business Analysts, and DevOps teams to translate complex business requirements into technical blueprints.
  • Develop and optimize scalable, modular, high-performance batch and real-time ETL/ELT pipelines using PySpark on Azure Databricks cluster environments.
  • Maintain and refactor existing on-premise SSIS packages and complex T-SQL code (stored procedures, views, performance tuning) to ensure business continuity during transition phases.
  • Implement Delta Lake features, managing transactions, schema evolution, and time-travel querying functionalities.
  • Orchestrate workflows seamlessly by integrating Azure Data Factory (ADF) with on-premise databases, cloud storage (ADLS Gen2), and Databricks activities.
  • Optimize Spark configurations and cluster utilization to drastically reduce cloud compute overhead and job execution windows.
  • Enforce data quality frameworks encompassing automated data validation, error reconciliation, and operational audit logging.
  • Implement CI/CD automations via Azure DevOps or GitHub Actions to manage environments, code repositories (Git), and deployment pipelines.
Skill Requirements
Technical Skills
  • Azure Big Data Stack: Deep, production-level knowledge of Azure Databricks, Delta Lake, Unity Catalog, and Azure Data Lake Storage (ADLS Gen2).
  • Programming Languages: Advanced proficiency in Python and PySpark core APIs (Spark SQL and DataFrames).
  • Legacy ETL & Databases: Mastery of SQL Server, Advanced T-SQL writing, and extensive experience configuring, deploying, and troubleshooting SSIS packages.
  • Orchestration: Hands-on experience mapping dependencies and scheduling triggers using Azure Data Factory (ADF).
  • Data Modeling: Expert understanding of relational (OLTP) and dimensional (OLAP) schemas, explicitly Star and Snowflake architectures.
  • DevOps & Security: Competency with Git branching, CI/CD automated release deployments, and secure data handling mechanisms (Azure Key Vault, network isolation).
Soft Skills & Leadership
  • Proven track record as a coding lead who splits time between management/design and hands-on execution.
  • Excellent cross-functional communication skills, comfortably discussing data strategies with both executive leaders and junior technical staff.
  • Strong debugging and root-cause analysis (RCA) skills for multi-tier platform failures.
  • Preferred Certifications: Microsoft Certified: Azure Data or Databricks
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Data Engineer Advisor
Data Engineer Advisor

NTT DATA, Inc. • Hyderabad

Presencial
INR 3.000.000 - 4.500.000
Data Engineer Advisor
Data Engineer Advisor

NTT DATA, Inc. • Hyderabad

Presencial
INR 4.000.000 - 6.000.000
Data Engg With Databricks PySpark -Sr Technical Lead-Data Engg
Data Engg With Databricks PySpark -Sr Technical Lead-Data Engg

Birlasoft • Pune District

Presencial
INR 1.500.000 - 2.200.000
Senior Databricks Engineer
Senior Databricks Engineer

DATABEAT • Hyderabad

Presencial
INR 1.500.000 - 2.500.000
Data Engg With Databricks - Technical Lead-Data Engg
Data Engg With Databricks - Technical Lead-Data Engg

Birlasoft • Pune District

Presencial
INR 1.200.000 - 1.800.000
Senior Technical Lead – Azure Data Engineer
Senior Technical Lead – Azure Data Engineer

USEReady Inc. • Bengaluru

Híbrido
INR 5.000.000 - 7.500.000
Sr. Databricks Engineer
Sr. Databricks Engineer

ExlService Holdings, Inc. • Pune District

Híbrido
INR 3.500.000 - 6.500.000
Data Engineer
Data Engineer

Intact Insurance • India

Presencial
INR 1.500.000 - 2.300.000
Databricks Engineer
Databricks Engineer

DATAECONOMY Inc • Pune District

Presencial
INR 2.500.000 - 4.000.000
Data Engineering Consultant - ADB, ADF, Azure, Spark and SQL
Data Engineering Consultant - ADB, ADF, Azure, Spark and SQL

Optum • Bengaluru

Presencial
INR 2.500.000 - 6.000.000