Lead Data Engineer

ThinkWise Consulting LLP; Hyderabad; Sur place; INR 2 000 000 - 4 000 000; Plein temps

Résumé du poste

ThinkWise Consulting LLP is seeking an experienced Data Engineer in Hyderabad to design, develop, and optimize scalable data pipelines and analytics solutions using Databricks, Microsoft Fabric, Azure Data Factory, or Snowflake. The role emphasizes Medallion Architecture, data lakehouse concepts, and seamless Power BI integration.

The candidate will work across cloud environments, implement data quality, governance, and CI/CD practices, and collaborate with cross-functional teams to drive

Qualifications

• 5–10 years of experience in data engineering.
• Hands-on with at least one data platform (Databricks, Fabric, or ADF).
• Strong SQL skills and experience with data modeling.

Responsabilités

• Design, develop, and maintain ETL/ELT pipelines using Databricks, Fabric, ADF, or Snowflake.
• Build and optimize data workflows for performance, scalability, and cost efficiency in cloud environments.
• Develop data processing and transformation solutions using Python and SQL.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data pipelines.
• Design and maintain data lakes, warehouses, and lakehouse architectures.
• Develop Power BI dashboards and reports to deliver actionable insights.
• Collaborate with cross-functional teams on data requirements and governance.

Connaissances

Databricks

Microsoft Fabric

Snowflake

Power BI

Python

PySpark

SQL

Medallion Architecture

Data Pipelines

Outils

Azure Data Factory

Git

Databricks optimization

Description du poste

we are looking for an experienced Data Engineer with strong expertise in Databricks, Microsoft Fabric, or Snowflake, along with Power BI. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines, implementing modern data architectures, and delivering analytics solutions to support business decision-making.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Databricks, Microsoft Fabric, Azure Data Factory (ADF), or Snowflake.
• Build and optimize data workflows for performance, scalability, and cost efficiency across cloud environments.
• Develop data processing and transformation solutions using Python, PySpark, and SQL.
• Implement Bronze, Silver, and Gold layers using Medallion Architecture.
• Design and maintain data lakes, data warehouses, and lakehouse architectures.
• Develop Power BI dashboards and reports to deliver actionable business insights.
• Integrate data engineering solutions with Power BI and downstream analytics applications.
• Implement data quality checks and ensure data integrity, security, and reliability.
• Collaborate with cross-functional teams to define data requirements, governance standards, and engineering best practices.
• Automate workflows and support CI/CD deployments using Git and DevOps practices.
• Monitor, troubleshoot, and optimize data pipelines and dashboards.
• Support data migration projects from legacy platforms to modern data platforms.

Required Technical Skills

• Data Platforms: Hands-on experience with at least one of Databricks, Microsoft Fabric, or Azure Data Factory.
• SQL: Strong SQL skills, including complex queries and performance optimization.
• Power BI: Hands-on experience with data modeling, DAX, dashboard development, and performance optimization.
• Data Architecture: Strong understanding of Medallion Architecture, Star Schema, dimensional modeling, and lakehouse concepts.
• Cloud Platforms: Experience with at least one major cloud platform: Azure, AWS, or GCP.
• Microsoft Fabric: Hands-on experience with Dataflow Gen2 for low-code data ingestion and transformation.
• Data Migration: Experience migrating data from legacy platforms, such as Informatica to Fabric, SSIS to ADF, or legacy systems to Databricks/Fabric.

Preferred Skills

• Experience with Delta Lake and modern cloud data architectures.
• Knowledge of Azure Synapse Analytics or Snowflake.
• Experience with Databricks platform optimization and its ecosystem.
• Familiarity with CI/CD, Git, DevOps, and Infrastructure as Code.
• Understanding of data governance, lineage, security, and compliance.
• Experience with streaming technologies such as Kafka, Azure Event Hubs, or Amazon Kinesis.
• Exposure to machine learning pipelines, advanced analytics, AI Ops, or GenAI data workloads.
• Databricks professional certification is an advantage but not mandatory.

Experience Requirements

• 5-10 years of relevant experience in Data Engineering.