LEAD DATA ENGINEER - Azure Databricks

Happiest Minds Technologies; Hyderabad; Sur place; INR 2 000 000 - 4 000 000; Plein temps

Résumé du poste

Happiest Minds Technologies in Hyderabad is seeking a Lead Data Engineer with 5–8 years of experience to design and implement robust data pipelines. You will work with Azure Databricks, Azure Data Factory, and Palantir to deliver scalable data solutions that empower business insights.

The role requires strong data modeling, ETL, and cloud data platform expertise, plus Python/SQL/Scala coding and cross-functional collaboration to drive successful data initiatives.

Qualifications

• Bachelor's degree in CS/IT or related field.
• 5-8 years of data engineering experience.
• Experience with Azure Databricks, Azure Data Factory and Palantir.
• Strong SQL/Python/Scala skills.
• Knowledge of data modeling, ETL, and data warehousing.

Responsabilités

• Design, build, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Collaborate with data scientists, analysts, and stakeholders to understand data needs.
• Ingest, transform, and store data into data lakes and warehouses; ensure quality and performance.
• Mentor junior engineers and share knowledge across the team.

Connaissances

Data modeling & ETL concepts

Python/SQL/Scala

Generative AI & ML concepts

Neural network fundamentals

Communication & collaboration

Formation

Bachelor's degree in Computer Science, IT, or related field

Outils

Azure Databricks

Azure Data Factory

Palantir

Description du poste

Job Title: Lead Data Engineer

Years of Experience: 5-8 Years

Job Summary: We are seeking a highly skilled Senior Data Engineer with extensive experience in Azure Databricks, Azure Data Factory, and Palantir to join our dynamic team in Hyderabad. The ideal candidate will have a strong background in data engineering, with a proven track record of designing and implementing data solutions that drive business insights and decision-making. This role requires a deep understanding of data architecture, ETL processes, and cloud-based data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Implement data integration processes to ingest, transform, and store data from various sources into data lakes and warehouses.
• Optimize data workflows and ensure data quality, reliability, and performance.
• Utilize Palantir for data analysis and visualization, providing actionable insights to the business.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.
• Stay updated with the latest industry trends and technologies related to data engineering and cloud computing.
• Mentor junior data engineers and contribute to team knowledge sharing.

Mandatory Skills

• Proven experience with Azure Databricks (deal breaker skill).
• Strong proficiency in Azure Data Factory for data integration and orchestration.
• Experience with Palantir for data analysis and visualization.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Proficient in programming languages such as Python, SQL, or Scala.
• Good understanding of Generative AI and Machine Learning concepts, including LLM, RAG, Vector Databases, Embeddings, Token Management, Feature Selection, Model Evaluation, and Neural Network fundamentals
• Good to have : Ability to demonstrate practical knowledge of ML algorithms such as KNN, Linear Regression, CNN

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Familiarity with big data technologies like Apache Spark, Hadoop, or Kafka.
• Knowledge of data governance and data security best practices.
• Experience with CI/CD pipelines and DevOps practices in data engineering.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5-8 years of experience in data engineering or a related role.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.