SENIOR DATA ENGINEER - Azure Databricks

Happiest Minds Technologies; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Happiest Minds Technologies in Hyderabad is seeking a Senior Data Engineer with extensive experience in Azure Databricks, Azure Data Factory, and Palantir to join our growing data engineering team. The role emphasizes designing scalable data pipelines and collaborating with data scientists and stakeholders to deliver high-quality data solutions.

The position requires strong skills in ETL, data warehousing, and cloud platforms, with mentoring responsibilities for junior engineers and a drive to

Qualifications

• Proven experience with Azure Databricks (deal breaker skill).
• Strong proficiency in Azure Data Factory for data integration and orchestration.
• Experience with Palantir for data analysis and visualization.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Proficient in Python, SQL or Scala.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Implement data integration processes to ingest, transform, and store data from various sources into data lakes and warehouses.
• Optimize data workflows and ensure data quality, reliability, and performance.
• Utilize Palantir for data analysis and visualization, providing actionable insights to the business.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.
• Stay updated with the latest industry trends and technologies related to data engineering and cloud computing.
• Mentor junior data engineers and contribute to team knowledge sharing.

Skills

Azure Databricks

Azure Data Factory

Palantir

Data modeling

ETL

Data warehousing

Python

SQL

Scala

Education

Bachelor's degree in CS/IT

Tools

Azure

Spark

Hadoop

Kafka

Job description

Job Title: Senior Data Engineer

Years of Experience: 5-8 Years

Job Summary

We are seeking a highly skilled Senior Data Engineer with extensive experience in Azure Databricks, Azure Data Factory, and Palantir to join our dynamic team in Hyderabad. The ideal candidate will have a strong background in data engineering, with a proven track record of designing and implementing data solutions that drive business insights and decision-making. This role requires a deep understanding of data architecture, ETL processes, and cloud-based data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Implement data integration processes to ingest, transform, and store data from various sources into data lakes and warehouses.
• Optimize data workflows and ensure data quality, reliability, and performance.
• Utilize Palantir for data analysis and visualization, providing actionable insights to the business.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.
• Stay updated with the latest industry trends and technologies related to data engineering and cloud computing.
• Mentor junior data engineers and contribute to team knowledge sharing.

Mandatory Skills

• Proven experience with Azure Databricks (deal breaker skill).
• Strong proficiency in Azure Data Factory for data integration and orchestration.
• Experience with Palantir for data analysis and visualization.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Proficient in programming languages such as Python, SQL, or Scala.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Familiarity with big data technologies like Apache Spark, Hadoop, or Kafka.
• Knowledge of data governance and data security best practices.
• Experience with CI/CD pipelines and DevOps practices in data engineering.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 5-8 years of experience in data engineering or a related role.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.

If you are passionate about data engineering and want to work in a fast-paced environment, we encourage you to apply for this exciting opportunity.