Senior Azure Data Engineer

Vidpro Consultancy Services; Hyderabad, Pune District, Bengaluru; Hybrid; INR 3,500,000 - 6,500,000; Full time

Job summary

Vidpro Consultancy Services in India seeks a seasoned Data Engineer to lead design, build, and deployment of Databricks-based data pipelines. You will collaborate with data scientists, analysts, and business teams to translate requirements into scalable, reliable solutions.

The role involves ensuring data quality, implementing DevOps CI/CD practices, mentoring juniors, and driving performance and security across data platforms. Hybrid work across Bangalore, Hyderabad, and Pune is offered.

Qualifications

• Minimum of 7 years of data engineering experience.
• Extensive experience with Databricks and cloud platforms.
• Strong knowledge of data pipeline architectures and ETL/ELT processes.
• Proficient in Python or Scala.
• Experience with Spark, Kafka and other big data technologies.
• Excellent problem-solving and leadership skills.

Responsibilities

• Lead design, development, and implementation of data engineering solutions utilizing Databricks.
• Collaborate with data scientists, analysts, and stakeholders to translate requirements into technical solutions.
• Oversee integration of data sources and systems, ensuring data quality and accessibility.
• Implement DevOps practices for CI/CD and pipeline optimization.
• Mentor and guide junior data engineers; promote best practices.
• Monitor and troubleshoot data infrastructure for performance and reliability.
• Stay current with industry trends to drive innovation in data engineering.
• Ensure adherence to security and compliance standards across data projects.

Skills

Databricks

Python

PySpark

SQL

ETL

Scala

Kafka

DevOps

CI/CD

Azure Cloud

Education

Bachelor's degree in CS/Engineering

Master's degree preferred

Tools

Spark

Git

Job description

Experience: 7 - 10 Yrs

Location: Bangalore, Hyderabad & Pune

Work Mode: Hybrid - 8 days in a month

Mandatory Skills: Databricks, Python, PySpark, SQL, ETL, Scala / Kafka, Devops, CI/CD and Azure Cloud.

Responsibilities:

• Lead the design, development, and implementation of data engineering solutions utilizing Databricks.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and transform business needs into technical solutions.
• Oversee the integration of various data sources and systems, ensuring data quality, consistency, and accessibility.
• Implement DevOps practices for continuous integration and deployment, optimizing the data engineering pipeline.
• Mentor and guide junior data engineers, promoting best practices in data management and engineering methodologies.
• Monitor and troubleshoot data infrastructure for performance optimization and reliability.
• Stay current with industry trends and emerging technologies to drive innovation within the data engineering team.
• Ensure adherence to security and compliance standards across all data projects.

Qualifications:

• Bachelor's degree in computer science, Engineering, or a related field; master's degree preferred.
• Minimum of 7 years of experience in data engineering, with extensive experience in Databricks and cloud platforms.
• Strong knowledge of data pipeline architectures and ETL/ELT processes.
• Proven experience with DevOps tools and practices, including CI/CD methodologies.
• Proficiency in programming languages such as Python or Scala.
• Experience with big data technologies, such as Spark, Kafka, or similar frameworks.
• Excellent problem-solving skills and ability to work in a fast-paced environment.
• Exceptional communication and leadership skills, with the ability to collaborate effectively with diverse teams.