Walk-in | Azure Data Engineer

Vidpro Consultancy Services; India; Remote; INR 1,400,000 - 2,100,000; Full time

Job summary

Vidpro Consultancy Services in India seeks a Databricks Data Engineer to design and optimize data processing pipelines on the Databricks platform, turning raw data into actionable insights. You will craft scalable data workflows and collaborate with analysts, data scientists, and stakeholders to meet data requirements and ensure quality.

Join a team that values cloud proficiency, data modeling, and cutting-edge data engineering practices to enable data-driven decisions across the organization.

Qualifications

• Minimum 5 years of data engineering experience.
• Experience designing data pipelines on Databricks.
• Strong knowledge of SQL and data warehousing concepts.

Responsibilities

• Develop and maintain data pipelines using Databricks, Spark, and other data processing technologies.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and accessibility.
• Implement and optimize ETL processes to support analytics and reporting initiatives.
• Monitor and troubleshoot data pipeline performance, ensuring timely delivery and reliability.
• Contribute to the design and implementation of data architecture and data models.
• Utilize Databricks notebooks for data exploration, visualization, and sharing insights with team members.
• Stay updated with the latest trends and advancements in data engineering and the Databricks platform.

Skills

Databricks

Apache Spark

Python

Scala

ETL

Data modeling

Data warehousing

SQL

Tableau

Power BI

Cloud platforms

AWS

Azure

GCP

Job description

Exp: 5 - 10 Yrs

Location: Hyderabad, Bangalore, Pune, Chennai and Gurgaon

Skills: Python, Pyspark, SQL, Databricks, Unity catalog, Azure Datafactory, Azure Synapse, ETL, Orchestration, Migration and Medallion Architecture

As a Databricks Data Engineer, you will play a crucial role in designing, developing, and optimizing data processing pipelines on the Databricks platform. You will be responsible for transforming raw data into actionable insights and enabling data-driven decision-making across the organization. Your expertise in data engineering and proficiency with the Databricks environment will be essential in building scalable and efficient data workflows.

Key Responsibilities:

• Develop and maintain data pipelines using Databricks, Spark, and other data processing technologies.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and accessibility.
• Implement and optimize ETL processes to support analytics and reporting initiatives.
• Monitor and troubleshoot data pipeline performance, ensuring timely delivery and reliability.
• Contribute to the design and implementation of data architecture and data models.
• Utilize Databricks notebooks for data exploration, visualization, and sharing insights with team members.
• Stay updated with the latest trends and advancements in data engineering and the Databricks platform.

Skills and Tools Required:

• Proficiency in Databricks and Apache Spark.
• Strong programming skills in Python or Scala for data processing.
• Experience with ETL processes and tools.
• Knowledge of data modeling and data warehousing concepts.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud.
• Experience with SQL and database technologies (e.g., MySQL, PostgreSQL, or others).
• Familiarity with data visualization tools (e.g., Tableau, Power BI) is a plus.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills to work effectively within a team.