Azure Databricks

VMC Soft Technologies, Inc; Akola District; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

VMC Soft Technologies, Inc is seeking an experienced data engineer to design, develop, and optimize end-to-end data pipelines in Azure. You will leverage Data Factory, Data Lake Gen 2, and Databricks to deliver scalable data solutions and analytics for business insights.

You will also create Power BI dashboards, collaborate with cross-functional teams, and ensure data quality, security, and compliance across projects. 7+ years of Azure data experience preferred.

Qualifications

• Bachelor's degree in CS/IT or related field.
• 7+ years of relevant experience with Azure data services.
• Strong analytical and problem solving skills.
• Effective communication and teamwork abilities.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Manage and optimize data storage using Azure Data Lake Gen 2.
• Build and process large-scale data solutions with Azure Databricks and Spark.
• Create interactive reports and dashboards in Power BI for business insights.
• Collaborate with IT and business teams to translate requirements into data solutions.
• Ensure data quality, security, and compliance.

Skills

Azure Data Factory

Azure Data Lake Gen 2

Azure Databricks / Apache Spark

Power BI

Education

Bachelor's degree in Computer Science, Information Technology, or related field

Tools

CI/CD Pipelines

DP-203 Certification

Unity Catalog

Job description

Role: Azure Databricks, PySpark, Azure Data Factory

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory.
• Manage and optimize data storage using Azure Data Lake Gen 2.
• Build and process large-scale data solutions with Azure Databricks and Apache Spark.
• Create interactive reports and dashboards in Power BI for business insights.
• Collaborate with cross-functional IT and business teams to translate requirements into data solutions.
• Ensure data quality, security, and compliance in line with organizational standards.

Core Skills Required

• Azure Data Factory: Experience in building and managing data pipelines.
• Azure Data Lake Gen 2: Proficient in data storage and management within Azure's data lake environment.
• Azure Databricks / Apache Spark: Hands-on skill with distributed data processing, transformations, and analytics.
• Power BI: Expertise in data visualization and reporting.

Nice-to-Have Skills

• Basic SQL Performance Tuning: Ability to write and optimize SQL queries.
• Data Governance & Unity Catalog: Understanding of data governance principles and experience with Unity Catalog for data management.
• Certification: Microsoft DP-203 (Azure Data Engineer Associate) certification.
• CI/CD Pipelines: Experience implementing CI/CD pipelines for Azure Data Factory or Databricks projects.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field (or equivalent experience).
• 4+ years of professional experience with Azure data services.
• Strong analytical and problem-solving skills.
• Effective communication and team collaboration abilities.

Preferred Traits

• Continuous learner, staying updated with Azure and data engineering advancements.
• Experience working in Agile environments.
• Passion for optimizing data workflows and enabling data-driven decisions
• Experience Required: 7+ relevant exp mandatory, anything below this will be rejected

Location and Work Mode

• Location : Pune
• Work Mode : WFO
• Experience : 7+