Databricks Engineer, Data Platform & Data Science Tooling

Magna International; Bengaluru; On-site; INR 5,500,000 - 7,500,000; Full time

Job summary

Magna International in Bengaluru seeks a seasoned data engineer to design, build, and optimize data pipelines using Databricks Delta Live Tables and Unity Catalog. You will enable AI/ML consumption through structured and unstructured data pipelines, with strong PySpark and SQL skills.

Expect collaboration with data scientists and analysts to deliver scalable data solutions, ensure data quality, and contribute to Power BI semantic layers.

Qualifications

• 4+ years hands-on data engineering experience.
• 2+ years Azure Databricks in production.
• Proven ability to build and deliver pipelines.
• Experience with diverse data sources (relational, file, API).
• Strong SQL and PySpark coding skills.

Responsibilities

• Design, develop, and implement data pipelines using Databricks Delta Live Tables.
• Build pipelines for structured and unstructured data to support AI/ML.
• Develop data models including fact/dimension tables and data marts.
• Write modular PySpark and SQL logic, testable and CI/CD deployable.
• Write and optimize complex SQL queries with Databricks SQL.
• Manage data governance and metadata with Unity Catalog.
• Collaborate with data scientists and analysts to meet data needs.
• Monitor, troubleshoot, and optimize pipelines and infra.
• Stay current with Databricks ecosystem and related trends.
• Contribute to semantic layer powering Power BI dashboards.

Skills

PySpark

Delta Lake

Workflows

Unity Catalog

Databricks

Delta Live Tables

Tools

Azure DevOps

GitHub Actions

Terraform

Power BI

Job description

Job Responsibilities

Job Responsibilities – Must Have

• Non-negotiable skill: 3+ years hands-on: PySpark, Delta Lake, Workflows, Unity Catalog, Databricks
• Design, develop, and implement efficient and reliable data pipelines using Databricks Delta Live Tables.
• Develop pipelines for structured and unstructured data (i.e. documents, JSON, Parquet, Excel) supporting AI and ML consumption downstream.
• Implement and extend data models (i.e. fact/dimension tables, domain data marts) following designs defined by the Senior DE and AI team.
• Write clean, modular, reusable PySpark and SQL transformation logic that is testable, documented, and deployable via CI/CD.
• Write and optimize complex SQL queries using Databricks SQL for data extraction, transformation, and loading.
• Implement and manage data governance and metadata using Databricks Unity Catalog to ensure data quality and discoverability.
• Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver appropriate data solutions.
• Monitor, troubleshoot, and optimize existing data pipelines and data infrastructure.
• Stay up-to-date with the latest trends and technologies in data engineering and the Databricks ecosystem.
• Contribute to the semantic layer that powers Power BI dashboards.

Orchestration and Data Ops–Should Have

• Build and manage Databricks Workflows: configuring task dependencies, retry policies, and failure alerting.
• Monitor & manage workspaces as Workspace admins.
• Follow and contribute to CI/CD practices: version control, pull requests, automated testing, and deployment to Dev/QA/Prod environments using Azure DevOps or GitHub Actions.
• Understanding of infrastructure as a service, Terraform and willingness to learn terraform for automation at infrastructure level.
• Package and deploy reusable logic as Python libraries following team standards.
• Monitor pipeline health, investigate failures, and resolve data issues within SLA.

FinOps Awareness–Should Have

• Write cost-conscious PySpark avoiding unnecessary full scans, optimizing joins, using appropriate cluster types.
• Apply Delta table best practices (i.e. VACUUM, OPTIMIZE, compaction) to manage storage costs.
• Follow cluster policies defined by platform leads and flag unusual resource consumption.

Experience / QUALIFICATIONS

• 4-6+ years of overall data engineering experience.
• 2+ years of hands-on Azure Databricks experience in production environments.
• Demonstrated ability to build and deliver pipelines — not just maintain or support them.
• Experience working within a defined architecture and contributing to its improvement.
• Comfortable working with multiple data source types — relational, file-based, API.

Awareness, Unity, Empowerment

At Magna, we believe that a diverse workforce is critical to our success. That’s why we are proud to be an equal opportunity employer. We hire on the basis of experience and qualifications, and in consideration of job requirements, regardless of, in particular, color, ancestry, religion, gender, origin, sexual orientation, age, citizenship, marital status, disability or gender identity. Magna takes the privacy of your personal information seriously. We discourage you from sending applications via email or traditional mail to comply with GDPR requirements and your local Data Privacy Law.

Regular / Permanent

Magna Corporate