Senior Data Engineer (Fabric and Databricks)

Orion Innovation; Chennai District; On-site; INR 2,000,000 - 3,000,000; Full time

Job summary

Orion Innovation is seeking an experienced Senior Data Engineer to design and optimize scalable data platforms, utilizing Microsoft Fabric and Databricks. This role involves creating end-to-end data solutions and collaborating with analytics and data science teams to deliver efficient data solutions.

The ideal candidate will have over 7 years of experience in data engineering, strong proficiency in PySpark and SQL, and expertise in cloud environments, particularly Azure.

Qualifications

• 7+ years of professional experience in data engineering or related discipline.
• Hands-on expertise with Microsoft Fabric, Databricks, or Snowflake.
• Advanced proficiency in PySpark and SQL.

Responsibilities

• Design, build, and maintain end-to-end data solutions.
• Implement scalable data ingestion and transformation.
• Collaborate with data scientists and ML engineers.

Skills

Microsoft Fabric

Databricks

PySpark

SQL

Data modeling

ETL/ELT patterns

Data governance

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science

Tools

Power BI

Tableau

Azure

Job description

Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.

Job Overview

We are seeking an experienced Senior Data Engineer to build and optimize scalable data platforms using Microsoft Fabric, Databricks (and/or Snowflake). The role focuses on designing reliable data pipelines, lakehouse and warehouse models, and semantic layers that enable enterprise analytics, BI, and AI/Gen AI use cases. You will work closely with analytics, BI, and data science teams to deliver high-quality, performant, and governed data solutions while driving best practices in data engineering, optimization, and platform design.

Key Responsibilities

• Design, build, and maintain end-to-end data solutions on Microsoft Fabric, Databricks, including Pipelines, Notebooks, Lakehouse, Data Warehouse, and Semantic Models.
• Implement scalable data ingestion, transformation, and loading (ETL/ELT) using Fabric Pipelines and PySpark.
• Develop robust data models and schemas optimized for analytics, reporting, and AI-driven consumption.
• Create and maintain semantic models to support Power BI and enterprise BI solutions.
• Engineer high-performance data solutions that meet requirements for throughput, scalability, quality, and security.
• Author efficient PySpark and SQL code for large-scale data transformation, data quality management, and business rule processing.
• Build reusable framework components for metadata-driven pipelines and automation.
• Optimize Lakehouse and Data Warehouse performance, including partitioning, indexing, Delta optimization, and compute tuning.
• Develop and maintain stored procedures and advanced SQL logic for operational workloads.
• Design and prepare feature-ready datasets for AI and GenAI applications.
• Collaborate with data scientists and ML engineers to productionize AI pipelines.
• Implement data governance and metadata practices required for responsible AI.
• Leverage Fabric, Databricks capabilities to orchestrate and monitor AI-related data workflows.
• Apply data governance, privacy, and security standards across all engineered assets.
• Implement monitoring, alerting, and observability best practices for pipelines and compute workloads.
• Drive data quality initiatives, including validation frameworks, profiling, and anomaly detection.
• Partner with analytics, BI, data science, and product teams to understand requirements and translate them into technical solutions.
• Mentor junior engineers and contribute to engineering standards and patterns.
• Participate in architecture reviews, technical design sessions, and roadmap planning.
• Develop and deliver dashboards and reports using Power BI and Tableau.

Required Skills

• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or equivalent experience.
• 7+ years of professional experience in data engineering or a related discipline.
• Mandatory hands-on expertise with Microsoft Fabric, Databricks or Snowflake including Pipelines, Notebooks, Lakehouse, Data Warehouse, and Semantic Models.
• Advanced proficiency in PySpark for distributed data processing.
• Strong command of SQL for analytical and operational workloads.
• Experience developing and optimizing stored procedures and complex SQL transformations.
• Understanding of GenAI architectures, vectorization, embedding pipelines, and data preparation for LLM use cases.
• Strong knowledge of data modeling, ETL/ELT patterns, and modern data lakehouse principles.
• Experience designing and optimizing large-scale data pipelines in cloud environments (Azure preferred).
• Excellent problem‑solving skills with the ability to analyze complex data workflows.
• Proficiency in creating interactive dashboards and reports using Power BI and Tableau.

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.