Data Engineer

Doist; Chennai District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Orion Innovation is seeking a senior Data Engineer specialized in Databricks migration to lead complex legacy-to-Databricks transformations. You will convert legacy SQL/T-SQL logic into scalable Databricks and PySpark pipelines, design robust ETL/ELT workflows, and validate data quality during migration.

The role requires deep data warehousing knowledge, strong communication, and collaboration with architects and migration teams to ensure successful delivery.

Qualifications

• 10+ years of Data Engineering experience.
• Hands-on Databricks, PySpark, and SQL/T-SQL.
• Experience migrating legacy SQL/ETL to Databricks/PySpark.
• Proficient in data warehousing concepts and ETL/ELT practices.
• Strong communication with technical and business teams.

Responsibilities

• Analyze legacy data warehouse and migrate to Databricks.
• Read and interpret complex T-SQL scripts and transforms.
• Convert legacy logic to Databricks and PySpark workflows.
• Design and maintain scalable ETL/ELT pipelines in Databricks.
• Ensure data quality, testing, and validation during migration.
• Collaborate with architects, engineers, and migration teams.

Skills

Databricks

PySpark

SQL/T-SQL

Oracle

SQL Server

ETL/ELT

Data Warehousing

Azure Data Factory

Delta Lake

Unity Catalog

Azure Synapse Analytics

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

Azure Data Factory

Delta Lake

Unity Catalog

Azure Synapse Analytics

CI/CD

Job description

Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity.

We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.

Data Engineer Location- Kochi/Coimbatore/Chennai

Job Overview

We are seeking a strong Data Engineer - Databricks Migration with 10+ years of experience and hands-on expertise in Databricks, PySpark, and SQL/T-SQL interpretation to support a legacy-to-Databricks migration program. The ideal candidate should have worked predominantly as a Data Engineer throughout their career, ideally beginning with Oracle, SQL Server/T-SQL, ETL, or Data Warehouse technologies and later transitioning into modern cloud-based Data Engineering. The role requires the ability to understand existing T-SQL logic and migrate transformation rules into scalable Databricks and PySpark solutions. Important screening criterion: The candidate must have worked predominantly and consistently as a Data Engineer throughout their career. Profiles that are primarily BI, reporting, dashboarding, analytics, or only recently shifted into data engineering should not be prioritized.

Key Responsibilities

• Analyze legacy data warehouse, SQL Server, Oracle, ETL, and T-SQL based implementations for migration to Databricks.
• Read and interpret complex T-SQL scripts, stored procedures, functions, views, joins, transformation rules, and data flows.
• Convert business logic from legacy SQL/T-SQL processes into optimized Databricks and PySpark workflows.
• Design, develop, and maintain scalable ETL/ELT pipelines in Databricks.
• Support ingestion, transformation, validation, and reconciliation of enterprise data across legacy and modern platforms.
• Collaborate with architects, senior engineers, business analysts, and migration teams to understand source-to-target mapping requirements.
• Perform data quality checks, unit testing, integration testing, and defect resolution during migration execution.
• Optimize PySpark jobs, SQL queries, and Databricks notebooks for better performance and reliability.
• Document migration logic, transformation rules, dependencies, and technical design details.
• Participate in code reviews and follow Data Engineering best practices.

Key Skills

• 5+ years of relevant Data Engineering experience, with the majority of career spent in Data Engineering roles.
• Strong hands-on experience in Databricks development.
• Strong PySpark programming experience for data transformation and pipeline development.
• Strong SQL/T-SQL understanding, especially the ability to read and interpret existing T-SQL code.
• Experience in legacy data technologies such as SQL Server, Oracle, traditional ETL tools, or Data Warehouse platforms.
• Experience in ETL/ELT development, data ingestion, transformation, and data pipeline implementation.
• Good understanding of Data Warehousing, Data Modeling, and dimensional concepts.
• Ability to migrate or re-engineer legacy SQL/ETL logic into Databricks/ PySpark .
• Strong analytical, debugging, and problem-solving skills.
• Good communication skills to work with technical and business teams.

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science , Information Technology, Engineering, or a related discipline.
• Azure Data Factory or Azure Data Platform exposure.
• Delta Lake experience.
• Unity Catalog exposure.
• Azure Synapse Analytics experience.
• CI/CD exposure for data pipelines.
• Experience working in Agile/Scrum delivery models.
• Data governance, security, and lineage awareness.
• Relevant certifications in Databricks, Azure Data Engineering, or Cloud Data Platforms will be an added advantage.
• Strong, hands-on Data Engineering background throughout career, not a partial or recent transition.
• Avoid profiles that are mainly BI Developers, Power BI Developers, Data Analysts, or reporting-only candidates.

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Candidate Privacy Policy Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains: What information we collect during our application and recruitment process and why we collect it; How we handle that information; and How to access and update that information. Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.