Senior Data Engineer

Adastra; India; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

A leading tech consultancy is seeking a Data Engineer to design and implement next-generation data platforms using Python and Databricks. The ideal candidate will have a strong background in data modeling, especially in building semantic layers programmatically. Responsibilities include overseeing the development of ETL/ELT processes, implementing data governance practices, and optimizing pipelines for performance. A bachelor's degree in a relevant field and a minimum of 5 years experience in Data Engineering are required.

Qualifications

• Minimum 5 years of professional experience in Data Engineering.
• Strong expertise in Python for data manipulation and modeling.
• Proven experience building Semantic/Serving layers programmatically.

Responsibilities

• Design and implement the Data/Semantic Layer using Python.
• Build the 'Gold' layer in Databricks using PySpark.
• Develop robust ETL/ELT processes to ingest data from diverse sources.

Skills

Python

SQL

Data Modeling

PySpark

Data Governance

Education

Bachelor’s or master’s degree in computer science, Engineering, or related field

Tools

Databricks

Azure DevOps

GitHub Actions

Job description

About the role

In this role, you will play a key role in designing and implementing next-generation data platforms. You will move beyond traditional ETL by focusing on the Semantic Layer, using Python to define complex business logic, metrics, and data models as code. You will bridge the gap between raw data and business consumption, ensuring consistent, governed data delivery via Databricks.

Your responsibilities

• Semantic Layer & Data Modeling (Python-first)
• Design and implement the Data/Semantic Layer using Python frameworks to define business metrics, dimensions, and KPIs as code.
• Build the "Gold" or "Serving" layer in Databricks using PySpark, embedding complex business logic directly into the data pipeline rather than downstream BI tools.
• Implement Metrics-as-Code practices to ensure consistent definitions across different analytical tools (Power BI, custom apps, data science models).
• Translate functional business requirements into Python-based transformation logic for the final serving layer.
• Design, build, and maintain scalable batch and streaming pipelines using Databricks Workflows and Apache Spark.
• Develop robust ETL/ELT processes to ingest data from diverse sources (IoT, ERPs, APIs) and move it through the Medallion Architecture (Bronze → Silver → Gold).
• Optimize Python/Spark code for performance, partition management, and cost efficiency.
• Contribute to the design of Lakehouse architectures, ensuring the semantic layer supports both self-service BI and advanced analytics.
• Implement data governance and quality checks within the Python pipeline code.
• Ensure the semantic layer aligns with data security standards (Row-Level Security, masking) and access controls.
• Work closely with Business Analysts to understand metric definitions and codify them in Python.
• Collaborate with Data Scientists to expose semantic features for Machine Learning models.

Your background

• Bachelor’s or master’s degree in computer science, Engineering, or related field.
• Minimum 5 years of professional experience in Data Engineering.
• Strong expertise in Python for data manipulation and modeling (pandas, PySpark).
• Proven experience building Semantic/Serving layers programmatically (not just drag-and-drop BI modeling).
• Experience with Databricks and the Delta Lake ecosystem.
• Familiarity with "Data-as-Code" or "Metrics-as-Code" concepts (e.g., using dbt with Python models, or custom Python semantic frameworks).
• Strong understanding of dimensional modeling (Star Schema) and how to implement it via Spark/Python.

Technical skills

• Languages: Python (Advanced), SQL (Strong).
• Semantic/Modeling: Building serving layers in PySpark, dbt (Python models), or Python-based metric layers.
• Data Architecture: Medallion Architecture (Bronze/Silver/Gold), Dimensional Modeling.
• DevOps: CI/CD for data pipelines (Azure DevOps/GitHub Actions), Git flow.
• Concepts: ACID transactions, Time Travel, Unity Catalog, Governance.