Sr. Data Engineer (Databricks & Cloud Data Platforms)

Datansh Solutions; Jaipur; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

Datansh Solutions in Jaipur, India is seeking a Data Engineer to design and implement the data layer and frameworks for business metrics. The role emphasizes Python programming and building serving layers using Databricks and PySpark.

The ideal candidate will have a minimum of 5 years' experience in Data Engineering, advanced skills in Python, and hands-on experience with Databricks. Responsibilities include collaborating with analysts, mentoring junior engineers, and ensuring data governance.

Qualifications

• Minimum 5 years of professional experience in Data Engineering.
• Advanced Python skills focused on data manipulation and modelling.
• Hands‑on experience with Databricks and Apache Spark.
• Knowledge of Data‑as‑Code practices.
• Solid understanding of dimensional modelling and Lakehouse architecture.

Responsibilities

• Collaborate with Business Analysts to understand metric definitions.
• Collaborate with Data Scientists for semantic features exposure.
• Mentor junior engineers in data modelling.
• Build scalable batch and streaming pipelines in Databricks.
• Implement data governance and quality checks within pipelines.

Skills

Python (Advanced)

SQL (Strong)

Databricks

Apache Spark (PySpark)

Data‑as‑Code practices

Tools

Azure Databricks

Delta Lake

Job description

Role Overview

Semantic Layer & Data Modelling (Python‑first) – design and implement the data/semantic layer and frameworks to define business metrics, dimensions, and KPIs as code. Build the Gold or Serving layer in Databricks using PySpark, embedding complex business logic directly into the data pipeline rather than downstream BI tools.

Implement Metrics‑as‑Code practices to ensure consistent definitions across different analytical tools (Power BI, custom apps, data science models). Translate functional business requirements into Python‑based transformation logic for the final serving layer.

Design, build, and maintain scalable batch and streaming pipelines in Databricks using Databricks Workflows and Apache Spark. Develop robust ETL/ELT processes to ingest data from diverse sources (IoT, ERPs, APIs) and move it through the Medallion Architecture (Bronze, Silver, Gold). Optimize Python/Spark code for performance, partition management, and cost efficiency.

Contribute to the design of Lakehouse architectures, ensuring the semantic layer supports both self‑service BI and advanced analytics. Implement data governance and quality checks within the Python pipeline code. Ensure alignment with clients’ data security standards (row‑level security, masking, access controls).

Responsibilities

• Collaborate closely with Business Analysts to understand metric definitions and codify them in Python.
• Collaborate with Data Scientists to expose semantic features for machine learning models.
• Mentor junior engineers for data modelling.
• Strong expertise in Python for data manipulation and modelling (pandas, PySpark).
• Proven experience building semantic/serving layers programmatically.
• Experience with Databricks and the Delta Lake ecosystem.
• Familiarity with Data‑as‑Code or Metrics‑as‑Code concepts (e.g., using dbt with Python models or custom Python semantic frameworks).
• Strong understanding of dimensional modelling (Star Schema) and how to implement it via Spark/Python.

Qualifications

• Minimum 5 years of professional experience in Data Engineering.
• Advanced Python skills with a focus on data manipulation and modelling.
• Hands‑on experience with Databricks, Azure Databricks, Apache Spark (PySpark), and Delta Lake.
• Knowledge of Data‑as‑Code or Metrics‑as‑Code practices.
• Solid understanding of dimensional modelling and Lakehouse architecture.
• Familiarity with CI/CD for data pipelines (Azure DevOps/GitHub Actions) and Git flow.
• Experience with ACID transactions, time travel, Unity Catalog, and data governance concepts.

Technical Skills

• Python (Advanced), SQL (Strong).
• Azure Databricks, Apache Spark (PySpark), Delta Lake.
• Building serving layers in PySpark, dbt (Python models), or Python‑based metric layers.

Data Architecture

• Medallion Architecture (Bronze, Silver, Gold).
• Dimensional modelling.
• CI/CD for data pipelines (Azure DevOps/GitHub Actions), Git flow.
• ACID transactions, time travel, Unity Catalog, governance.