Data Engineer

Alike Thoughts; India; Hybrid; INR 4,500,000 - 7,000,000; Full time

Job summary

Alike Thoughts seeks a senior data engineer to architect and deliver Databricks-based data solutions in the Insurance domain. You will leverage Delta Lake, Spark, PySpark, Spark SQL, Python and Scala to build robust ETL/ELT pipelines on AWS, model data using Star/Snowflake schemas, and enable analytics and ML workflows with MLflow.

Collaborate with architects, BI and data science teams, implement CI/CD with Databricks and Git, uphold governance and security standards, and apply Agile delivery

Qualifications

• 8–12 years of hands-on Databricks development experience.
• Strong ETL/ELT pipeline expertise.
• Experience with Delta Lake and ACID transactions.
• Insurance domain experience is a plus.

Responsibilities

• Design and implement Databricks-based data pipelines.
• Build scalable data models (Star/Snowflake).
• Collaborate with architects, BI and data science teams.
• Ensure governance, security, and compliance.

Skills

Databricks

ETL/ELT

Spark / PySpark

Python / Scala

Delta Lake

Data Modeling

AWS

CI/CD

Git

MLflow

Agile / Scrum

Insurance domain

Tools

Databricks

Spark

MLflow

Git

CI/CD tooling

AWS S3

Job description

Job Description:

• Databricks Development: 8-12 years Hands-on development skill using Databricks for large-scale data processing and analytics
• Data Engineering: Strong experience building and maintaining ETL / ELT pipelines
• Processing Frameworks: Apache Spark, PySpark, Spark SQL, Python, Scala, Power BI
• Delta Lake: Hands-on experience with Delta Lake (ACID transactions, schema evolution, performance optimization)
• Data Pipelines: Development of batch and near real-time data ingestion pipelines
• Data Modelling: Implementation of analytical and dimensional data models (Star / Snowflake)
• Cloud Storage: Experience with AWS cloud-based storage systems
• Development Best Practices: Adherence to coding standards, documentation, and data engineering best practices
• Collaboration: Working closely with architects, BI teams, and data science teams to deliver end-to-end solutions
• Exposure to ML/AI workloads on Databricks, including feature engineering
• Familiarity with MLflow or similar model tracking tools
• Experience with Databricks CI/CD and Git-based version control
• Understanding of data governance, security, and compliance concepts
• Familiarity with Agile / Scrum delivery methodologies
• Domain experience in Insurance