Databricks Data Engineer: Scalable Pipelines & Real-Time Data

Elitez India; Singapore; On-site; SGD 110,000 - 170,000; Full time

Job summary

Elitez India is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms. You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and machine learning workloads.

You will collaborate with analytics, product, and infrastructure teams to advance the data platform, governance, monitoring, and reliability. The role emphasizes building robust pipelines, scheduling, and real-time processing capabilities.

Qualifications

• 3+ years of data engineering experience with scalable pipelines.
• Experience in data modelling and distributed computing architectures.
• Hands-on with PySpark, Spark SQL and Databricks notebooks/jobs.
• Experience orchestrating pipelines with ADF, Airflow or similar tools.
• Experience with real-time and batch data processing.
• Experience on Azure; AWS experience beneficial.
• Proficiency in SQL with window functions and optimization.
• Familiarity with DevOps, Git workflows, and CI/CD.
• Familiarity with Scrum; working in Scrum teams.
• Strong problem-solving and collaboration mindset.
• Experience with streaming tech like Kafka, Flink, or Kinesis.
• Databricks Certified Data Engineer Associate; Databricks Certified Data Engineer Professional preferred.

Responsibilities

• Develop and maintain ETL pipelines for centralized data storage (Delta Lake).
• Integrate data from databases, APIs, logs, and streaming platforms.
• Transform data to clean, normalize, and aggregate.
• Apply processing techniques to complex datasets.
• Contribute to frameworks and deployment practices.
• Implement data governance aligned with standards.
• Partner with analytics and product leaders to design pipelines.
• Collaborate with infrastructure to advance cloud platforms.
• Monitor pipelines and resolve issues promptly.
• Develop monitoring tools, alerts, and error handling.
• Validate data extraction requirements and data accuracy.
• Support UAT and production implementation.

Skills

PySpark

Spark SQL

Databricks

SQL (Window functions)

DevOps & CI/CD

Scrum experience

Streaming tech

Problem solving

Collaboration

Tools

Azure Data Factory

Airflow

Databricks notebooks

Git

CI/CD pipelines

Python/Scala

Job description

Elitez India is seeking a Data Engineer to design, develop, and maintain scalable data pipelines on Databricks and cloud platforms. You will integrate diverse data sources, ensure data quality, and support analytics, reporting, and machine learning workloads.

You will collaborate with analytics, product, and infrastructure teams to advance the data platform, governance, monitoring, and reliability. The role emphasizes building robust pipelines, scheduling, and real-time processing capabilities.