Senior Big Data Engineer - Databricks

Digit88 Technologies; Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Benefits offered by this job

Comprehensive insurance coverage

Flexible work model

Accelerated learning opportunities

High ownership and accountability

Job summary

Digit88 Technologies in Bengaluru is looking for a Senior Data Engineer to design, build, and operate scalable data platforms for global clients. The ideal candidate will have over 8 years of experience in data engineering and expertise in ETL/ELT pipeline creation using PySpark and Databricks.

You will be instrumental in shaping data architecture, with responsibilities including modernizing data systems and optimizing workflows for high reliability and performance.

Qualifications

• 8+ years of experience in data engineering.
• Hands-on expertise in ETL/ELT processes and distributed data systems.
• Experience working with Azure data ecosystem.

Responsibilities

• Design and implement scalable data platforms and pipelines.
• Modernize legacy architectures to meet scalable demands.
• Optimize Spark jobs for better performance and cost efficiency.

Skills

PySpark

ETL/ELT pipelines

Data modeling

Databricks

Event streaming (Kafka)

Agile delivery

Data quality checks

Event-driven architecture

Education

BE/MS in Computer Science or related field

Tools

Databricks

Delta Lake

SQL

Kafka

Job description

Digit88 is an AI‑native product engineering partner helping startups and enterprises build, scale and operate intelligent software products.

We are a lean, high-impact team of 75+ technologists, backed by leaders with deep experience across startups and global enterprises. We build strong, outcome-driven engineering teams that solve complex, real-world problems.

From GenAI applications and data platforms to enterprise‑grade SaaS, we deliver scalable, reliable, production‑ready systems. Our expertise spans AI/ML, RAG systems, agentic workflows and large‑scale data engineering - enabling businesses to move from idea to production with speed and confidence.

Our teams operate as a true extension of our clients, with full ownership and flexible engagement models focused on measurable business outcomes – not just delivery.

With 80+ AI implementations and proven success in scaling dedicated teams and driving significant cost efficiencies, we partner for long‑term impact.

We bring experience across B2B and B2C SaaS, web and mobile platforms, e‑commerce and domains such as Conversational AI, HealthTech, IoT, ESG/Energy and Data Engineering – thriving in fast‑paced, high‑ownership environments.

Vision

To be the most trusted AI‑native product engineering partner for innovative software companies worldwide, delivering ownership, speed and measurable outcomes.

Opportunity

As a Senior Data Engineer, you will design, build and operate scalable data platforms and pipelines for global customers. You will work closely with customers, product teams and engineers to deliver reliable, production‑grade data systems.

You will play a key role in shaping data architecture, data engineering best practices and AI‑driven data platforms at Digit88, enabling customers to move from raw data to actionable insights and intelligent systems.

Key Responsibilities

• Modernize legacy, non‑scalable architectures and define a scalable target‑state platform
• Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake and Databricks components such as Delta Live Tables (DLT), Delta Sharing and Workflows (LakeFlow, LakeBase)
• Design and build scalable, reliable and production‑ready ETL/ELT pipelines using PySpark, SQL and Databricks notebooks to ingest and transform data from diverse sources
• Create and manage workflows using Databricks Workflows (Jobs) or orchestration tools to automate pipelines and dependencies
• Optimize Spark jobs for performance, scalability and cost efficiency (partitioning, caching, query tuning, cluster optimization)
• Implement data quality checks (e.g., DLT Expectations) and enforce governance via Unity Catalog (access control, PII masking, lineage)
• Design and implement event‑driven and streaming pipelines (Kafka or equivalent)
• Ensure high data reliability through monitoring, observability and alerting

Requirements

• BE/MS in Computer Science or a related field with 8+ years of experience in data engineering
• Strong experience in ETL/ELT pipelines, data modeling, and distributed data systems, with hands‑on expertise in Databricks
• Deep proficiency in PySpark, including performance optimization, job orchestration, and large‑scale data processing
• Good understanding of event streaming systems such as Kafka or equivalent technologies
• Experience working with Azure data ecosystem (ADLS, Azure services, AHDS or similar data platforms)
• Strong foundation in event‑driven architecture and scalable distributed systems
• Proven ability to design, review, and clearly articulate system architecture
• Experience leveraging AI‑assisted development tools (e.g., Claude, Antigravity, etc.) to improve productivity
• Solid experience in Agile delivery, estimation, and program execution
• Experience working with global customers (US/EU) in a client‑facing role
• Excellent written and verbal communication skills across engineering, business, and customer stakeholders
• Strong analytical thinking and structured problem‑solving ability
• High ownership, reliability, and execution focus; consistently delivers despite constraints
• Strong attention to detail while maintaining a clear big‑picture perspective

Good to Have Skills

• Experience in Healthcare, EHR/EMR data migration, Clinical Trials, or Life Sciences domains (at least one)
• Exposure to handling large‑scale EMR/EHR integrations and reducing technical complexity across multiple data sources
• Experience working with healthcare data standards such as CDA, FHIR, and HL7, including data normalization into modern data models
• Ability to build and manage scalable data pipelines for ingestion, transformation (FHIR), data quality, governance, and near real‑time processing
• Understanding of interoperability challenges across diverse healthcare systems and approaches to solve them
• Experience building patient‑centric data platforms (e.g., Patient 360, Master Patient Index)
• Familiarity with data privacy, security, and compliance standards such as HIPAA

Comprehensive insurance coverage (Life, Health, Accident, parents and in-laws are optional)

Flexible work model focused on outcomes

Accelerated learning with non‑linear growth opportunities

Flat organization with high ownership & accountability

Opportunities to work on cutting‑edge AI and SaaS products with global customers (primarily North America, Australia, EU and UAE)

Direct exposure to building and scaling real‑world systems across Conversational AI, Energy/Utilities, ESG, HealthTech, IoT and more.

High‑impact roles with the ability to influence product, architecture and business outcomes globally

Learn from a founding team of serial entrepreneurs with multiple exits – high growth, high ownership and real challenges.

Join Digit88

This is an exciting time to join Digit88 – build, scale and grow with us as part of our journey!