Lead Data Engineer - Databricks

Searce Inc; India; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Searce Inc in India is seeking a Lakehouse Architect to lead end-to-end Databricks-based data platforms. You will design scalable lakehouse architectures, guide batch and streaming pipelines, and implement AI-ready data foundations using Unity Catalog and MLflow.

As a hands-on leader you will work with clients as a technical DRI, translate complex business questions into data services, optimize Spark jobs for performance and cost, and build reusable frameworks that accelerate future projects.

Qualifications

• 6+ years of professional experience in end-to-end data product development with hands-on Databricks.
• Deep hands-on experience with Spark, Delta Lake, Unity Catalog, and scalable data pipelines.
• Fluent in PySpark, SQL, and cloud-native lakehouse design across AWS/Azure/GCP.

Responsibilities

• Architect and lead end-to-end development of scalable, cloud-native lakehouse platforms on Databricks.
• Build robust ingestion and transformation frameworks using Spark, Delta Lake, and Delta Live Tables.
• Design AI-ready data foundations and govern with Unity Catalog and MLflow.
• Act as technical DRI for client success and deliver data services with Agile methods.
• Tune Spark jobs, monitor pipelines, and optimize costs and performance at scale.
• Create reusable Databricks frameworks and libraries to accelerate future projects.

Skills

Databricks

PySpark

SQL

Delta Lake

Unity Catalog

Delta Live Tables

Apache Spark

Cloud platforms

CI/CD for data

GitHub Actions

Git

Terraform

MLflow

Tools

Databricks Workflows

Terraform

MLflow

Job description

Solver? Absolutely. But not the usual kind. We're searching for the architects of the audacious & the pioneers of the possible. If you're the type to dismantle assumptions, re-engineer 'best practices,' and build lakehouse solutions on Databricks that make the future possible NOW, then you're speaking our language.

Your Responsibilities

What you will wake up to solve.

• Lead Lakehouse Design & Data Architecture: Architect and lead the end-to-end development of scalable, cloud-native lakehouse platforms on Databricks. You'll guide the squad on critical architectural decisions-Batch vs. Streaming, ETL vs. ELT, medallion layering (Bronze/Silver/Gold)-while remaining 100% hands-on, contributing high-quality, production-grade code in PySpark and SQL.
• Build High-Velocity Data Pipelines on Databricks: Drive the implementation of robust ingestion and transformation frameworks using Apache Spark, Delta Lake, and Delta Live Tables. You will build integration layers that unify heterogeneous sources (SaaS, RDBMS, NoSQL, streaming) into high-availability Delta Lakehouse environments, orchestrated with Databricks Workflows.
• Drive AI-Ready Data Strategy: Be the expert in designing Databricks foundations optimized for AI and Machine Learning. You will champion Unity Catalog for governance and lineage, MLflow for the ML lifecycle, and feature-ready "clean room" environments that fuel advanced analytics and generative AI models.
• Partner with Clients as a Technical DRI: Act as the Directly Responsible Individual for client success. Translate ambiguous business questions into elegant data services on the Lakehouse, manage project deliverables using Agile methodologies, and ensure that the data provided is accurate, consistent, and mission-critical.
• Troubleshoot & Optimize for Scale & Cost: Own the reliability and economics of the platform. You will proactively monitor pipelines, tune Spark jobs and cluster configurations, troubleshoot complex transformation bottlenecks, and relentlessly improve Databricks performance and cost-efficiency (Photon, cluster policies, and workload optimization).
• Innovate and Build Reusable IP: Spearhead the creation of reusable Databricks frameworks, custom operators, and transformation libraries that accelerate future projects and establish Searce's unique technical advantage in the market.

Welcome to Searce

The AI-Native tech consultancy that's rewriting the rules.

Searce is an AI-native, engineering-led, modern tech consultancy that empowers clients to futurify their business by delivering intelligent, impactful, real business outcomes. Searce solvers co-innovate with clients as their trusted transformational partners ensuring sustained competitive advantage. Searce clients realize smarter, faster, better business outcomes delivered by AI-native Searce solver squads.

Functional Skills

the solver personas.

• The Lakehouse Architect: This persona deconstructs ambiguous business goals into scalable, elegant Databricks blueprints. They don't just move data; they design the foundation—from Delta table design and medallion layering to partitioning, liquid clustering, and Unity Catalog governance—that allows data scientists and analysts to thrive, foreseeing technical bottlenecks and making pragmatic trade-offs.
• The Player-Coach: As a hands-on leader, this persona leads from the front by writing exemplary, production-grade PySpark and SQL while simultaneously mentoring and elevating the squad's Databricks skills. Their success is measured by the team's ability to deliver high-quality, maintainable code and their growth as engineers.
• The Pragmatic Innovator: This individual balances a passion for modern data tech (like Generative AI, real-time streaming, and Delta Live Tables) with a sharp focus on business outcomes. They champion new capabilities where they add real value but are disciplined enough to choose stable, cost-effective solutions to meet deadlines and deliver robust products.
• The Client-Facing Technologist: This persona acts as the crucial technical bridge between the data squad and the client. They build trust by listening actively, explaining complex data concepts (like data latency, idempotency, or cost/performance trade-offs) in simple terms, and demonstrating how engineering decisions on the Lakehouse align with the client's strategic goals.
• The Quality Craftsman: This individual possesses an unwavering commitment to data integrity and treats data engineering as a craft. They are the guardian of the platform—advocating for robust testing, data validation frameworks (Delta constraints and expectations), and clean, modular code to ensure the long-term reliability of the Lakehouse.

Experience & Relevance

• Engineering Depth: 6+ years of professional experience in end-to-end data product development, with deep hands-on expertise on Databricks. You have a portfolio that proves your ability to build complex, high-velocity pipelines for both Batch and Streaming workloads using Apache Spark and Delta Lake.
• Databricks & Cloud-Native Fluency: Deep, hands-on experience designing and deploying scalable Databricks solutions on at least one major cloud platform (AWS, Azure, or GCP). You are fluent in PySpark (Scala a plus), Delta Lake, Unity Catalog, and Databricks Workflows, and comfortable with performance and cost tuning at scale.
• AI-Native Workflow: You don't just build for AI; you build with AI. You must be proficient in using AI coding assistants (e.g., GitHub Copilot, Databricks Assistant) to accelerate your delivery, and have a track record of building the data foundations—including MLflow-based workflows—required for Generative AI.
• Architectural Portfolio: Evidence of leading 2-3 large-scale transformations—including lakehouse migrations (e.g., legacy warehouse or Hadoop to Databricks), Delta Lakehouse builds, or real-time analytics architectures.
• Engineering Rigor & Client-Facing Acumen: Hands-on experience with CI/CD for data (Databricks Asset Bundles, Git, GitHub Actions/Azure DevOps) and infrastructure-as-code (Terraform). You also have direct, consultative client-facing experience—confidently translating a CEO's business vision into a Lead Engineer's technical specification without losing anything in translation.

Join the 'real solvers'

ready to futurify?