Lead Data Engineer

Eightfold AI

Bengaluru

On-site

INR 1,800,000 - 3,200,000

Full time

18 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Eightfold AI is building the next-generation analytics stack to support AI-driven talent solutions. Join a skilled team migrating from Redshift/Databricks to StarRocks across multiple regions and clouds, while delivering reliable, scalable data platforms.

You will design lakehouse architectures, run analytical query layers, and mentor engineers to advance platform initiatives and reliability at scale.

Qualifications

  • 5+ years designing and operating large-scale data platforms.
  • Hands-on with Iceberg, Hudi, and Parquet.
  • Experience with StarRocks, ClickHouse, Trino or similar.
  • Proficient in batch and streaming pipelines (Spark/Flink).
  • Production multi-region cloud platforms (AWS/Azure/OCI).

Responsibilities

  • Architect and manage the lakehouse with Iceberg tables.
  • Own analytical query layer; tune and scale data warehouses.
  • Optimize Parquet layout and storage efficiency.
  • Build robust batch/streaming ETL/ELT pipelines.
  • Enable event-driven autoscaling on Kubernetes (KEDA).
  • Manage cloud infra with Terraform, Helm, GitOps (ArgoCD/Flux).
  • Tune warehouses for latency and cost; diagnose bottlenecks.
  • Define SLOs, monitor with Prometheus/Grafana, be on-call.
  • Mentor engineers and guide data-platform direction.

Skills

Iceberg/Hudi
Parquet
StarRocks
ClickHouse
Trino
Spark
Flink
Kubernetes
Terraform
GitOps

Tools

Spark
Flink
Trino

Job description

About The Team

Eightfold offers a suite of AI-powered products, including Talent Acquisition, Talent Management, Workforce Exchange, and AI Interviewer. The Analytics team builds the customer-facing solutions that show how Eightfold delivers value across these products. Our work helps customers’ leadership teams make strategic workforce decisions and helps their recruiting teams operate more effectively day to day.

  • Native reporting platform: a reporting experience built directly into the Eightfold product.
  • Agentic Dashboard Authoring: AI-assisted dashboard creation, so users can go from question to insight without building reports by hand.
  • Conversational Analytics: natural-language access to data, letting users ask questions and get answers in plain English.
  • Secure Talent Lake connectivity: secure pipelines that let customers ETL their Eightfold data into their own data warehouses.

We’re in the middle of a major revamp of our analytics stack. Today we run on Redshift and Databricks, and we’re migrating to StarRocks across 8+ regions distributed globally on AWS, Azure, and Oracle Cloud. As Lead Platform / Data Engineer, you’ll play a central role in shaping this migration and the multi-cloud platform that comes out of it.

What You’ll Do
  • Architect and manage the lakehouse. Design and maintain Apache Iceberg tables, including schema evolution, partitioning strategies, snapshot management, compaction, and metadata optimization.
  • Own the analytical query layer. Deploy, operate, and performance-tune open-source datawarehouse clusters, covering FE/BE sizing, materialized views, query profiling, caching, and resource isolation.
  • Optimize storage formats. Make deliberate choices about Parquet file layout, including row group sizing, compression codecs, encoding, and predicate pushdown, so data stays cheap to store and fast to query.
  • Build and maintain ETL/ELT pipelines. Develop robust batch and streaming pipelines with strong guarantees around data quality, idempotency, backfills, and observability.
  • Run event-driven autoscaling on Kubernetes. Design and implement KEDA-based scaling for data workloads, triggered by queue depth, Kafka lag, or custom metrics, so you balance cost against latency.
  • Manage infrastructure as code. Provision and maintain cloud and Kubernetes infrastructure using tools like Terraform, Helm, and GitOps workflows (ArgoCD/Flux).
  • Tune self-hosted open-source warehouses. Diagnose bottlenecks across CPU, memory, I/O, and network. Tune configurations, run benchmarks, and drive measurable gains in query latency and cost efficiency.
  • Champion reliability. Define SLOs, build monitoring and alerting (Prometheus, Grafana, or similar), participate in on‑call, and lead incident retrospectives.
  • Mentor and influence. Review designs and code, mentor engineers, and help set technical direction for the data platform.
What We Need
  • 5+ years of experience designing, building, and operating large-scale data platforms, lakehouse architectures, and distributed analytical systems
  • Hands‑on expertise with Apache Iceberg / Hudi and Parquet, including partitioning, schema evolution, compaction, snapshot management, and storage optimization.
  • Experience operating and performance-tuning analytical query engines such as StarRocks, ClickHouse, Trino, or similar distributed data warehouses.
  • Proficiency in building reliable batch and streaming pipelines using Apache Spark and Flink, with strong fundamentals in data quality, idempotency, fault tolerance, and observability.
  • Experience operating production‑grade, multi‑region cloud platforms across AWS, Azure, or OCI, with a focus on scalability, reliability, performance, and cost efficiency. Prior experience in mentoring engineers and leading projects while driving platform initiatives end‑to‑end.
About Eightfold

Eightfold is a global leader in an AI-native enterprise talent platform, trusted by the world’s largest & most respected Fortune 500 organizations. Our platform is built from the ground up, operating at scale across Azure and AWS, deployed in multiple regions globally, including IL4‑compliant environments for US Government, supporting users in 100+ countries and 30+ languages. Today, Eightfold is at the forefront of agentic AI, delivering intelligent agents that actively drive outcomes across hiring and talent workflows, while much of the industry is still experimenting with prototypes. Backed by over $410M in funding and valued at $2B+, we are defining the next era of agentic talent systems.

What sets Eightfold apart is not just the technology & our mission, but the team behind it. We are a deeply technical, execution‑driven organization that values ownership, collaboration, and high standards. Our engineers, product leaders, and go‑to‑market teams work closely together in person and across functions to build systems that scale in the real world. If you’re excited to work on hard problems, move with urgency, raise the bar every day, and help build agentic systems that transform how the world works, Eightfold is the place to do it.

Founders

Varun Kacholia (https://www.linkedin.com/in/varunkacholia/) & Ashutosh Garg (https://www.linkedin.com/in/ashutoshgarg893/)

Engineering Culture At Eightfold

We are a team of self‑starters who excel in their fields. We believe in giving you responsibility, not a task. We want you to have ownership and pride in your work and see your work's positive impact on your colleagues, our customers, and the world. We believe in providing transparency and support so you can do the best work of your career.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff Engineer
Staff Engineer

Eightfold • Bengaluru

On-site
INR 1,800,000 - 2,400,000
Director, Engineering
Director, Engineering

Eightfold AI • Bengaluru

Hybrid
INR 3,000,000 - 6,000,000
Hybrid work model
Office presence in Bengaluru/Noida
Director, Engineering
Director, Engineering

Eightfold • Bengaluru

Hybrid
INR 5,000,000 - 9,000,000
Senior Engineer - Talent Management
Senior Engineer - Talent Management

Eightfold.ai • Bengaluru

Hybrid
INR 900,000 - 1,400,000
Hybrid work model
Manager, Customer Engineering (Post go-live support and integrations)
Manager, Customer Engineering (Post go-live support and integrations)

Eightfold • Bengaluru

Hybrid
INR 3,500,000 - 6,500,000
Hybrid work model
Manager, Customer Engineering (Post go-live support and integrations)
Manager, Customer Engineering (Post go-live support and integrations)

Eightfold AI • Bengaluru

Hybrid
INR 2,600,000 - 4,200,000
Product Solutions Expert I
Product Solutions Expert I

Eightfold AI • Bengaluru

Hybrid
INR 600,000 - 900,000
Hybrid work model
EEO policy
Staff, Engineering Operations
Staff, Engineering Operations

Eightfold • Bengaluru

On-site
INR 3,000,000 - 6,500,000
Product Manager II - Talent Forge
Product Manager II - Talent Forge

Eightfold • Bengaluru

Hybrid
INR 1,800,000 - 3,200,000
Hybrid work model
Equity awards / discretionary bonuses
Employee benefits
Senior Engineer, Agentic AI
Senior Engineer, Agentic AI

Eightfold • Bengaluru

Hybrid
INR 1,500,000 - 2,500,000
Family medical, vision, dental coverage
Competitive base salary
Eligibility for equity awards and bonuses