Associate Director Data Engineering

Cloudkeeper; Dadri, New Delhi, Gurugram District; On-site; INR 4,500,000 - 7,500,000; Full time

Job summary

CloudKeeper in Noida, India, is seeking an Associate Director, Data Engineering to own the data-platform layer for multi-cloud billing data. You will define target architecture, consolidate ingestion from AWS, Azure, and Google Cloud, and build a high-performing engineering team.

You’ll lead delivery against a roadmap with data contracts and published SLAs, ensure data freshness and accuracy, and represent data engineering in architecture, security, and customer discussions.

Qualifications

• 10+ years in data engineering
• 3+ years leading data teams
• Hands-on with cloud billing data

Responsibilities

• Lead data engineering pod
• Own data-platform roadmap and SLAs
• Report data freshness and accuracy metrics
• Represent data engineering in security and customer conversations

Skills

Data engineering

Leadership

Python

SQL

Spark

Tools

Snowflake

BigQuery

Redshift

Airflow

Kubernetes

Job description

Associate Director, Data Engineering

Location: Noida, India
Work Mode: 5 days WFO

About CloudKeeper

CloudKeeper is a cloud cost optimization partner that combines the power of group buying &
commitments management, expert cloud consulting & support, and an enhanced visibility &
analytics platform to reduce cloud cost & help businesses maximize the value from AWS,
Microsoft Azure, & Google Cloud.
A certified AWS Premier Partner, Azure Technology Consulting Partner, Google Cloud Partner,
and FinOps Foundation Premier Member, CloudKeeper has helped 350+ global companies
save an average of 20% on their cloud bills, modernize their cloud set-up and maximize value.
CloudKeeper Platform Suite is a leading enterprise FinOps platform that helps organizations
optimize and govern their multi-cloud spend across AWS, Azure, and Google Cloud. Our
platform processes billions of dollars in cloud spend, delivering actionable cost optimization
recommendations, automated governance, and financial accountability to enterprises
worldwide.
To know more - https://www.cloudkeeper.com/platform-suite

The Opportunity

CloudKeeper's products are federated, but they stand on one shared foundation: cost and
usage data. Every CK-Lens dashboard, every CK-Commit purchase decision, every CK-Tuner
recommendation, and every FinOps for AI insight is only as good as the billing data underneath
it. Today that data flows through pipelines that grew up inside individual products, spanning the
Stormus backend, Jenkins-driven data jobs, and Airflow DAGs.
We are hiring an Associate Director, Data Engineering to own this layer as a product in its own
right. You will define the target architecture for a multi-cloud billing data platform, consolidate
ingestion across AWS, Azure, and Google Cloud, set the warehouse and lakehouse strategy,
and build the team that runs it. The bar is unforgiving: customers make financial decisions on
these numbers, so every figure has to reconcile to the invoice.

What You'll Own

Multi-Cloud Billing Data Platform

• A single ingestion layer for AWS Cost and Usage Reports and Data Exports, Azure Cost
  Management exports and Billing APIs, and GCP BigQuery billing exports, on daily and
  intra-day schedules
• Normalization of provider-specific line items into a common cost model, aligned to the
  FinOps Foundation's FOCUS specification
• Correct handling of restatements, late-arriving usage, credits, discounts, marketplace
  charges, and consolidated billing across payer accounts, so reported cost always
  matches what the customer is invoiced

Warehouse and Lakehouse Architecture

• The analytics engine behind 100+ CK-Lens dashboards, anomaly detection, custom
  reports, and LensGPT natural-language queries, which runs on Snowflake today
• The target architecture for the next three years: where a warehouse ends and a
  lakehouse begins, which workloads belong in each, and how the platform scales with
  customer count without scaling spend at the same rate
• Warehouse and lake cost governance. CloudKeeper sells data-platform cost
  optimization, and the platform should be the proof

Data Foundation for New Products

• Ingestion for FinOps for AI: GPU instance usage, managed AI services such as Bedrock,
  Azure OpenAI, and Vertex AI, and LLM API consumption from providers such as OpenAI
  and Anthropic
• Kubernetes cost telemetry from Prometheus and OpenCost feeding CK-Tuner
  Kubernetes
• Cost allocation and virtual-tagging datasets that power unit economics reporting
• A governed semantic layer and data contracts so that LensGPT and the LensGPT MCP
  server can answer questions about cost data reliably

Responsibilities

Data Engineering Leadership

• Build and lead the data engineering pod: hire and mentor Data Engineers and Senior
  Engineers in data engineering, set coding, testing, and review standards, and grow the
  team as the product lines scale

• Own delivery against a data-platform roadmap agreed with Product Management, with
  explicit data contracts for each product and published service levels for data freshness
  and accuracy
• Run the platform like a product: define and report its metrics, including data freshness,
  reconciliation accuracy against source billing, pipeline reliability, and warehouse cost per
  customer, and review them monthly
• Represent data engineering in architecture, security, and customer conversations,
  including due-diligence and security questionnaires from enterprise customers

Technical Strategy

• Define the target data architecture across batch and streaming, warehouse and lake,
  and three cloud providers, and sequence the migration from today's product-embedded
  pipelines without disrupting delivery
• Design the reconciliation and data-quality framework: every pipeline proves its output
  against the source invoice, and every restatement is traceable
• Establish data governance that satisfies CloudKeeper's SOC 2 Type 2 controls and UK
  GDPR obligations: access control, retention, lineage, and customer data isolation
• Build the integration points with CK-Lens for visibility, CK-Commit and CK-Tuner for
  optimization, and the FinOps for AI and Kubernetes products for new data domains
• Lead the team's adoption of AI-assisted development. CloudKeeper builds AI-first and
  expects measurable output gains from AI in the software development lifecycle

Technical Landscape You'll Navigate

Cloud Billing and Usage Sources

• AWS: Cost and Usage Reports, Data Exports, Organizations consolidated billing, S3,
  CloudWatch
• Azure: Cost Management exports, Billing and Consumption APIs, Azure Monitor
• GCP: BigQuery billing export, Cloud Billing APIs
• Standards: FinOps Foundation FOCUS specification

Warehouse and Lakehouse

• Warehouse: Snowflake (current analytics platform); BigQuery, Redshift, or Synapse
  experience relevant
• Lake and table formats: S3, ADLS, or GCS object storage with Apache Iceberg, Delta
  Lake, or Hudi; Parquet
• Modeling and transformation: dimensional and semantic-layer modeling; dbt or
  equivalent transformation frameworks

Processing and Orchestration

• Compute: Apache Spark (SQL, DataFrames, Structured Streaming); Hadoop
  ecosystem familiarity
• Orchestration: Apache Airflow; Jenkins-driven jobs in the current estate
• Messaging: Apache Kafka, RabbitMQ, AWS SQS
• Languages: Python plus Java or Scala

Operational Stack

• Application data: MySQL, MongoDB, behind Java and Spring Boot services
• Observability: Prometheus, Grafana, OpenTelemetry, OpenCost
• Platform: Kubernetes, multi-tenant SaaS on AWS with Azure and GCP footprints

Role Requirements

Experience

• 10+ years in data engineering with 3+ years leading data engineering teams of 410
  people building and shipping SaaS or data products. Leadership experience in services,
  consulting, or systems integration (SI) teams does not apply
• Hands-on production experience with the data ecosystems of at least two of AWS,
  Azure, and Google Cloud
• Has built and operated both a cloud data warehouse (Snowflake, BigQuery, Redshift, or
  Synapse) and a data lake or lakehouse (Iceberg, Delta Lake, or Hudi on object storage)
  at multi-terabyte scale, and can explain when to use which
• Track record of pipelines whose output reconciles to a financial source of truth, such as
  billing, payments, or ledgers, strongly preferred
• Experience with cloud billing data (Cost and Usage Reports, Azure cost exports,
  BigQuery billing export) or FinOps practice is a significant advantage

Technical Depth

• Expert-level Spark and SQL; strong Python, with Java or Scala
• Warehouse performance and cost engineering: clustering, partitioning, materialization
  strategy, query cost governance
• Streaming with Kafka and batch orchestration with Airflow in production
• Data modeling for analytics: dimensional models, semantic layers, and data contracts
  consumed by multiple products
• Data quality, lineage, metadata, and governance tooling, and the judgment to apply them
  without slowing delivery
• Working understanding of SOC 2 and GDPR obligations as they apply to a data platform

Leadership

• Ability to operate in a 'founding team' mode: small team, high ownership, rapid shipping
• Experience hiring and building data engineering teams from scratch
• Comfortable wearing multiple hats: architecture, coding, hiring, product input
• Strong communicator who can influence product, platform, security, and customer-facing
  teams, and can explain a reconciliation gap to a CFO as clearly as to an engineer
• Sets the expectation for AI-assisted development by using it personally

Why CloudKeeper

• Market Opportunity: Cloud spend optimization is a $10B+ market growing rapidly as
  enterprises accelerate cloud adoption
• Data at the Core: The platform processes billions of dollars in cloud spend. Data
  engineering is the foundation every product stands on, not a support function
• Platform Ownership: Define and build the shared data platform for a federated product
  portfolio within a revenue-generating company: startup speed with enterprise stability
• New Domains: FinOps for AI, Kubernetes cost, and data-platform cost optimization are
  new data problems being solved for the first time, and this role owns their data
  foundation
• Impact: The north-star metric for the whole product line is savings delivered per
  customer per month. Every improvement in data accuracy, freshness, and coverage
  moves it directly
• Growth: Build the team and the platform from their current state to scale, with the
  metrics and ownership that come with running it as a product

CloudKeeper is an equal opportunity employer. We celebrate diversity and are committed
to creating an inclusive environment for all employees.