Lead Cloud Data Engineer — Data Mesh + AI (Hybrid SF)

Akraya; San Francisco (CA); Hybrid; USD 124,000 - 138,000; Full time

Job summary

Akraya is seeking a Lead Cloud Data Engineer – Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives.

The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices, driving scalable data products with federated governance while providing technical leadership across cross-functional teams.

Qualifications

• 8+ years in Data Engineering with at least 5+ years designing enterprise AWS Data Platforms.
• Strong hands-on experience with AWS S3, Glue, EMR, Redshift, Lambda, Kinesis, Lake Formation, and Step Functions.
• Proven experience implementing Data Mesh architecture, domain-driven data products, and federated governance.
• Expertise with Databricks (Delta Lake, Unity Catalog), Starburst/Trino, Collibra, and Immuta.
• Strong knowledge of ETL/ELT development, data modeling, metadata management, data lineage, and data quality frameworks.
• Experience with Terraform, CloudFormation, Docker, ECS, GitHub Actions/GitLab CI, Jenkins, and DevSecOps best practices.
• Experience implementing AI-enabled engineering solutions using Amazon Bedrock, LLMs, RAG architectures, Copilot, Claude Code, or similar AI frameworks.
• Strong SQL, Python, and cloud-native data engineering expertise.
• Experience supporting secure, compliant cloud environments, including AWS GovCloud or regulated environments.
• Excellent leadership, mentoring, architecture design, communication, and stakeholder management skills.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using AWS services including Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, and S3.
• Architect and implement enterprise Data Mesh solutions with domain-oriented ownership, federated governance, and reusable self-service data product frameworks.
• Develop modern data platforms leveraging Databricks, Delta Lake, Unity Catalog, Starburst/Trino, Collibra, and Immuta.
• Implement enterprise data governance, metadata management, data lineage, access controls, and automated data quality monitoring frameworks.
• Build and maintain CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation), containerized deployments, and DevSecOps automation.
• Design and implement AI-powered solutions including RAG pipelines, agentic AI workflows, Amazon Bedrock, LLM integrations, and AI-assisted SDLC automation.
• Optimize cloud infrastructure for scalability, security, performance, and cost efficiency across hybrid cloud environments.
• Provide L3 production support, mentor engineering teams, and establish best practices for modern data engineering and Data Mesh adoption.
• Lead technical architecture reviews, collaborate with business stakeholders, and translate business requirements into scalable technical solutions.
• Create and maintain technical architecture documentation, implementation standards, and engineering best practices.

Skills

AWS Data Engineering

Data Mesh Architecture

Databricks

DevSecOps

AI-Augmented Data Eng

SQL

Python

Tools

Terraform

CloudFormation

Docker

ECS

GitHub Actions

GitLab CI

Jenkins

Job description

Akraya is seeking a Lead Cloud Data Engineer – Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives.

The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices, driving scalable data products with federated governance while providing technical leadership across cross-functional teams.