Lead Cloud Data Engineer : 26-02136

Akraya; San Francisco (CA); Hybrid; USD 124,000 - 138,000; Full time

Job summary

Akraya is seeking a Lead Cloud Data Engineer – Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives.

The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices, driving scalable data products with federated governance while providing technical leadership across cross-functional teams.

Qualifications

• 8+ years in Data Engineering with at least 5+ years designing enterprise AWS Data Platforms.
• Strong hands-on experience with AWS S3, Glue, EMR, Redshift, Lambda, Kinesis, Lake Formation, and Step Functions.
• Proven experience implementing Data Mesh architecture, domain-driven data products, and federated governance.
• Expertise with Databricks (Delta Lake, Unity Catalog), Starburst/Trino, Collibra, and Immuta.
• Strong knowledge of ETL/ELT development, data modeling, metadata management, data lineage, and data quality frameworks.
• Experience with Terraform, CloudFormation, Docker, ECS, GitHub Actions/GitLab CI, Jenkins, and DevSecOps best practices.
• Experience implementing AI-enabled engineering solutions using Amazon Bedrock, LLMs, RAG architectures, Copilot, Claude Code, or similar AI frameworks.
• Strong SQL, Python, and cloud-native data engineering expertise.
• Experience supporting secure, compliant cloud environments, including AWS GovCloud or regulated environments.
• Excellent leadership, mentoring, architecture design, communication, and stakeholder management skills.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using AWS services including Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, and S3.
• Architect and implement enterprise Data Mesh solutions with domain-oriented ownership, federated governance, and reusable self-service data product frameworks.
• Develop modern data platforms leveraging Databricks, Delta Lake, Unity Catalog, Starburst/Trino, Collibra, and Immuta.
• Implement enterprise data governance, metadata management, data lineage, access controls, and automated data quality monitoring frameworks.
• Build and maintain CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation), containerized deployments, and DevSecOps automation.
• Design and implement AI-powered solutions including RAG pipelines, agentic AI workflows, Amazon Bedrock, LLM integrations, and AI-assisted SDLC automation.
• Optimize cloud infrastructure for scalability, security, performance, and cost efficiency across hybrid cloud environments.
• Provide L3 production support, mentor engineering teams, and establish best practices for modern data engineering and Data Mesh adoption.
• Lead technical architecture reviews, collaborate with business stakeholders, and translate business requirements into scalable technical solutions.
• Create and maintain technical architecture documentation, implementation standards, and engineering best practices.

Skills

AWS Data Engineering

Data Mesh Architecture

Databricks

DevSecOps

AI-Augmented Data Eng

SQL

Python

Tools

Terraform

CloudFormation

Docker

ECS

GitHub Actions

GitLab CI

Jenkins

Job description

Primary Skills: AWS Data Engineering (Expert), Data Mesh Architecture (Expert), Databricks & Modern Data Platforms (Expert), DevSecOps & Infrastructure as Code (Advanced), AI-Augmented Data Engineering (Advanced) Contract Type: W2 Only Duration: 18+ Months Location: San Francisco, CA (Hybrid) Pay Range: $90 - $100 on W2

Job Summary: We are seeking a Lead Cloud Data Engineer – Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives. The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices. This role will drive the implementation of scalable data products, federated governance, and AI-augmented solutions while providing technical leadership across cross-functional engineering teams.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using AWS services including Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, and S3.
• Architect and implement enterprise Data Mesh solutions with domain-oriented ownership, federated governance, and reusable self-service data product frameworks.
• Develop modern data platforms leveraging Databricks, Delta Lake, Unity Catalog, Starburst/Trino, Collibra, and Immuta.
• Implement enterprise data governance, metadata management, data lineage, access controls, and automated data quality monitoring frameworks.
• Build and maintain CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation), containerized deployments, and DevSecOps automation.
• Design and implement AI-powered solutions including RAG pipelines, agentic AI workflows, Amazon Bedrock, LLM integrations, and AI-assisted SDLC automation.
• Optimize cloud infrastructure for scalability, security, performance, and cost efficiency across hybrid cloud environments.
• Provide L3 production support, mentor engineering teams, and establish best practices for modern data engineering and Data Mesh adoption.
• Lead technical architecture reviews, collaborate with business stakeholders, and translate business requirements into scalable technical solutions.
• Create and maintain technical architecture documentation, implementation standards, and engineering best practices.

Must-have Skills

• 8+ years of experience in Data Engineering with at least 5+ years designing enterprise AWS Data Platforms.
• Strong hands-on experience with AWS S3, Glue, EMR, Redshift, Lambda, Kinesis, Lake Formation, and Step Functions.
• Proven experience implementing Data Mesh architecture, domain-driven data products, and federated governance.
• Expertise with Databricks (Delta Lake, Unity Catalog), Starburst/Trino, Collibra, and Immuta.
• Strong knowledge of ETL/ELT development, data modeling, metadata management, data lineage, and data quality frameworks.
• Experience with Terraform, CloudFormation, Docker, ECS, GitHub Actions/GitLab CI, Jenkins, and DevSecOps best practices.
• Experience implementing AI-enabled engineering solutions using Amazon Bedrock, LLMs, RAG architectures, Copilot, Claude Code, or similar AI frameworks.
• Strong SQL, Python, and cloud-native data engineering expertise.
• Experience supporting secure, compliant cloud environments, including AWS GovCloud or regulated environments.
• Excellent leadership, mentoring, architecture design, communication, and stakeholder management skills.

Nice-to-have Skills

• Experience supporting enterprise data warehouse modernization initiatives.
• Knowledge of hybrid cloud and on-premises data platform integration.
• Experience implementing self-service analytics and reusable data product frameworks.
• Familiarity with cost optimization strategies for large-scale cloud data platforms.
• Experience working within highly regulated industries such as Financial Services or Government.

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related technical field.
• AWS Certifications (Solutions Architect, Data Analytics, or Developer) preferred.
• Experience leading enterprise-scale cloud modernization and AI transformation initiatives.
• Strong background in enterprise architecture, technical leadership, and cross-functional collaboration.