AWS DevOps Lead: Data Platform & Databricks

UsefulBI; Alameda (CA); Hybrid; USD 140,000 - 180,000; Full time

Job summary

UsefulBI is seeking an experienced AWS DevOps Lead/Architect to lead cloud infrastructure, DevOps automation, and platform engineering for enterprise data platforms. The role combines 70% DevOps/Platform Engineering and 30% Data Engineering support, with emphasis on AWS, CI/CD, Kubernetes, and Databricks.

The candidate will design scalable pipelines, manage IaC, automate Python workflows, and ensure security and performance across cloud environments.

Qualifications

• 3 days per week onsite in Alameda, CA.
• Hands-on AWS cloud services experience (S3, IAM, Glue, Lambda, EC2, Redshift).
• Proficiency with Terraform for provisioning and automation.
• Strong Python programming for data engineering and automation.
• Extensive Databricks experience (Spark, Delta Lake, orchestration).
• Solid data engineering fundamentals: ETL/ELT, data modeling.
• CI/CD tooling experience (GitHub Actions, Jenkins, or equivalents).
• SQL and relational database proficiency.

Responsibilities

• Design, develop, and optimize scalable data pipelines with Databricks.
• Build and manage cloud infrastructure using IaC with Terraform on AWS.
• Develop automation scripts and data workflows in Python.
• Maintain cloud-based data solutions using AWS services (S3, IAM, Glue, Lambda, EC2, Redshift).
• Ensure high availability, scalability, and performance of data platforms.
• Collaborate with analysts and engineers for end-to-end data lifecycle management.
• Implement CI/CD pipelines for automated deployment of infra and data workflows.
• Monitor, troubleshoot, and optimize production systems; enforce security best practices.

Skills

AWS

Terraform

Python

Databricks

Kubernetes

CI/CD

SQL

Spark

EKS

Tools

Terraform

GitHub Actions

Jenkins

GitLab CI/CD

Azure DevOps

Job description

UsefulBI is seeking an experienced AWS DevOps Lead/Architect to lead cloud infrastructure, DevOps automation, and platform engineering for enterprise data platforms. The role combines 70% DevOps/Platform Engineering and 30% Data Engineering support, with emphasis on AWS, CI/CD, Kubernetes, and Databricks.

The candidate will design scalable pipelines, manage IaC, automate Python workflows, and ensure security and performance across cloud environments.