AWS DevOps Lead/Architect

UsefulBI; Alameda (CA); Hybrid; USD 140,000 - 180,000; Full time

Job summary

UsefulBI is seeking an experienced AWS DevOps Lead/Architect to lead cloud infrastructure, DevOps automation, and platform engineering for enterprise data platforms. The role combines 70% DevOps/Platform Engineering and 30% Data Engineering support, with emphasis on AWS, CI/CD, Kubernetes, and Databricks.

The candidate will design scalable pipelines, manage IaC, automate Python workflows, and ensure security and performance across cloud environments.

Qualifications

• 3 days per week onsite in Alameda, CA.
• Hands-on AWS cloud services experience (S3, IAM, Glue, Lambda, EC2, Redshift).
• Proficiency with Terraform for provisioning and automation.
• Strong Python programming for data engineering and automation.
• Extensive Databricks experience (Spark, Delta Lake, orchestration).
• Solid data engineering fundamentals: ETL/ELT, data modeling.
• CI/CD tooling experience (GitHub Actions, Jenkins, or equivalents).
• SQL and relational database proficiency.

Responsibilities

• Design, develop, and optimize scalable data pipelines with Databricks.
• Build and manage cloud infrastructure using IaC with Terraform on AWS.
• Develop automation scripts and data workflows in Python.
• Maintain cloud-based data solutions using AWS services (S3, IAM, Glue, Lambda, EC2, Redshift).
• Ensure high availability, scalability, and performance of data platforms.
• Collaborate with analysts and engineers for end-to-end data lifecycle management.
• Implement CI/CD pipelines for automated deployment of infra and data workflows.
• Monitor, troubleshoot, and optimize production systems; enforce security best practices.

Skills

AWS

Terraform

Python

Databricks

Kubernetes

CI/CD

SQL

Spark

EKS

Tools

Terraform

GitHub Actions

Jenkins

GitLab CI/CD

Azure DevOps

Job description

JOB DESCRIPTION

Job Summary:

UsefulBI is looking for an experienced AWS DevOps Lead/Architect to lead cloud infrastructure, DevOps automation, and platform engineering initiatives for enterprise-grade data platforms.
This role involves 70% DevOps / Platform Engineering and 30% Data Engineering support, with a strong focus on AWS cloud infrastructure, CI/CD automation, Kubernetes, and Databricks platform management. The ideal candidate should be comfortable working closely with application teams, data engineers, and client stakeholders.

Key Responsibilities:

• Design, develop, and optimize scalable and resilient data pipelines using Databricks.
• Build and manage cloud infrastructure using Infrastructure as Code (IaC) with Terraform on AWS
• Develop automation scripts and data processing workflows using Python
• Implement and maintain cloud-based data solutions on AWS services such as S3, IAM, Glue, Lambda, EC2, and Redshift
• Ensure high availability, scalability, and performance of data platforms and pipelines
• Collaborate with data analysts, engineers to support end-to-end data lifecycle management
• Implement CI/CD pipelines for automated deployment of infrastructure and data workflows
• Monitor, troubleshoot, and optimize production systems to ensure reliability and performance
• Enforce security best practices and compliance standards across cloud environments

Required Skills & Qualifications:

• Hybrid role requiring 3 days per week onsite in Alameda, CA
• Strong hands-on experience with AWS cloud ecosystem (S3, IAM, Glue, Lambda, EC2, Redshift, etc.)
• Proficiency in Terraform for infrastructure provisioning and automation
• Strong programming experience in Python for data engineering and automation
• Extensive experience with Databricks, including Spark, Delta Lake, and job orchestration
• Solid understanding of data engineering principles, ETL/ELT pipelines, and data modeling
• Experience with CI/CD tools such as GitHub Actions, Jenkins, or equivalent
• Strong knowledge of SQL and relational databases
• 5+ years of experience in DevOps / Cloud Engineering / Platform Engineering.
• Strong hands-on expertise with AWS cloud services and architecture.
• Mandatory experience with Databricks administration, deployment, and optimization.
• Strong experience in Terraform / CloudFormation / AWS CDK.
• Experience with Docker, Kubernetes, and EKS.
• Hands-on experience with CI/CD tools such as Jenkins, GitHub Actions, GitLab CI/CD, or Azure DevOps.
• Strong scripting skills in Python.