Senior Cloud Data Architect – AWS & Databricks

Brillio; Northern (KY); Hybrid; USD 180,000 - 190,000; Full time

Job summary

Brillio seeks a senior cloud architect to design and implement scalable AWS and Databricks data architectures. You will build ETL pipelines, manage infrastructure with Terraform, and drive CI/CD for notebooks and data jobs.

The role requires 15+ years experience, strong Python/PySpark skills, and hands-on expertise across AWS services and Databricks. You will work onsite with a client in St. Louis, three days per week, and collaborate across teams to ensure secure, cost-efficient data platforms.

Qualifications

• Bachelor’s degree or higher in CS/IT or related field.
• AWS Solutions Architect – Professional certification.
• Databricks or Spark certification preferred.

Responsibilities

• Design and architect scalable, secure AWS and Databricks solutions.
• Develop and maintain ETL/ELT pipelines using PySpark in Databricks.
• Implement medallion architecture (bronze/silver/gold) and optimize Spark jobs.
• Provision and manage cloud infra with Terraform for Databricks on AWS/Azure.
• Write SQL for data transformation, validation, and analytics in Databricks.
• Build CI/CD pipelines for notebooks, jobs, and Terraform with Jenkins and GitHub Actions.
• Integrate data from diverse sources into cloud storage and processing layers.
• Optimize cloud costs with auto-scaling clusters and cost-aware patterns.
• Monitor pipeline performance and implement observability and alerts.

Skills

AWS Networking

AWS Kinesis

AWS Elastic Cache

AWS Redshift

AWS PaaS (Lambda, S3, Glue)

AWS EKS

AWS Cognito

Databricks

Python

PySpark

SQL

Terraform

CI/CD (Jenkins, GitHub Actions)

Education

Bachelor’s degree in Computer Science, IT, or closely related discipline

AWS Certified Solutions Architect – Professional

Databricks Certified Data Engineer Professional or Apache Spark certification

Tools

Terraform

Jenkins

GitHub Actions

Databricks

Job description

Brillio seeks a senior cloud architect to design and implement scalable AWS and Databricks data architectures. You will build ETL pipelines, manage infrastructure with Terraform, and drive CI/CD for notebooks and data jobs.

The role requires 15+ years experience, strong Python/PySpark skills, and hands-on expertise across AWS services and Databricks. You will work onsite with a client in St. Louis, three days per week, and collaborate across teams to ensure secure, cost-efficient data platforms.