Principal Architect – R01570796

Brillio; Northern (KY); Hybrid; USD 180,000 - 190,000; Full time

Job summary

Brillio seeks a senior cloud architect to design and implement scalable AWS and Databricks data architectures. You will build ETL pipelines, manage infrastructure with Terraform, and drive CI/CD for notebooks and data jobs.

The role requires 15+ years experience, strong Python/PySpark skills, and hands-on expertise across AWS services and Databricks. You will work onsite with a client in St. Louis, three days per week, and collaborate across teams to ensure secure, cost-efficient data platforms.

Qualifications

• Bachelor’s degree or higher in CS/IT or related field.
• AWS Solutions Architect – Professional certification.
• Databricks or Spark certification preferred.

Responsibilities

• Design and architect scalable, secure AWS and Databricks solutions.
• Develop and maintain ETL/ELT pipelines using PySpark in Databricks.
• Implement medallion architecture (bronze/silver/gold) and optimize Spark jobs.
• Provision and manage cloud infra with Terraform for Databricks on AWS/Azure.
• Write SQL for data transformation, validation, and analytics in Databricks.
• Build CI/CD pipelines for notebooks, jobs, and Terraform with Jenkins and GitHub Actions.
• Integrate data from diverse sources into cloud storage and processing layers.
• Optimize cloud costs with auto-scaling clusters and cost-aware patterns.
• Monitor pipeline performance and implement observability and alerts.

Skills

AWS Networking

AWS Kinesis

AWS Elastic Cache

AWS Redshift

AWS PaaS (Lambda, S3, Glue)

AWS EKS

AWS Cognito

Databricks

Python

PySpark

SQL

Terraform

CI/CD (Jenkins, GitHub Actions)

Education

Bachelor’s degree in Computer Science, IT, or closely related discipline

AWS Certified Solutions Architect – Professional

Databricks Certified Data Engineer Professional or Apache Spark certification

Tools

Terraform

Jenkins

GitHub Actions

Databricks

Job description

Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption.Brillio, renowned for its world-class professionals, referred to as “Brillians”, distinguishes itself through their capacity to seamlessly integrate cutting‑edge digital and design thinking skills with an unwavering dedication to client satisfaction.

Brillio takes pride in its status as an employer of choice, consistently attracting the most exceptional and talented individuals due to its unwavering emphasis on contemporary, groundbreaking technologies, and exclusive digital projects. Brillio’s relentless commitment to providing an exceptional experience to its Brillians and nurturing their full potential consistently garners them the Great Place to Work® certification year after year.

Job Requirements

Experience Range: With at least 15 years of experience in cloud architecture, including significant hands‑on expertise designing and implementing AWS and Databricks‑based solutions

Key Responsibilities:

• 1. Design and architect scalable, secure, and high‑performance AWS and Databricks solutions, leveraging AWS Networking, Kinesis, Elastic Cache, Redshift, EKS, Cognito, and PaaS services
• 2. Develop and maintain robust ETL/ELT pipelines using PySpark and Python within Databricks, including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance
• 3. Implement medallion architecture (bronze, silver, gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability, addressing partitioning, skew, caching, and adaptive query execution
• 4. Provision and manage cloud infrastructure using Terraform for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles and permissions, and related resources on Azure and AWS
• 5. Write efficient SQL queries for data transformation, validation, and analytics within Databricks, ensuring data quality, lineage, security, and compliance
• 6. Implement and maintain CI/CD pipelines using Jenkins and GitHub Actions for automated testing and deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations
• 7. Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers
• 8. Optimize cloud costs through auto‑scaling clusters, spot instances, job scheduling, and efficient resource usage
• 9. Monitor pipeline performance, troubleshoot failures, and implement alerting and observability using Databricks tools, cloud monitoring services, or third‑party solutions

Required Skills:

• 1. Advanced proficiency in AWS Networking and VPC design
• 2. Expertise in AWS Kinesis for real‑time data streaming and analytics
• 3. Deep experience with AWS Elastic Cache for distributed caching
• 4. Extensive experience with AWS Redshift for data warehousing
• 5. Strong knowledge of AWS PaaS services including Lambda, S3, and Glue
• 6. Hands‑on experience with AWS EKS for container orchestration
• 7. Proficiency in AWS Cognito for identity and access management
• 8. Expertise in Databricks, including Delta Lake, Unity Catalog, and Delta Live Tables
• 9. Strong Python and PySpark skills for distributed data processing
• 10. Advanced SQL skills for complex querying and optimization
• 11. Proficiency in Terraform for infrastructure automation and management
• 12. Experience with CI/CD tools such as Jenkins and GitHub Actions

Preferred Skills:

• 1. Knowledge of Azure Data Lake, Data Factory, Synapse, and Key Vault
• 2. Understanding of big data modeling (star/snowflake, dimensional) and lakehouse architecture
• 3. Experience with performance tuning in Spark and Databricks environments
• 4. Familiarity with data governance, lineage, and compliance in cloud environments

Desired Qualifications:

• 1. Bachelor’s degree in Computer Science, Information Technology, or a closely related discipline
• 2. AWS Certified Solutions Architect – Professional
• 3. Databricks Certified Data Engineer Professional or Apache Spark certification

Additional Information:

This role requires presence in St. Louis and attendance at the client office three days a week.

Equal Employment Opportunity Declaration

Brillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.

#LI-AY1

Salary: 180,000-190,000 USD per year salary