Principal Architect

Xcede; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Xcede is seeking a Principal Architect with 15+ years of cloud architecture experience to lead scalable AWS and Databricks data platforms across infra, data engineering, governance, and DevOps.

You will design solutions using AWS Networking, Kinesis, Redshift, EKS, Cognito; build ETL/ELT pipelines with Python/PySpark in Databricks; implement Bronze-Silver-Gold medallion architecture and drive Terraform, CI/CD with Jenkins and GitHub Actions.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field
• AWS Certified Solutions Architect - Professional preferred
• Databricks Certified Data Engineer Professional or Apache Spark certification preferred

Responsibilities

• Design scalable, secure, and high-performance AWS and Databricks solutions across cloud data platforms.
• Build and maintain ETL/ELT pipelines using Python and PySpark within Databricks.
• Implement Bronze, Silver, and Gold medallion architecture and optimize Spark jobs for performance and cost.
• Use Terraform to provision and manage Databricks workspaces, clusters, jobs, and storage.
• Develop advanced SQL for data transformation and analytics.
• Build CI/CD pipelines using Jenkins and GitHub Actions.
• Integrate data from databases, APIs, streaming platforms, and files.
• Optimize cloud costs through autoscaling and resource management.
• Monitor pipelines and implement observability and alerting.

Skills

AWS Networking

Kinesis

Redshift

Lambda

S3

Glue

EKS

Cognito

Databricks

Delta Lake

Unity Catalog

Delta Live Tables

Python

PySpark

Advanced SQL

Terraform

Jenkins

GitHub Actions

Education

Bachelor's degree in Computer Science, Information Technology, or related field

AWS Certified Solutions Architect - Professional

Databricks Certified Data Engineer Professional or Apache Spark certification

Tools

Databricks

Delta Lake

Unity Catalog

Delta Live Tables

Jenkins

GitHub Actions

Job description

We are seeking a Principal Architect with 15+ years of cloud architecture experience and strong hands-on expertise designing and implementing AWS and Databricks solutions. This person will lead the architecture of scalable, secure, and high-performance cloud data platforms while working across infrastructure, data engineering, governance, and DevOps.

Key Responsibilities

• Design scalable, secure, and high-performance AWS and Databricks solutions using AWS Networking, Kinesis, ElastiCache, Redshift, EKS, Cognito, and other PaaS services.
• Build and maintain ETL/ELT pipelines using Python and PySpark within Databricks, including notebooks, jobs, Delta Lake, and Unity Catalog.
• Implement Bronze, Silver, and Gold medallion architecture and optimize Spark jobs for performance, scalability, reliability, and cost.
• Use Terraform to provision and manage Databricks workspaces, clusters, jobs, S3/ADLS storage, networking, IAM, and related infrastructure.
• Develop advanced SQL for data transformation, validation, analytics, and data quality.
• Build CI/CD pipelines using Jenkins and GitHub Actions.
• Integrate data from databases, APIs, streaming platforms, and file-based sources.
• Optimize cloud costs through autoscaling, spot instances, scheduling, and resource management.
• Monitor pipelines, troubleshoot failures, and implement observability and alerting.

Required Skills

• AWS Networking and VPC design
• AWS Kinesis
• AWS Redshift
• AWS Lambda, S3, and Glue
• AWS EKS
• AWS Cognito
• Databricks, Delta Lake, Unity Catalog, and Delta Live Tables
• Python and PySpark
• Advanced SQL
• Terraform
• Jenkins and GitHub Actions

Preferred Skills

• Azure Data Lake, Data Factory, Synapse, and Key Vault
• Big data and dimensional modeling
• Lakehouse architecture
• Spark and Databricks performance tuning
• Data governance, lineage, and compliance

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field
• AWS Certified Solutions Architect - Professional preferred
• Databricks Certified Data Engineer Professional or Apache Spark certification preferred