Senior Databricks Architect - AWS Lakehouse Leader

Valiant Technologies LLC; Atlanta (GA); In loco; USD 170.000 - 210.000; Tempo pieno

Descrizione del lavoro

Valiant Technologies LLC in Atlanta seeks a highly experienced Databricks Architect to design and implement enterprise-scale Databricks solutions on AWS.

You will lead lakehouse architecture, Delta Lake, Unity Catalog, and Spark workloads, building scalable data pipelines, governance, and optimization across the data platform. This role requires hands-on expertise and leadership in delivering secure, high-performance data solutions.

Competenze

• Must have extensive hands-on Databricks development and architecture experience.
• Strong expertise in Databricks Lakehouse, Delta Lake, and PySpark.
• Experience architecting enterprise-scale Databricks solutions on AWS.
• Knowledge of Delta Live Tables / Lakeflow Declarative Pipelines.
• Ability to optimize Spark performance and manage pipelines.

Mansioni

• Design end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines with Databricks and Spark.
• Develop ETL/ELT workflows using PySpark and Spark SQL.
• Implement and manage Unity Catalog for data governance.
• Orchestrate jobs with Databricks Workflows and automate pipelines.
• Lead platform architecture and collaborate with data, cloud, and EA teams.
• Integrate with AWS services like S3, IAM, and Glue.
• Establish data quality, monitoring, and performance practices.

Conoscenze

Databricks on AWS

Spark (PySpark)

Spark SQL

Lakehouse Architecture

Delta Lake

Unity Catalog

Databricks Workflows

ETL/ELT design

Strumenti

Databricks

Apache Spark

PySpark

Delta Live Tables

Lakeflow Declarative Pipelines

Descrizione del lavoro

Valiant Technologies LLC in Atlanta seeks a highly experienced Databricks Architect to design and implement enterprise-scale Databricks solutions on AWS.

You will lead lakehouse architecture, Delta Lake, Unity Catalog, and Spark workloads, building scalable data pipelines, governance, and optimization across the data platform. This role requires hands-on expertise and leadership in delivering secure, high-performance data solutions.