Senior Databricks Architect

Cloud Analytics Technologies, LLC; Atlanta (GA); In loco; USD 150.000 - 210.000; Tempo pieno

Descrizione del lavoro

Cloud Analytics Technologies, LLC is seeking a highly experienced Databricks Architect to design and implement enterprise Databricks Lakehouse solutions on AWS. The role demands hands-on expertise with Delta Lake, Unity Catalog, and Spark technologies, delivering scalable data platforms and governance across the organization.

The ideal candidate will lead architecture, optimize pipelines, and collaborate with data engineering, cloud, and enterprise teams to drive high-performance data solutions

Competenze

• Must have extensive hands-on Databricks development and architecture experience.
• Strong Databricks Lakehouse, Delta Lake, and PySpark expertise.
• Experience architecting enterprise-scale Databricks solutions on AWS.
• Candidates with only general AWS security/governance experience without Databricks hands-on skills will not be considered.

Mansioni

• Design end-to-end enterprise Databricks Lakehouse architecture on AWS.
• Build and optimize scalable data pipelines using Databricks and Apache Spark.
• Develop complex ETL/ELT workflows with PySpark and Spark SQL.
• Implement and manage Delta Lake architecture and optimization.
• Design Unity Catalog for centralized data governance and access control.
• Optimize Spark jobs, cluster config, partitioning, caching, and queries.
• Implement Databricks Workflows for orchestration and automation.
• Develop scalable batch and streaming data processing solutions.
• Implement Medallion Architecture (Bronze/Silver/Gold).
• Integrate Databricks with AWS services (S3, IAM, Glue).
• Establish data quality, monitoring, and performance best practices.
• Lead Databricks platform architecture and technical design.

Conoscenze

Databricks architecture

Lakehouse architecture

Spark optimization

ETL/ELT design

Data governance

Data modeling

Data security

Python

SQL

Scala/Java

Strumenti

Databricks on AWS

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Delta Live Tables

Lakeflow Declarative Pipelines

S3

IAM

Glue

dbt

Airflow

Git/SVN

JIRA/Confluence

Lambda/Step Functions

Redshift

Snowflake

Descrizione del lavoro

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization:US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

We are seeking a highly experiencedDatabricks Architectwith deep, hands-on expertise in designing, developing, implementing, and optimizing enterprise-scale Databricks solutions on AWS.

The ideal candidate must have strong expertise in the Databricks ecosystem, including Lakehouse architecture, Delta Lake, Unity Catalog, Apache Spark, and Databricks Workflows. This role requires a hardcore Databricks professional with extensive hands-on experience building scalable, secure, and high-performance data platforms.

Key Responsibilities:

Design and implement end-to-end enterprise Databricks Lakehouse architecture on AWS.

Build and optimize scalable data pipelines using Databricks and Apache Spark.

Develop complex ETL/ELT workflows using PySpark and Spark SQL.

Implement and manage Delta Lake architecture, including Delta Tables, Delta Live Tables / Lakeflow Declarative Pipelines, and Delta Lake optimization.

Design and implement Unity Catalog for centralized data governance, access control, and metadata management.

Optimize Spark jobs, cluster configurations, partitioning, caching, and query performance.

Implement Databricks Workflows for job orchestration, scheduling, and pipeline automation.

Develop scalable batch and streaming data processing solutions.

Implement Medallion Architecture (Bronze, Silver, and Gold layers).

Integrate Databricks with AWS services such as S3, IAM, Glue, and cloud-native data services.

Establish best practices for data quality, reliability, monitoring, and performance optimization.

Lead Databricks platform architecture, technical design, and implementation.

Collaborate with data engineering, cloud, and enterprise architecture teams.

Required Technical Skills:

Strong hands-on experience withDatabricks on AWS.

Advanced expertise inApache Spark, PySpark, and Spark SQL.

Deep understanding ofDelta Lake and Lakehouse Architecture.

Strong experience implementingUnity Catalog.

Expertise in Databricks Workflows and job orchestration.

Experience with Delta Live Tables / Lakeflow Declarative Pipelines.

Strong knowledge of Medallion Architecture.

Hands-on experience building complex ETL/ELT data pipelines.

Experience with batch and real-time/streaming data processing.

Strong understanding of Spark performance tuning and optimization.

Experience integrating Databricks with AWS S3, IAM, and Glue.

Proficiency in Python and SQL.

Proficiency in Scala and SQL.

Proficiency in SQL and Python.

Experience with SQL Server.

Experience with Snowflake.

Ability to build and test data pipelines for real-time, batch, and stream processing.

Ability to integrate Databricks with cloud services including Amazon S3, Glue, and Redshift.

Experience with Spark Performance Tuning.

Experience with Functional Programming languages such as Scala, and Java.

Extensive experience with JIRA and Confluence.

Experience with version control (Git, SVN).

Experience with Data Build Tool (dbt) or similar ETL tool.

Experience with Python.

Experience with job scheduling frameworks (Airflow, DAG, etc.).

Experience with Lambda Functions.

Strong knowledge of data architecture.

Strong knowledge of data modeling and data platform.

Strong knowledge of data partitioning, hashing, and clustering.

Strong knowledge of data storage and retention policies.

Strong knowledge of data security and access control.

Strong knowledge of data governance best practices.

Strong knowledge of data profiling and data cleansing.

Strong knowledge of data quality metrics.

Strong knowledge of data lineage mapping.

Strong knowledge of data migration best practices.

Experience in data migration and integration.

Experience in data migration to cloud platforms.

Experience with cloud-native services such as Lambda and Step Functions.

Experience with data processing frameworks such as ETL.

Experience with generative AI use cases.

Experience with building large language models.

Minimum 5 years experience in specified role.

Must provide a project sample.

Mandatory Requirements:

Must have extensive hands-on Databricks development and architecture experience.

Must have strong Databricks Lakehouse, Delta Lake, and PySpark expertise.

Must have experience architecting and implementing enterprise-scale Databricks solutions on AWS.

Candidates with only general AWS Cloud Security, IAM, or governance experience without strong Databricks hands-on expertise will not be considered.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Some of our Business Ventures - Knowledge Analytics

Do restrictive necessities, your capability for advanced analytics or your business model need custom-made applications and infrastructure configurations? opt for our hosted managed services to urge comprehensive analytics solutions supported industry-leading Cloud Analytics Technologies, LLC code - delivered by the Cloud Analytics Technologies, LLC consultants UN agency understand these solutions best. Through a mix of with-it technology and dedicated service and support, Cloud Analytics Technologies, LLC hosted managed services will facilitate you: