Databricks Lakehouse Architect: AI & Data Governance

CCS IT LIMITED; Washington (District of Columbia); On-site; USD 160,000 - 210,000; Full time

Job summary

CCS IT LIMITED in Washington, DC seeks a senior data engineering leader to architect and optimize lakehouse data pipelines using Databricks, Spark, and Lakeflow. You will design governance, security, and ABAC/RBAC controls while ensuring reliable ingestion from diverse sources and robust observability throughout production systems.

The role requires deep platform expertise and the ability to collaborate across engineering, security, architecture, and business units to deliver enterprise data

Qualifications

• Deep understanding of Databricks Lakeflow connect capabilities.
• Extensive experience addressing ingestion challenges from unsupported sources by Lakeflow connect. CDC handling and reconciliation techniques.
• Expertise on data quality libraries and configuration based framework for plugins in DQ into ingestion pipelines.
• Deep understanding of AWS analytics and ingestion related services.
• Extensive experience designing enterprise Databricks lakehouse architectures.
• Deep knowledge of Databricks, Spark, Delta Lake, Lakeflow, and Unity Catalog.
• Strong experience with medallion architecture and Bronze/Silver/Gold data patterns.
• Experience designing enterprise data governance, lineage, security, and access-control frameworks.
• Strong understanding of ABAC, RBAC, data classification, tagging, and least-privilege security.
• Experience designing production-grade data pipelines and reusable ingestion frameworks.
• Strong understanding of observability, data quality, CI/CD, deployment automation, and production support.
• Experience designing semantic layers and governed enterprise data products.
• Experience architecting AI/GenAI solutions that operate against governed enterprise data.
• Strong communication skills with the ability to work across engineering, security, architecture, and business teams.

Skills

Databricks Lakeflow connect

Ingestion from unsupported sources

Data quality frameworks

AWS analytics services

Databricks/Spark/Delta Lake/Lakeflow/​

Medallion architecture (Bronze/Silver/

Data governance/ lineage/ security

ABAC/RBAC and data classification

Production-grade data pipelines

Observability/ CI/CD/ deployment

Semantic layers and governed data

GenAI governance

Cross-team communication

Tools

Databricks

Spark

Delta Lake

Lakeflow

Unity Catalog

AWS

Job description

CCS IT LIMITED in Washington, DC seeks a senior data engineering leader to architect and optimize lakehouse data pipelines using Databricks, Spark, and Lakeflow. You will design governance, security, and ABAC/RBAC controls while ensuring reliable ingestion from diverse sources and robust observability throughout production systems.

The role requires deep platform expertise and the ability to collaborate across engineering, security, architecture, and business units to deliver enterprise data