Lakehouse Architect / Databricks SME

CACI International; United States; Remote; USD 115,000 - 252,000; Full time

Job summary

CACI International is seeking a Lakehouse Architect / Databricks SME to support the DHS Office of Inspector General. You will design and implement a secure, scalable data lakehouse on Azure Databricks and Azure Data Lake Storage that serves as the authoritative data foundation for AI, analytics, and investigative workflows in a government cloud.

You will lead tiered storage, catalog integration, compute configurations, and governance while collaborating with data engineers, government

Qualifications

• Bachelor's degree plus 15 years of experience in data architecture/engineering or related field (equivalencies allowed).
• Ability to obtain Active DHS/EOD clearance as required.
• Deep expertise in Azure Databricks, Delta Lake, Unity Catalog, and medallion architectures.

Responsibilities

• Lead the design and implementation of a secure, scalable enterprise data lakehouse on Azure Databricks and Azure Data Lake Storage.
• Architect tiered storage zones (raw/bronze, curated/silver, analytics-ready/gold) with governance at each layer.
• Configure environments including workspace architecture, compute clusters, autoscaling, and cost optimization.
• Design data catalog integration, metadata management, and SQL-accessible tables/views for governed discovery and lineage.
• Support analytical models including SQL analytics, notebooks (Python/R/Scala), ML, and visualization integration.
• Migrate analyst workflows from local desktop processing to cloud-based notebooks and SQL analytics.
• Provide surge support for complex data architecture challenges and performance optimization.
• Develop ML/AI infrastructure including MLflow, feature stores, and integration between Databricks and AI services.
• Create training content and user enablement materials for investigators, analysts, and data scientists.
• Collaborate with government stakeholders, data engineers, and governance specialists to align with mission analytics requirements.

Skills

Analytical thinking

Stakeholder communication

Education

Bachelor's degree

Tools

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Lake Storage

PySpark

SQL

MLflow

Job description

Job Title: Lakehouse Architect / Databricks SME

Job Category: Information Technology

Time Type: Full time

Minimum Clearance Required to Start: None

Employee Type: Regular

Percentage of Travel Required: None

Type of Travel: None

The Opportunity

CACI is seeking an expert Lakehouse Architect / Databricks Subject Matter Expert to support the Department of Homeland Security (DHS) Office of the Inspector General (OIG). This role offers a unique opportunity to architect and implement a modern enterprise data platform that transforms how federal oversight professionals access, analyze, and derive insights from investigative and operational data.

As a Lakehouse Architect / Databricks SME, you will design and lead the implementation of a secure, scalable data lakehouse using Azure Databricks and Azure Data Lake that serves as the authoritative data foundation for AI, advanced analytics, and investigative workflows. You will architect tiered storage zones, design data catalog integration, establish compute configurations optimized for diverse analytical workloads, and modernize analyst workflows from local desktop processing to collaborative, cloud-based environments. From supporting relational, graph, and geospatial data models to enabling machine learning infrastructure, you will shape the technical architecture that powers insights from investigative records, hotline complaints, and federal datasets. Working within secure Azure Government environments, you will build the data platform foundation for a transformational program of national importance. Join us to make a meaningful impact by architecting the enterprise data infrastructure that enables next-generation federal oversight capabilities.

Responsibilities

• Lead the design and implementation of a secure, scalable enterprise data lakehouse platform using Azure Databricks, Azure Data Lake Storage, Delta Lake, and Unity Catalog to centralize investigative, operational, and analytical data
• Architect lakehouse storage zones supporting tiered data organization (raw/bronze, curated/silver, analytics-ready/gold) with data quality, transformation, and governance controls at each layer
• Design and implement environment configurations including workspace architecture, compute cluster strategies, autoscaling policies, job orchestration, and cost optimization approaches for diverse analytical and AI/ML workloads
• Design data catalog integration, metadata management, and SQL-accessible tables/views over stored files to enable governed data discovery, lineage tracking, and self-service analytics
• Support multiple analytical models and use cases including relational data warehousing, graph/network analysis for investigative linkages, geospatial analytics, and machine learning feature engineering
• Lead modernization of analyst and data scientist workflows from local workstation-based processing (Excel, R scripts, desktop tools) to collaborative, cloud-based notebooks, SQL analytics, and platform‑native tools
• Provide technical expertise and surge support for complex data architecture challenges, analytical database design, performance optimization, and specialized data analysis requirements
• Design and implement machine learning engineering infrastructure including MLflow integration, feature stores, model training environments, and integration patterns between Databricks and AI platform services
• Develop technical content for data platform training, best practices guidance, workflow documentation, and user enablement materials tailored to investigators, analysts, auditors, and data scientists
• Collaborate with Government stakeholders, data engineers, AI engineers, and governance specialists to align lakehouse architecture with mission analytics requirements, data governance policies, and Agile delivery processes.

Qualifications

• Bachelor's degree + 15 years of experience in data architecture, data engineering, analytics engineering, or related field; equivalencies considered (Master's + 12 years; 21 years with no degree; AA + 17 years)
• Must be able to obtain a Active DHS/ EOD Clearance as required.
• Deep expertise in Azure Databricks with demonstrated experience architecting and implementing lakehouse solutions using Delta Lake, Unity Catalog, and medallion architecture patterns
• Proven experience designing enterprise data platforms supporting diverse analytical workloads including SQL analytics, data science notebooks (Python, R, Scala), machine learning, and visualization integration
• Strong background in data modeling across multiple paradigms including dimensional modeling, relational databases, graph databases, and data lake/lakehouse architectures
• Extensive hands‑on experience with Azure Data Lake Storage, PySpark, SQL, performance optimization, compute configuration, and cost management for cloud analytics platforms

Desired –

• Experience implementing Databricks in Azure Government or other secure government cloud environments (AWS GovCloud, secure enclaves) with understanding of FedRAMP, NIST 800-53, and federal security requirements
• Hands‑on experience integrating Databricks with Azure AI services, MLflow, feature engineering frameworks, and supporting machine learning operations (MLOps) and AI/ML model pipelines
• Background in federal government, law enforcement, investigative analytics, or national security data environments with understanding of sensitive data handling, data governance, and analytical workflows for oversight missions

What You Can Expect

A culture of integrity.

At CACI, we place character and innovation at the center of everything we do. As a valued team member, you’ll be part of a high‑performing group dedicated to our customer’s missions and driven by a higher purpose – to ensure the safety of our nation.

An environment of trust.

CACI values the unique contributions that every employee brings to our company and our customers - every day. You’ll have the autonomy to take the time you need through a unique flexible time off benefit and have access to robust learning resources to make your ambitions a reality.

A focus on continuous growth.

Together, we will advance our nation's most critical missions, build on our lengthy track record of business success, and find opportunities to break new ground — in your career and in our legacy.

Pay Range

There are a host of factors that can influence final salary including, but not limited to, geographic location, Federal Government contract labor categories and contract wage rates, relevant prior work experience, specific skills and competencies, education, and certifications. Our employees value the flexibility at CACI that allows them to balance quality work and their personal lives. We offer competitive compensation, benefits and learning and development opportunities. Our broad and competitive mix of benefits options is designed to support and protect employees and their families. At CACI, you will receive comprehensive benefits such as; healthcare, wellness, financial, retirement, family support, continuing education, and time off benefits.

Since this position can be worked in more than one location, the range shown is the national average for the position.

The proposed salary range for this position is:

$114,600-$252,100

CACI is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, age, national origin, disability, status as a protected veteran, or any other protected characteristic.