Remote Senior Azure Databricks Platform Engineer

ICF International; Reston (VA); Remote; USD 140,000 - 190,000; Full time

Job summary

ICF International seeks a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US Government Agency. You’ll develop reliable batch and streaming pipelines, administer Databricks workspaces, implement governance, and automate delivery through infrastructure as code and CI/CD.

Remote role with a focus on security, compliance, and scalable data platforms.

Qualifications

• 5+ years of data engineering experience and 3+ years specifically supporting Azure Databricks and related Azure data services.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala; design and optimize large-scale ETL/ELT pipelines.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.

Responsibilities

• Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals for enterprise initiatives.
• Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.
• Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.
• Configure Unity Catalog governance, including RBAC, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage.
• Develop frameworks and interfaces that enable AI and analytics teams to access governed training, inference, and reporting data across CDMA, AlphaRex, and future DMS initiatives.
• Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools.
• Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services as required by the solution architecture.
• Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and use Azure Monitor, Databricks logs, Splunk, or comparable tools to support root cause analysis and operational reliability.

Skills

Azure Databricks

Python

SQL

PySpark

Delta Lake

Delta Live Tables

Terraform

GitHub Actions

Azure DevOps

CI/CD

Data Governance

Security & Compliance

Education

Bachelor's degree in CS, Engineering, Data Science or related fields

MS degree preferred

Tools

Databricks

Unity Catalog

Azure Data Factory

ADLS Gen2

Kafka

Airflow

Snowflake

Synapse

Terraform

Job description

ICF International seeks a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US Government Agency. You’ll develop reliable batch and streaming pipelines, administer Databricks workspaces, implement governance, and automate delivery through infrastructure as code and CI/CD.

Remote role with a focus on security, compliance, and scalable data platforms.