Senior Databricks Platform Engineer - Remote Cloud DataOps

ICF Olson; United States; Remote; USD 99,000 - 203,000; Full time

Job summary

ICF is seeking a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US Government Agency. This role supports data products from ingestion to production, with remote work available.

You will automate deployments, govern data assets, and collaborate with cloud architects, security analysts, and mission owners to ensure scalable, compliant analytics platforms across the enterprise.

Qualifications

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.
• Minimum 5 years of data engineering experience with 3+ years Azure Databricks & related Azure data services.
• Expertise in Spark, Delta Lake, Python, SQL, and Scala, with large-scale ETL/ELT design.

Responsibilities

• Design, configure, and operate Azure Databricks workspaces, clusters, jobs, and governance.
• Build batch and real-time ingestion/transformation pipelines with PySpark, SQL, Delta Lake.
• Implement medallion architecture, schema evolution, and data quality checks.
• Configure Unity Catalog governance, RBAC, and secure integration with storage.
• Develop interfaces enabling AI/analytics teams to access governed data assets.
• Automate deployments with Terraform, Databricks Asset Bundles, and CI/CD.
• Integrate Databricks with ADF, Kafka, Airflow, and other data services.
• Monitor health and perform root-cause analysis with Azure Monitor and logs.
• Support security, compliance, DR/BC planning, and operational readiness.

Skills

Azure Databricks

Python

SQL

Scala

Apache Spark

Delta Lake

Terraform

GitHub Actions

Azure DevOps

Data governance

Education

BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience

Tools

Terraform

Databricks Asset Bundles

GitHub Actions

Azure DevOps

Job description

ICF is seeking a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US Government Agency. This role supports data products from ingestion to production, with remote work available.

You will automate deployments, govern data assets, and collaborate with cloud architects, security analysts, and mission owners to ensure scalable, compliant analytics platforms across the enterprise.