Remote Senior Databricks Platform Engineer

ICF; United States; Remoto; USD 99.000 - 203.000; Tempo pieno

Descrizione del lavoro

ICF is seeking a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US government agency. This remote role focuses on enterprise analytics, AI operations, and data initiatives in a cloud-based environment.

The candidate will configure Databricks workspaces, pipelines, and Unity Catalog governance, while integrating with ADLS Gen2 and other enterprise services. Strong CI/CD and security focus are required.

Competenze

• 5+ years of data engineering experience with Azure Databricks and related Azure data services.
• Expertise in Spark, Delta Lake, Python, SQL, and Scala for large ETL/ELT pipelines.
• Experience with CI/CD or infrastructure as code for data platforms (Git, Terraform, GitHub Actions, Azure DevOps).
• Strong collaboration with security, governance, and enterprise data teams.

Mansioni

• Design, configure, and operate Azure Databricks workspaces, clusters, and jobs.
• Build batch and real-time ingestion pipelines using Python, PySpark, and Delta Lake.
• Implement medallion architecture, data quality checks, and lineage governance.
• Configure Unity Catalog governance and integration with ADLS Gen2.
• Develop interfaces for AI/analytics teams to access governed data.
• Automate deployments with Terraform and Databricks Asset Bundles.
• Integrate Databricks with ADF, Kafka, Airflow, and other services.
• Monitor health and perform root cause analysis using Azure Monitor and logs.
• Support security, compliance, and disaster recovery for data platforms.
• Participate in Agile planning, code reviews, and architecture documentation.

Conoscenze

Azure Databricks

Python

SQL

Scala

Delta Lake

Spark

CI/CD

Terraform

GitHub Actions

Azure DevOps

Formazione

BA/BS/MS in Computer Science/Engineering/Data Science

Strumenti

Azure Data Factory

Azure Data Lake Storage

Azure Synapse Analytics

Databricks Asset Bundles

Unity Catalog

ADLS Gen2

Kafka

Descrizione del lavoro

ICF is seeking a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US government agency. This remote role focuses on enterprise analytics, AI operations, and data initiatives in a cloud-based environment.

The candidate will configure Databricks workspaces, pipelines, and Unity Catalog governance, while integrating with ADLS Gen2 and other enterprise services. Strong CI/CD and security focus are required.