Senior Databricks Platform Engineer (Remote US)

ICF; United States; Remote; USD 99,000 - 203,000; Full time

Job summary

ICF is seeking a Senior Databricks Platform Engineer to design, build, secure, and operate governed Azure Databricks Lakehouse capabilities for a US government agency. This remote role focuses on enterprise analytics, AI operations, and data initiatives in a cloud-based environment.

The candidate will configure Databricks workspaces, pipelines, and Unity Catalog governance, while integrating with ADLS Gen2 and other enterprise services. Strong CI/CD and security focus are required.

Qualifications

• 5+ years of data engineering experience with Azure Databricks and related Azure data services.
• Expertise in Spark, Delta Lake, Python, SQL, and Scala for large ETL/ELT pipelines.
• Experience with CI/CD or infrastructure as code for data platforms (Git, Terraform, GitHub Actions, Azure DevOps).
• Strong collaboration with security, governance, and enterprise data teams.

Responsibilities

• Design, configure, and operate Azure Databricks workspaces, clusters, and jobs.
• Build batch and real-time ingestion pipelines using Python, PySpark, and Delta Lake.
• Implement medallion architecture, data quality checks, and lineage governance.
• Configure Unity Catalog governance and integration with ADLS Gen2.
• Develop interfaces for AI/analytics teams to access governed data.
• Automate deployments with Terraform and Databricks Asset Bundles.
• Integrate Databricks with ADF, Kafka, Airflow, and other services.
• Monitor health and perform root cause analysis using Azure Monitor and logs.
• Support security, compliance, and disaster recovery for data platforms.
• Participate in Agile planning, code reviews, and architecture documentation.

Skills

Azure Databricks

Python

SQL

Scala

Delta Lake

Spark

CI/CD

Terraform

GitHub Actions

Azure DevOps

Education

BA/BS/MS in Computer Science/Engineering/Data Science

Tools

Azure Data Factory

Azure Data Lake Storage

Azure Synapse Analytics

Databricks Asset Bundles

Unity Catalog

ADLS Gen2

Kafka

Job description

Description

The Work

ICF is seeking a Senior Databricks Platform Engineer to support an independent US Government Agency where you will design, build, secure, and operate governed Azure Databricks Lakehouse capabilities supporting a cloud-based analytics platform, enterprise AI operations, and emerging data initiatives.

Job Location: Remote

What You Will Do

• Design, configure, and operate Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals for enterprise initiatives.
• Build and optimize batch and real-time ingestion and transformation pipelines using Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.
• Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.
• Configure Unity Catalog governance, including role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage.
• Develop frameworks and interfaces that enable AI and analytics teams to access governed training, inference, and reporting data across CDMA, AlphaRex, and future DMS initiatives.
• Automate Databricks deployments and environment configuration using Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable tools.
• Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services as required by the solution architecture.
• Monitor pipeline, cluster, and workspace health; diagnose performance and data issues; and use Azure Monitor, Databricks logs, Splunk, or comparable tools to support root cause analysis and operational reliability.
• Support security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness activities for Databricks environments and data products.
• Participate in Agile planning, code reviews, demos, and retrospectives; document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories.

What You Will Bring With You

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience, with more than 5 years of data engineering experience and at least 3 years specifically supporting Azure Databricks and related Azure data services.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala, with hands-on experience designing and optimizing large-scale ETL/ELT pipelines.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.
• Experience configuring and managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization.
• Experience implementing CI/CD or infrastructure as code for data platforms using Git and tools such as Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable technologies.
• Experience with Agile delivery, data governance, data security, performance troubleshooting, and clear communication with technical and business stakeholders.
• Ability to satisfy FDIC personnel eligibility and background-investigation requirements
• U.S. Citizenship required due to federal contract requirements.
• Candidate must reside in the U.S., be authorized to work in the U.S., and all work must be performed in the U.S.

What We Would Like You To Bring With You

• Experience with Azure Databricks, ADLS Gen2, Unity Catalog, Delta Live Tables, Auto Loader, Azure Data Factory, and Azure Synapse.
• Databricks Certified Data Engineer Associate or Professional certification, Databricks Generative AI Engineer certification, or comparable demonstrated experience.
• Experience with streaming and real-time data using Kafka, Confluent, Structured Streaming, or similar technologies.
• Experience with medallion architecture, CDC, dimensional modeling, Apache Iceberg, Snowflake, Oracle, SQL Server, or other enterprise data platforms.
• Experience supporting Generative AI, NLP, RAG applications, vector stores, agents, or Databricks AI/ML workloads.
• Experience with MLflow, model deployment pipelines, AWS SageMaker, AWS Bedrock, or cross-cloud data and AI architectures.
• Experience implementing BCDR, high availability, disaster recovery, data retention, and enterprise security controls for cloud data platforms.
• Experience supporting ATO processes, NIST or federal compliance controls, vulnerability remediation, SonarQube, Checkmarx, or Zero Trust implementation.
• Experience with Docker, Kubernetes, REST APIs, FastAPI, Node.js, or cloud-native application integration.
• Experience supporting federal data modernization programs and communicating technical decisions to agency stakeholders.

Professional Skills

• Explain data platform architecture, trade-offs, risks, and operational impacts to technical and nontechnical stakeholders.
• Work effectively across data engineering, AI, cloud, security, application, and operations teams.
• Use structured problem solving to diagnose pipeline, performance, access, and deployment issues.
• Organize work across concurrent Agile priorities and maintain clear technical documentation and status communication.

Bot and Third-Party Applications

Please note that this application must be submitted directly by the applicant for consideration. Failure to do so may result in the application being excluded for consideration . Applicants needing an accommodation for disability or religious purposes in connection with the application process should contact [e-mail] for assistance.

Working at ICF

ICF is a global advisory and technology services provider, but we’re not your typical consultants. We combine unmatched expertise with cutting-edge technology to help clients solve their most complex challenges, navigate change, and shape the future.

We can only solve the world's toughest challenges by building a workplace that allows everyone to thrive. We are an equal opportunity employer . Together, our employees are empowered to share their expertise and collaborate with others to achieve personal and professional goals. For more information, please read our EEO policy.

We will consider for employment qualified applicants with arrest and conviction records.

Reasonable Accommodations are available, including, but not limited to, for disabled veterans, individuals with disabilities, and individuals with sincerely held religious beliefs, in all phases of the application and employment process. To request an accommodation, please email [e-mail] and we will be happy to assist . All information you provide will be kept confidential and will be used only to the extent required to provide needed reasonable accommodations.

Read more about workplacediscriminationrigh t s or our benefit offerings which are included in the Transparency in (Benefits) Coverage Act.

Candidate AI Usage Policy

At ICF, we are committed to ensuring a fair interview process for all candidates based on their own skills and knowledge. As part of this commitment, the use of artificial intelligence (AI) tools to generate or assist with responses during interviews (whether in-person or virtual) is not permitted . This policy is in place to maintain the integrity and authenticity of the interview process.

However, we understand that some candidates may require accommodation that involves the use of AI. If such an accommodation is needed, candidates are instructed to contact us in advance at [e-mail] . We are dedicated to providing the necessary support to ensure that all candidates have an equal opportunity to succeed.

Pay Range - There are multiple factors that are considered in determining final pay for a position, including, but not limited to, relevant work experience, skills, certifications and competencies that align to the specified role, geographic location, education and certifications as well as contract provisions regarding labor categories that are specific to the position.

The pay range for this position based on full-time employment is :
$98,614.00 - $202,850.00

Nationwide Remote Office (US99)