Senior Databricks Data Engineer

N2IA Technologies

Washington (District of Columbia)

Hybrid

USD 150,000 - 190,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

N2IA Technologies is seeking a Senior Databricks Data Engineer to design, build, and maintain scalable data pipelines on the Databricks Lakehouse Platform for federal clients. You will collaborate with cloud engineers, data architects, and AI developers to ensure data is clean, secure, and ready for analytics and AI/ML workflows.

This role is remote supporting a Quantico, VA customer with onsite work required at the client site full-time.

Qualifications

  • 3+ years building production pipelines on Databricks.
  • 10+ years of technical professional experience.
  • Strong proficiency with Apache Spark (PySpark) and Delta Lake.
  • Experience with Unity Catalog for data governance and lineage.
  • Secret clearance.

Responsibilities

  • Design, build, and maintain scalable data pipelines and ETL/ELT workflows on Databricks using Delta Lake and the medallion architecture.
  • Develop and orchestrate workflows using Databricks Workflows/Jobs.
  • Implement data quality checks, validation, and metadata management with Unity Catalog.
  • Collaborate with cloud engineers, data architects, and AI developers to ensure datasets meet requirements.
  • Troubleshoot pipeline issues and optimize Spark jobs and clusters for performance and cost.

Skills

Databricks
Spark
Azure/AWS
Delta Lake
Unity Catalog
SQL
Secret clearance

Tools

Databricks Workflows/Jobs
Delta Live Tables
Auto Loader
Structured Streaming
Photon

Job description

Senior Databricks Data Engineer

Our Company

N2IA Technologies is a consulting company specializing in acquisition/contracting support, cost/FinOps, and technology optimization for federal clients. We deliver tailored strategies, robust software solutions, and streamlined operations to help organizations achieve their goals. At N2IA, we are committed to developing innovative financial and compliance-based solutions that address evolving business and regulatory needs. If you are passionate about accounting, financial accuracy, and supporting government programs, N2IA is the place for you.

Overview

N2IA is seeking a Senior Databricks Data Engineer to build, refine, and maintain the core data pipelines on the Databricks Lakehouse Platform that enable enterprise AI, analytics, and mission systems. This role is essential for making data accessible, reliable, and usable across the agency. You will work with cloud engineers, data architects, and AI developers to ensure data is clean, connected, secure, and optimized for advanced analytics and AI/ML workflows. This role is remote supporting a Quantico, VA customer and onsite work is required at the client site full-time.

Key Responsibilities
  • Design, build, and maintain scalable data pipelines and ETL/ELT workflows on Databricks using Delta Lake, Delta Live Tables (DLT), and the medallion (bronze/silver/gold) architecture across cloud and on-prem systems.
  • Develop and orchestrate workflows using Databricks Workflows/Jobs.
  • Implement data quality checks, validation, lineage tracking, and metadata management using Unity Catalog and Lakehouse Monitoring / DLT expectations.
  • Collaborate with other teams to ensure datasets meet requirements and are discoverable through Unity Catalog.
  • Automate ingestion and transformation processes using Databricks-native patterns such as Auto Loader, Structured Streaming, and the COPY INTO command.
  • Support API strategy, data modeling, and domain-driven data structures within the Lakehouse.
  • Troubleshoot pipeline issues and optimize Spark jobs and clusters for performance and cost (e.g., Photon, cluster sizing, Z-ordering, liquid clustering, and workload tuning).
Required Qualifications
  • 3+ years of hands-on data engineering experience building production pipelines on Databricks.
  • 10+ years of technical professional experience.
  • Strong proficiency with Apache Spark (PySpark and Spark SQL), Delta Lake, and the Databricks Lakehouse Platform.
  • Hands-on experience with Databricks on Azure and/or AWS (e.g., Azure Databricks, Azure Data Factory, AWS Glue, or similar orchestration tooling).
  • Experience with Unity Catalog for data governance, access control, and lineage.
  • Experience integrating structured and unstructured data sources.
  • Knowledge of SQL, schema design, data modeling, and cloud-based storage patterns (e.g., Delta tables, cloud object storage).
  • Secret clearance.
Preferred Qualifications
  • Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
  • Experience supporting AI/ML pipelines, vectorization, or embedding generation, ideally using MLflow, Databricks Mosaic AI, Model Serving, or Vector Search.
  • Experience with Delta Live Tables and streaming pipelines (Structured Streaming / Auto Loader).
  • Azure, AWS, or multi-cloud data ecosystem familiarity.
  • Experience with data governance, access controls, or sensitive data protection (Unity Catalog, row/column-level security).
  • Exposure to DevOps/CI-CD patterns for data workflows (e.g., Databricks Asset Bundles, Repos, Terraform).
  • MuleSoft implementation experience.

N2IA is committed to fostering a diverse and inclusive work environment. We are an Equal Employment Opportunity Employer and encourage applications from all qualified individuals, regardless of gender, race, ethnicity, sexual orientation, disability, or veteran status.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

SiloSmashers, Inc. • Alexandria (VA)

On-site
USD 140,000 - 155,000
Senior Data Engineer
Senior Data Engineer

Accion Labs • United States

Remote
USD 120,000 - 170,000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 160,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Virtue America • New York (NY)

On-site
USD 140,000 - 190,000
Senior Data Engineer, Azure Databricks
Senior Data Engineer, Azure Databricks

Nextech Solutions • United States

Remote
USD 130,000 - 170,000
Data Engineer - Databricks
Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000
Senior Databricks Data Engineer - Hybrid/Onsite
Senior Databricks Data Engineer - Hybrid/Onsite

N2IA Technologies • Washington

Hybrid
USD 150,000 - 190,000
Staff AI & Data Engineer – Databricks
Staff AI & Data Engineer – Databricks

Eliassen Group • United States

Remote
USD 110,000 - 124,000
Databricks Engineer
Databricks Engineer

Quantum Technologies LLC • Washington

On-site
USD 140,000 - 180,000
Databricks Engineer
Databricks Engineer

Ethereum Technologies LLC • Washington

On-site
USD 140,000 - 210,000