Data Engineer

D2 Technologies; Herndon (VA); Hybrid; USD 100,000 - 150,000; Full time

Job summary

D2 Technologies in Herndon, VA is seeking an experienced Data Engineer to design and optimize enterprise-grade data platforms on the Databricks Lakehouse Platform. You will build scalable ETL/ELT pipelines, tune Spark and SQL performance, and support analytics and data transformation.

You will collaborate with data architects, software engineers, analysts, and stakeholders to ensure reliable data operations, governance, and security across enterprise platforms.

Qualifications

• Bachelor’s degree or higher in CS/IS/Engineering or related field.
• Strong programming experience with Spark (PySpark) and Spark SQL.
• Expert SQL skills; ability to write, optimize, and troubleshoot complex queries.
• Experience designing, developing, and supporting ETL/ELT pipelines.
• Hands-on Databricks experience and familiarity with Delta Lake and Unity Catalog.
• Experience with Git/version control and DevOps practices.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks and Apache Spark.
• Build and optimize high-performance ETL/ELT pipelines for large data volumes.
• Develop, tune, and optimize complex SQL queries for analytics.
• Analyze requirements and translate them into scalable technical solutions.
• Troubleshoot production issues and optimize data pipeline performance.
• Ensure data quality, integrity, security, and governance across platforms.
• Collaborate with cross-functional teams and participate in code reviews.
• Create documentation for solutions, deployments, and operations.
• Support production releases and ongoing operations.

Skills

Spark programming

SQL proficiency

Data modeling

ETL/ELT design

Git/version control

Communication

Problem solving

Security & governance

Education

Bachelor’s degree in CS/IS/Engineering

Tools

Databricks

Databricks Lakehouse Platform

Apache Spark

PySpark

Spark SQL

SQL

ETL/ELT

Delta Lake

Git

Azure

AWS

Google Cloud Platform

Databricks Delta Live Tables

Unity Catalog

PL/SQL

CI/CD

Job description

D2 Technologies is hiring an experienced Data Engineer for a hybrid role based in Herndon, VA (US Federal project). In this position, you will design and optimize enterprise-grade data platforms on the Databricks Lakehouse Platform, building scalable ETL/ELT pipelines and improving Spark and SQL performance to support analytics and data transformation.

You’ll work at the intersection of data engineering and engineering best practices, collaborating with data architects, software engineers, analysts, and business stakeholders. The role includes production support, troubleshooting, and documentation to help ensure reliable data operations, governance, and security across enterprise data platforms.

What you’ll do

• Design, develop, and maintain scalable data engineering solutions using Databricks and Apache Spark
• Build and optimize high-performance ETL/ELT pipelines to process large volumes of structured and unstructured data
• Develop, tune, and optimize complex SQL queries for data transformation and analytics
• Analyze business requirements and translate them into scalable technical solutions
• Troubleshoot production issues and optimize data pipeline performance
• Ensure data quality, integrity, security, and governance across enterprise data platforms
• Collaborate with cross-functional teams and participate in code reviews using software engineering best practices
• Create and maintain technical documentation for solutions, deployments, and operational support
• Support production releases and provide operational support as needed

What you bring

• U.S. Citizenship required
• Top Secret (TS) Security Clearance or Public Trust Clearance preferred
• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience)
• Hands-on experience developing data engineering solutions using Databricks, including strong experience with the Databricks Lakehouse Platform
• Strong programming experience with Apache Spark (including PySpark / Spark SQL)
• Expert-level SQL skills, including writing, optimizing, and troubleshooting complex queries
• Experience designing, developing, and supporting ETL/ELT pipelines
• Strong understanding of data modeling, data warehousing, and Delta Lake
• Experience using Git or other version control systems
• Excellent verbal and written communication skills

Tools you’ll work with

Databricks, Databricks Lakehouse Platform, Apache Spark, PySpark, Spark SQL, SQL, ETL, ELT, Delta Lake, Git, Azure, AWS, Google Cloud Platform, Databricks Delta Live Tables (DLT), Unity Catalog, PL/SQL, CI/CD

Preferred

• Strong analytical and problem-solving skills
• Strong SQL query writing, optimization, and performance tuning skills
• Experience with Azure, AWS, or Google Cloud Platform
• Knowledge of CI/CD pipelines and DevOps practices
• Experience with Databricks Delta Live Tables (DLT) and Unity Catalog
• Nice to have: Experience with PL/SQL

Compensation

$100,000 - $150,000 per year (negotiable).

Work location: Hybrid remote in Herndon, VA 20192