Senior Data Engineer – Azure, Spark & Databricks Pipelines

Prometheus Federal Services; Fairfax (VA); On-site; USD 130,000 - 155,000; Full time

Benefits offered by this job

Health, dental, vision insurance

Flexible spending accounts

Disability and life insurance

Retirement plan

Paid time off

Other programs

Job summary

Prometheus Federal Services is seeking an experienced Data Engineer to support VA programs. The role focuses on building scalable data pipelines, optimizing databases, and ensuring data quality across on‑prem and cloud environments.

You will collaborate with developers, data scientists, and stakeholders to deliver reliable, high‑performance data products that improve accessibility and operational efficiency for VA analytics and reporting.

Qualifications

• Bachelor’s degree and 5+ years in data engineering or DBA work.
• Strong SQL, Python, PySpark and Databricks for ingestion, cleaning, and transformation.
• Experience with Azure analytics services and production relational databases.

Responsibilities

• Design, build, deploy, and maintain scalable ETL/ELT pipelines with SQL, Spark, Databricks, and Azure tools.
• Implement workflow orchestration, scheduling, and automated data delivery.
• Develop data quality monitoring and lineage tracking across pipelines and databases.
• Optimize ingestion, transformation, and storage for performance and cost.
• Administer and optimize SQL Server, Azure SQL, and PostgreSQL databases.
• Apply security, access controls, and auditing per VA requirements.
• Support backup strategies, disaster recovery, and high availability.
• Troubleshoot data issues and drive continuous improvement of data engineering processes.
• Create automated quality checks and anomaly detection; build monitoring dashboards.

Skills

SQL

Python

PySpark

Spark

Databricks

Azure

DBA practices

Education

Bachelor’s degree

Tools

Azure Data Factory

Azure Synapse

Azure Data Lake Storage

SQL Server

PostgreSQL

GitHub Actions

GitHub

GitLab

Job description

Prometheus Federal Services is seeking an experienced Data Engineer to support VA programs. The role focuses on building scalable data pipelines, optimizing databases, and ensuring data quality across on‑prem and cloud environments.

You will collaborate with developers, data scientists, and stakeholders to deliver reliable, high‑performance data products that improve accessibility and operational efficiency for VA analytics and reporting.