Senior Data Engineer - Azure, Spark & Databricks Expert

Prometheus Federal Services; United States; Remote; USD 130,000 - 155,000; Full time

Benefits offered by this job

Health coverage

Retirement plan

Paid time off

Job summary

Prometheus Federal Services is seeking an experienced Data Engineer to support VA program analytics. You will design and maintain scalable data pipelines across on‑prem and cloud environments, focusing on data quality, performance, and accessibility.

The role requires strong SQL, Spark, Databricks, and Azure expertise, with collaboration across developers, data scientists, and stakeholders to deliver reliable analytics products for VA programs.

Qualifications

• Bachelor’s degree required.
• 5+ years of data engineering, ETL/ELT or DBA experience.
• Strong proficiency in SQL, Python, PySpark, Spark, and Databricks, incl. performance tuning and indexing.

Responsibilities

• Design, build, deploy, and maintain scalable ETL/ELT pipelines using SQL, Spark, Databricks, and Azure‑native tools.
• Implement workflow orchestration, scheduling, and monitoring for reliable data delivery.
• Develop data quality monitoring and observability to track lineage and integrity across pipelines.
• Optimize ingestion, transformation, and storage for performance and cost.

Skills

SQL

Python

PySpark

Spark

Databricks

Azure

GitHub

GitLab

GitHub Actions

Education

Bachelor’s degree

Tools

Azure Data Factory

Azure Synapse

Azure Data Lake Storage

Azure SQL

SQL Server

PostgreSQL

Job description

Prometheus Federal Services is seeking an experienced Data Engineer to support VA program analytics. You will design and maintain scalable data pipelines across on‑prem and cloud environments, focusing on data quality, performance, and accessibility.

The role requires strong SQL, Spark, Databricks, and Azure expertise, with collaboration across developers, data scientists, and stakeholders to deliver reliable analytics products for VA programs.