Cloud Data Engineer - PySpark & Data Lakehouse

Piper Companies; United States; Remote; USD 110,000 - 140,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k)

Paid Time Off

Paid Holidays

Job summary

Zachary Piper Solutions is seeking a Data Cloud Engineer to support a long-term Department of Defense modernization program at Fort Meade, MD. The Data Cloud Engineer will design, build, and maintain scalable cloud-based data solutions supporting enterprise modernization and mission-critical analytics within secure DoD environments.

You will work with Azure Synapse, Databricks, AWS Glue; PySpark; implement data lakes/warehouses; ensure data quality, governance, and compliance with RMF, STIG, and

Qualifications

• 10+ years of experience designing and supporting production-grade cloud data pipelines.
• Strong experience with Azure Synapse, Databricks, Azure Data Factory, and/or AWS data services.
• Advanced experience with PySpark, Spark SQL, Python, and SQL.
• Experience with ETL/ELT, CDC, data lakes/warehouses, data quality, and governance.
• Experience implementing CI/CD for cloud data environments.
• Bachelor's degree in a related technical field.
• Active Secret clearance required.

Responsibilities

• Design and maintain scalable ETL/ELT data pipelines using Azure Synapse, Databricks, AWS Glue, and related cloud technologies.
• Develop large-scale data transformations using PySpark, Spark SQL, Python, and SQL.
• Build data ingestion frameworks for structured and unstructured data, including relational databases, CSV, JSON, and REST APIs.
• Implement full and incremental data loads, CDC, and rerunnable pipeline architectures.
• Design and support cloud-based data lakes, data warehouses, and analytics-ready datasets.
• Implement data quality and governance controls including schema validation, RBAC, lineage, cataloging, and credential management.
• Monitor and troubleshoot data pipelines for performance, failures, latency, logging, and operational reliability.
• Implement CI/CD pipelines supporting automated deployments and environment promotion.
• Partner with cloud and cybersecurity teams to maintain compliance with RMF, STIG, FedRAMP, and DoD security requirements.
• Support cloud data migration, integration, and modernization initiatives across Azure and AWS environments.

Skills

ETL/ELT pipelines

PySpark

Python

SQL

CI/CD

RBAC

Data governance

Cloud data pipelines

Education

Bachelor's degree in a related technical field

Tools

Azure Synapse

Databricks

Azure Data Factory

AWS Glue

Job description

Zachary Piper Solutions is seeking a Data Cloud Engineer to support a long-term Department of Defense modernization program at Fort Meade, MD. The Data Cloud Engineer will design, build, and maintain scalable cloud-based data solutions supporting enterprise modernization and mission-critical analytics within secure DoD environments.

You will work with Azure Synapse, Databricks, AWS Glue; PySpark; implement data lakes/warehouses; ensure data quality, governance, and compliance with RMF, STIG, and