Data Cloud Engineer

Piper Companies; Fort Meade (MD); A distancia; USD 125.000 - 140.000; Jornada completa

Descripción de la vacante

Zachary Piper Solutions seeks a Data Cloud Engineer to modernize DoD enterprise data infrastructure, building scalable pipelines and secure data lakes in a remote role based from Fort Meade, VA. You will implement CI/CD for data pipelines, ensure data quality, and collaborate across cybersecurity, cloud, and DevOps teams to meet RMF/DMF requirements.

The role requires 10+ years of experience with Azure Synapse or Databricks, PySpark, and strong Python/SQL skills.

Formación

• 10+ years of experience designing and operating scalable, production-grade data pipelines using Azure Synapse and/or Databricks, with a Bachelor’s degree in Computer Science, Data Science, Engineering, Information Systems, or a related field.
• Strong hands-on PySpark development experience with high-performance ETL pipelines.
• Advanced experience with Spark SQL, Python, and SQL for large-scale data transformation and optimization.
• Experience ingesting structured and unstructured data and implementing full/incremental loads, CDC, and rerunnable pipeline architectures.
• Experience with data governance, RBAC, credential management, and pipeline monitoring.

Responsabilidades

• Design, develop, and maintain production-grade ETL/ELT pipelines with PySpark focusing on performance.
• Build scalable data pipelines using Azure Synapse Analytics, Databricks, AWS Glue, and related cloud technologies.
• Develop ingestion frameworks for structured and unstructured data (relational, JSON, flat files, REST APIs).
• Implement full and incremental data loads, including CDC and late-arriving data.
• Design and maintain cloud data lakes/warehouses and analytics-ready datasets.
• Implement data quality controls, schema validation, and lineage with RBAC and cataloging.
• Monitor pipelines for performance, latency, and reliability; troubleshoot issues.

Conocimientos

PySpark

Spark SQL

Python

SQL

CI/CD

DevOps

RBAC

Credential management

Data governance

Data ingestion

Educación

Bachelor's degree in Computer Science / Data Science / Engineering / Information Systems

Herramientas

Azure Synapse Analytics

Databricks

AWS Glue

Azure Data Factory

Redshift

S3

Descripción del empleo

Zachary Piper Solutions is seeking a Data Cloud Engineer to support a long-term federal modernization program focused on enterprise cloud data solutions and mission-critical analytics within secure DoD environments. This a fully remote position, based out of Fort Meade, VA, and requires an active Secret security clearance*

The Data Cloud Engineer will play a key role in modernizing the program’s enterprise data infrastructure, building the foundation for secure and scalable data integration, and enabling reliable access to mission-critical data for advanced analytics, reporting, and operational decision-making across the DoD environment.

Responsibilities of the Data Cloud Engineer:

• Design, develop, and maintain production-grade ETL/ELT pipelines, with a strong emphasis on hands-on, high-performance PySpark development and Spark SQL optimization.
• Build and maintain scalable data pipelines using Azure Synapse Analytics, Databricks, AWS Glue, and related cloud technologies.
• Develop ingestion frameworks for structured and unstructured data, including relational databases, flat files, JSON, nested data, and REST APIs.
• Implement full and incremental data loads, including CDC, late-arriving records, and rerunnable pipeline architectures.
• Design and maintain cloud data lakes, warehouses, and analytics-ready datasets supporting enterprise reporting and operational decision-making.
• Implement data quality and governance controls, including schema validation/enforcement, schema drift, RBAC, lineage, cataloging, and credential management.
• Monitor and troubleshoot pipelines for performance, latency, failures, logging, alerting, and operational reliability.
• Support CI/CD, automated deployments, rollback strategies, and environment promotions across cloud data environments.
• Collaborate with cybersecurity, cloud, DevOps, and software engineering teams to support RMF, STIG, FedRAMP, and DoD security requirements, data migrations, integrations, and Agile delivery.

Qualifications for the Data Cloud Engineer:

• 10+ years of recent experience designing and operating scalable, production-grade data pipelines using Azure Synapse and/or Databricks, w/ a Bachelors degree in Computer Science, Data Science, Engineering, Information Systems, or a related field.
• Strong hands-on PySpark development experience is required, including demonstrated high-performance coding skills developing ETL pipelines; candidates without this experience will not be considered.
• Advanced experience with Spark SQL, Python, and SQL, including large-scale data transformation and optimization.
• Experience ingesting structured and unstructured data and implementing full/incremental loads, CDC, and rerunnable pipeline architectures.
• Strong experience with data quality, schema validation/enforcement, pipeline monitoring, logging, alerting, and troubleshooting.
• Experience implementing CI/CD practices and automated deployments for data pipelines and cloud environments.
• Experience with data governance, RBAC, and credential management in secure cloud environments.
• Hands-on experience with Azure and/or AWS data services, including Azure Synapse, Azure Data Factory, Databricks, AWS Glue, Redshift, and S3.
• Active Secret clearance*

Compensation for the Data Cloud Engineer:

• Salary: $125,000 - $140,000 *depending on experience*
• Comprehensive Benefits: Medical, Dental, Vision, 401k Plan, PTO, Holidays, Sick Leave if required by law

This job opens for applications on 10/05. Applications for this job will be accepted for at least 30 days from the posting date.

#LI-MS1

#LI-REMOTE

Keywords: Cloud Data Engineer, Data Engineer, Cloud Engineer, Data Cloud Engineer, Senior Data Engineer, Azure Data Engineer, AWS Data Engineer, PySpark, Spark, Apache Spark, Spark SQL, Python, SQL, ETL, ELT, ETL Pipelines, Data Pipelines, Data Engineering, Data Integration, Data Transformation, Data Ingestion, High-Performance Computing, Performance Optimization, Query Optimization, Azure, Microsoft Azure, AWS, Amazon Web Services, Azure Synapse, Azure Synapse Analytics, Synapse, Databricks, Azure Databricks, AWS Glue, Azure Data Factory, ADF, Amazon Redshift, Redshift, Amazon S3, S3, Data Lake, Data Lakes, Data Warehouse, Data Warehousing, Analytics, Data Analytics, Structured Data, Unstructured Data, Relational Databases, JSON, REST API, REST APIs, Flat Files, Nested Data, Full Loads, Incremental Loads, Incremental Loading, Change Data Capture, CDC, Late-Arriving Data, Rerunnable Pipelines, Data Quality, Data Validation, Schema Validation, Schema Enforcement, Schema Drift, Data Governance, Data Lineage, Data Catalog, Data Cataloging, RBAC, Role-Based Access Control, Credential Management, Pipeline Monitoring, Logging, Alerting, Troubleshooting, Pipeline Reliability, CI/CD, Continuous Integration, Continuous Deployment, DevOps, Automated Deployment, Deployment Automation, Environment Promotion, Rollback, Cloud Migration, Data Migration, Cloud Integration, Agile, RMF, Risk Management Framework, STIG, DISA STIG, FedRAMP, DoD, Department of Defense, DISA, Fort Meade, Secret Clearance, Active Secret, Security Clearance, Cleared Data Engineer, Secure Cloud, Secure Data Environment, Production-Grade Data Pipelines, Enterprise Data, Enterprise Data Infrastructure, Mission-Critical Data, Cloud Data Platform, Cloud Data Architecture