Hybrid Data Engineer: End-to-End Big Data Pipelines

West 4th Strategy; Arlington (VA); On-site; USD 150,000 - 165,000; Full time

Job summary

West 4th Strategy seeks an experienced Data Engineer to support the National Guard Bureau in Arlington, VA. This full-time role focuses on designing, building, and maintaining the Minuteman data infrastructure within the DoW analytics platform.

You will collaborate with cross-functional teams to translate data requirements into scalable data models and architectures. Applicants should have strong ETL, Databricks, Python, Spark, SQL, and data governance experience, with a Department of War

Qualifications

• 3+ to 10 years of directly related experience.
• Available to work onsite in Arlington, VA 1-2 days per week
• 3+ years of experience as a Data Engineer
• Proficiency in ETL processes and firsthand experience with technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks
• Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently
• Understanding of data modeling/visualization, database design principles, and data governance practices

Responsibilities

• Design and implement end-to-end data pipelines using ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL
• Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures
• Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing
• Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets
• Leverage big data technologies to efficiently process and analyze large datasets
• Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes
• Troubleshoot data-related problems and provide innovative solutions to address complex data challenges
• Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices

Skills

Databricks

Python

Spark

Scala

JavaScript/JSON

SQL

Jupyter Notebooks

ETL processes

Big data

Education

B.A./B.S. or M.A./M.S. in a related field

Tools

Palantir Foundry

Advana

APIs

Job description

West 4th Strategy seeks an experienced Data Engineer to support the National Guard Bureau in Arlington, VA. This full-time role focuses on designing, building, and maintaining the Minuteman data infrastructure within the DoW analytics platform.

You will collaborate with cross-functional teams to translate data requirements into scalable data models and architectures. Applicants should have strong ETL, Databricks, Python, Spark, SQL, and data governance experience, with a Department of War