Data Lake Engineer

SOSi; Town of Florida (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology and services integrator in defense is seeking a Data Lake Engineer to develop and maintain scalable Data Lake architectures. The role involves optimizing data ingestion processes, ensuring compliance with security standards, and requires an active TS/SCI Clearance. Ideal candidates will have a Master's degree in a related field or extensive experience in managing data architecture projects, along with expertise in cloud platforms and various programming languages. This opportunity offers a unique chance to support mission-driven analytics for a DoD customer.

Qualifications

• Active TS/SCI Clearance required.
• Minimum of eleven years managing complex technical projects.
• Proficiency in cloud security best practices.

Responsibilities

• Design, implement, and maintain scalable Data Lake architectures.
• Optimize ETL pipelines and manage data ingestion.
• Confirm compliance with access policies quarterly.

Skills

Data Lake platform administration

ETL/ELT pipeline development

Cloud integration (AWS, Azure, GCP)

Python

Apache Spark

SQL/NoSQL

Terraform

Education

Master’s degree or higher in Computer Science or related field

Tools

Databricks

Git

Job description

Founded in 1989, SOSi is among the largest private, founder‑owned technology and services integrators in the defense and government services industry. We deliver tailored solutions, tested leadership, and trusted results to enable national security missions worldwide.

Job Description

SOSi is seeking a Data LakeEngineer to support mission requirements for a structured approach to further develop, integrate, and sustain a scalable, federated data ecosystem that enhances interoperability, governance, and mission‑driven analytics for a DoD customer. The primary objective of the program is to bridge the operational gaps between DoD, IC, interagency, and non‑traditional international partners to enable real‑time information sharing, dynamic data integration, and mission‑tailored analytical capabilities.

• The contractor shall design, implement, and maintain scalable Data Lake architectures to support structured and unstructured data ingestion, ensuring efficient data access and retrieval.
• The contractor shall configure and manage the integration interface between the Data Lake and the knowledge graph platform (Stardog), including SPARQL endpoint access, metadata federation, and catalog alignment.
• The contractor shall follow access control policies and usage scope defined by the Government and other coordinated Work Orders.
• The contractor shall confirm compliance with access policies on a quarterly basis and document the results in the Data Governance & Compliance Report.
• The contractor shall optimize ETL pipelines for high‑volume data transformation, ensuring compliance with DoD IL‑4/IL‑5 security standards.
• The contractor shall implement storage tiering strategies and access controls, ensuring data is properly classified, retained, and accessed per DoD governance requirements.
• The contractor shall submit the Data Lake Performance & Optimization Report, detailing ingestion efficiency, access control improvements, and storage utilization metrics.

Qualifications

• Active TS/SCI Clearance.
• Master’s degree or higher (e.g., Ph.D.) in Computer Science, Information Technology, Systems Engineering, Data Science, Business Administration, Engineering Management, or a closely related field,or
  • a minimum of eleven (11) years of experience managing complex technical projects in enterprise data architecture, Databricks administration, and cloud‑based data platforms.
• Knowledge and capability to support Data Lake platform administration and enterprise data architecture for DoD data‑driven projects.
• Skilled in Data Lake platform administration, including workspace management and configuration, cluster optimization and performance tuning, cloud integration, and Unity Catalog integration for secure data governance.
• Proficient in ETL/ELT pipeline development, Delta Lake architecture and optimization, AI/ML workflow integration, and Data Lakehouse optimization for DoD analytics and mission‑critical data workflows.
• Experienced in SysEngOps, DevSecOps, version control systems (Git), and CI/CD pipelines to streamline Data Lake development and deployment.
• Knowledgeable in identity and access management (IAM), role‑based access control (RBAC), and cloud security best practices across AWS, Azure, and GCP.
• Hands‑on expertise in Python, SQL/NoSQL, Apache Spark, Databricks SQL, Terraform, and cloud‑native data services for large‑scale data processing and analytics.

Additional Information

• Normal office conditions

Working at SOSi

All interested individuals will receive consideration and will not be discriminated against for any reason.