Data Lake Engineer

Prescient Edge; Town of Florida (NY); On-site; USD 100,000 - 130,000; Full time

Benefits offered by this job

Competitive salary with performance bonuses

Comprehensive healthcare benefits

Substantial retirement plan

Career development opportunities

Job summary

Prescient Edge is looking for a Data Lake Engineer to support a Federal Government client in New York. The ideal candidate will design and maintain Data Lake architectures, ensuring effective data ingestion and compliance with security standards.

This position offers a competitive salary with benefits including healthcare, retirement plans, and opportunities for career development. An active TS/SCI clearance is required.

Qualifications

• Possess knowledge and capability to support Data Lake platform administration.
• Skilled in Data Lake Platform administration, including Unity Catalog integration.
• Proficient in ETL pipeline development, Delta Lake architecture.

Responsibilities

• Design, implement, and maintain scalable Data Lake architectures.
• Optimize ETL pipelines for high-volume data transformation.
• Ensure compliance with DoD governance requirements.

Skills

Data Lake platform administration

ETL pipeline development

Python

SQL/NoSQL

Apache Spark

Terraform

Cloud security best practices

Education

Master’s degree or higher in related fields

Tools

Databricks

AWS

Azure

GCP

Job description

Prescient Edge is seeking aData Lake Engineerto support a Federal Government client.

Please note that the availability of this position is contingent upon contract award.

AtPrescient Edge, we believe that acting with integrity and serving our employees is the key to everyone's success. To that end, we provide employees with a best-in‑class benefits package that includes:

• A competitive salary with performance bonus opportunities.
• Comprehensive healthcare benefits, including medical, vision, dental, and orthodontia coverage.
• A substantial retirement plan with no vesting schedule.
• Career development opportunities, including on‑the‑job training, tuition reimbursement, and networking.
• A positive work environment where employees are respected, supported, and engaged.

Description

• Design, implement, and maintain scalable Data Lake architectures to support structured and unstructured data ingestion, ensuring efficient data access and retrieval.
• Configure and manage the integration interface between the Data Lake and the managed knowledge graph platform (Stardog), including SPARQL endpoint access, metadata federation, and catalog alignment.
• Follow access control policies and usage scope.
• Confirm compliance with access policies on a quarterly basis and document the results in the Data Governance & Compliance Report.
• Optimize ETL pipelines for high‑volume data transformation, ensuring compliance with DoD IL‑4/IL‑5 security standards.
• Implement storage tiering strategies and access controls, ensuring data is properly classified, retained, and accessed per DoD governance requirements.
• Submit the Data Lake Performance & Optimization Report, detailing ingestion efficiency, access control improvements, and storage utilization metrics.

Job Requirements

Experience

• Possess the knowledge and capability to support Data Lake platform administration and enterprise data architecture for DoD data‑driven projects.
• Skilled in Data Lake Platform administration, including workspace management, cluster optimization, and Unity Catalog integration for secure data governance.
• Proficient in ETL pipeline development, Delta Lake architecture, and Data Lakehouse optimization for DoD analytics and mission critical data workflows.
• Experienced in SysEngOps, DevSecOps, version control systems (Git), and CI/CD pipelines to streamline Data Lake development and deployment.
• Knowledgeable in identity and access management (IAM), role-based access control (RBAC), and cloud security best practices across AWS, Azure, and GCP.
• Hands‑on expertise in Python, SQL/NoSQL, Apache Spark, Databricks SQL, Terraform, and cloud native data services for large‑scale data processing and analytics.
• Has strong background in:
  • Databricks platform administration, including workspace configuration, Unity Catalog, cluster performance tuning, and cloud integration.
  • Data pipeline development (ETL/ELT), Delta Lake optimization, and AI/ML workflow integration.
  • SysEngOps, DevSecOps, version control systems (Git), and CI/CD pipelines for automated deployment and governance.
  • Cloud security, role-based access control (RBAC), and compliance with DoD data policies across AWS, Azure, and GCP.

Education

• A Master’s degree or higher (e.g. Ph.D.) in Computer Science, Information Technology, Systems Engineering, Data Science, Business Administration, Engineering Management, or a closely related field; or a minimum of 11+ years of experience managing complex technical projects in enterprise data architecture, Databricks administration, and cloud‑based data platforms.

Security clearance

• Active TS/SCI clearance.

Location

Prescient Edge is a Veteran‑Owned Small Business (VOSB) founded as a counterintelligence (CI) and Human Intelligence (HUMINT) company in 2008. We are a global operations and solutions integrator delivering full‑spectrum intelligence analysis support, training, security, and RD&E support solutions to the Department of Defense and throughout the intelligence community. Prescient Edge is an Equal Opportunity Employer (EEO). All applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other characteristic that is protected by law. We strive to foster equity and inclusion throughout our organization because we believe that diversity of thought is critical for creating a safe and engaging work environment while also enabling the organization’s success.