Data Engineer: High-Throughput Pipelines & Delta Lake

Nyla Technology Solutions; Arlington (VA); Hybrid; USD 110,000 - 160,000; Full time

Benefits offered by this job

Nyla FLEX program

4 weeks annual leave

11 holidays

Medical, dental, vision coverage

Job summary

Nyla Technology Solutions in Arlington, VA is seeking a Data Engineer to design, build, and optimize high-throughput data pipelines. You will implement automated data extraction and ingestion using PySpark and Databricks Unity Catalog, supporting enterprise observability and DoW CDAO analytics.

You will collaborate with Platform and Solution Architects to connect Delta Tables with Vector Databases under RBAC, and work on cloud infrastructure optimization to deliver resilient, scalable data

Qualifications

• Proficiency in Python, PySpark, and Databricks for building automated data ingestion pipelines.
• Experience with Delta Lake architecture, Delta Tables, and Unity Catalog for versioning, backups, and rollbacks.
• Strong SQL and ETL/ELT automation skills.
• Familiarity with RBAC and connecting data to Vector Databases.
• Active Top Secret, SCI Eligible.
• 0-5+ years of progressive professional experience in data engineering or related field.
• Bachelor’s Degree in Computer Science, Data Science, Information Technology, or related field.

Responsibilities

• Pipeline Engineering: Develop and operate end-to-end, automated data ingestion pipelines using PySpark and Databricks.
• Data Architecture: Maintain and optimize data structures using Delta Tables to enable seamless data versioning, backups, and rollback safeguards.
• Architectural Integration: Coordinate with Platform and Solution Architects to connect Delta Tables with Vector Databases while enforcing RBAC.

Skills

Python

PySpark

Databricks

Delta Tables

Delta Lake

Unity Catalog

SQL

RBAC

Vector Databases

Education

Bachelor's Degree in Computer Science, Data Science, Information Technology, or a related field

Tools

Databricks

Unity Catalog

Delta Lake

Delta Tables

Job description

Nyla Technology Solutions in Arlington, VA is seeking a Data Engineer to design, build, and optimize high-throughput data pipelines. You will implement automated data extraction and ingestion using PySpark and Databricks Unity Catalog, supporting enterprise observability and DoW CDAO analytics.

You will collaborate with Platform and Solution Architects to connect Delta Tables with Vector Databases under RBAC, and work on cloud infrastructure optimization to deliver resilient, scalable data