Data Engineer: Pipelines, Delta Lake & Databricks

Illumination Works; United States; On-site; USD 110,000 - 170,000; Full time

Job summary

Illumination Works seeks an experienced Data Engineer to design and maintain data pipelines and structures for advanced analytics and data science capabilities. You will collaborate with data scientists and stakeholders to ensure reliable, high-quality data for analysis.

Required 4+ years in data engineering, strong SQL/Python, Databricks, Delta Lake, and experience with ETL/ELT workflows. Must hold a Secret clearance and pass background checks. This role is based in the United States.

Qualifications

• 4+ years of experience as a Data Engineer or in a similar data-focused role.

Responsibilities

• Develop, maintain, and optimize data pipelines using Python, SQL, and Databricks.
• Develop and maintain Databricks notebooks supporting data ingestion, transformation, and delivery.
• Design and maintain data tables, schemas, and Delta Lake structures for analytics and data science needs.
• Collaborate with data scientists to understand data requirements and adapt pipelines accordingly.
• Create complex SQL queries using joins, aggregations, window functions, and advanced techniques.
• Integrate and transform data from multiple structured and semi-structured sources.
• Support data quality through validation, troubleshooting, and issue resolution.
• Document data sources, transformations, flows, schemas, and pipeline configurations.
• Partner with engineers, analysts, and stakeholders to deliver data solutions for mission goals.

Skills

SQL

Python

Databricks

ETL/ELT

Delta Lake

Data modeling

Data integration

Troubleshooting

Communication

Analytical thinking

Education

Bachelor's degree in Computer Science, Mathematics, Data Science, Engineering, or a comparable academic discipline

Tools

Databricks

Job description

Illumination Works seeks an experienced Data Engineer to design and maintain data pipelines and structures for advanced analytics and data science capabilities. You will collaborate with data scientists and stakeholders to ensure reliable, high-quality data for analysis.

Required 4+ years in data engineering, strong SQL/Python, Databricks, Delta Lake, and experience with ETL/ELT workflows. Must hold a Secret clearance and pass background checks. This role is based in the United States.