Data Engineer - Big Data, AI/ML Pipelines

Probity Inc.; Chantilly (VA); On-site; USD 120,000 - 160,000; Full time

Job summary

Probity Inc. is seeking a Data Engineer with active Top Secret/SCI clearance to design, ingest, cleanse, transform, and display large data sets. The role emphasizes building scalable data pipelines in a Big Data environment and collaborating with data scientists on AI/ML initiatives.

Requirements include 10+ years of relevant software development with Python/Java and strong SQL, ETL, and Linux experience. A TS/SCI with polygraph clearance is mandatory for consideration.

Qualifications

• 10+ years of software development experience with data focus.
• Experience with Python, Java, and SQL in Big Data.
• Experience with data lifecycle from ingest to display.
• Hands-on Python techs incl. async, environment management, FastAPI.
• Experience with ETL tasks in Linux and data pipelines.

Responsibilities

• Design and optimize large-scale data ingestion, cleansing, and transformation systems.
• Develop ad-hoc ETL pipelines for new data sources.
• Leverage Big Data for prototypical AI/ML analytics.
• Create custom solutions to ingest and exploit data sources.
• Develop data profiling, deduping, and matching logic.
• Maintain Data Layer documentation and collaborate with data scientists.
• Collaborate with vendors and users to improve methods.
• Provide light admin for a small number of Linux servers.
• Communicate risks and constraints of software solutions.

Skills

Python

Java

SQL

Data analysis

ETL

Education

High School Diploma/GED

Tools

Apache Airflow

FastAPI

RESTful APIs

Linux

Job description

Probity Inc. is seeking a Data Engineer with active Top Secret/SCI clearance to design, ingest, cleanse, transform, and display large data sets. The role emphasizes building scalable data pipelines in a Big Data environment and collaborating with data scientists on AI/ML initiatives.

Requirements include 10+ years of relevant software development with Python/Java and strong SQL, ETL, and Linux experience. A TS/SCI with polygraph clearance is mandatory for consideration.