Data Engineer

Collabera; New York (NY); On-site; USD 96,000 - 110,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401(k) retirement plan

Life insurance

Long-term disability insurance

Short-term disability insurance

Paid parking/public transportation

Paid time off

Paid sick leave

Job summary

Collabera Senior Data Engineer – GenAI & Agentic Data Platforms is a client-facing, lead role responsible for designing and delivering scalable data architectures and pipelines powering machine learning, GenAI, and agentic systems.

The ideal candidate has deep Python/SQL expertise, hands-on Databricks or Snowflake experience, and production-grade AI data solutions including RAG, embeddings, and vector search. Strong MLOps/LLMOps and security practices are required.

Qualifications

• 6+ years of experience in Data Engineering with leadership of technical workstreams.
• Expert-level Python and SQL.
• Hands-on with Databricks or Snowflake (mandatory).
• Experience with Spark and orchestration tools (Airflow/Dagster/Kedro).
• Production AI data solutions including RAG, embeddings, vector search.

Responsibilities

• Design scalable data architectures and pipelines for ML, GenAI, and agentic systems.
• Develop end-to-end RAG solutions with embeddings and vector databases.
• Implement MLOps/LLMOps pipelines, CI/CD, model observability, retraining, and agent evaluation.
• Enforce data security and governance, including PII/PHI protection and RBAC.
• Lead feature engineering workstreams and deliver scalable data solutions.
• Provide mentorship to junior engineers.
• Collaborate with cross-functional teams and client stakeholders.

Skills

Python

SQL

Scala/Java

MLOps/LLMOps

APIs

Leadership

Tools

Databricks

Snowflake

Apache Spark

Airflow

Dagster

Kedro

LangChain

Terraform

AWS

Azure

Job description

Senior Data Engineer – GenAI & Agentic Data Platforms

Duration: 12+ Months

Locations: Atlanta, GA | Boston, MA | Chicago, IL | New Jersey | New York City, NY | Raleigh, NC | San Francisco, CA | Washington, DC

Travel: Client-facing role with 80% and 4 day per week.

Pay Rate: $70 - 80/hr

About the Role

We are seeking an experienced Senior Data Engineer to lead data engineering workstreams supporting Generative AI (GenAI) and Agentic AI solutions for enterprise clients. This role will focus on designing and building scalable data architectures, pipelines, and platforms that power machine learning, GenAI, and agentic systems.

The ideal candidate will have strong hands-on experience with Python, SQL, Databricks or Snowflake, and production-grade AI data solutions, including Retrieval-Augmented Generation (RAG), embeddings, and vector search. Experience working in government, defense, aerospace, utilities, or oil and gas environments is highly desirable.

Key Responsibilities

• Design, develop, and implement scalable data architectures and pipelines to support machine learning, Generative AI, and agentic systems.
• Build and deliver end-to-end Retrieval-Augmented Generation (RAG) solutions, including embeddings, vector databases, context construction, and output handling.
• Develop and implement MLOps and LLMOps capabilities, including CI/CD pipelines for data, model observability, automated retraining, and agent evaluation.
• Implement data security and governance practices, including PII/PHI protection, authentication, and Role-Based Access Control (RBAC).
• Lead feature engineering workstreams and ensure the delivery of scalable and reliable data solutions.
• Provide technical guidance and mentorship to junior engineers.
• Collaborate with cross-functional teams and client stakeholders to deliver AI-driven data engineering solutions.

Required Qualifications

• 6+ years of experience in Data Engineering, including ownership and leadership of technical workstreams.
• Expert-level proficiency in Python and SQL.
• Experience with Scala or Java based on client technology requirements.
• Strong hands-on experience with Databricks or Snowflake (mandatory).
• Experience with Apache Spark and data orchestration tools such as Airflow, Dagster, or Kedro.
• Experience with Lang Chain for building AI and data pipelines.
• Production-level experience with Generative AI or Agentic AI data solutions, including:
• Retrieval-Augmented Generation (RAG)
• Embeddings
• Vector databases and vector search
• Hands-on experience with Terraform and at least one major cloud platform: AWS, Azure.
• Strong experience designing and developing APIs.

Preferred Qualifications

• Experience with agent evaluation and observability tools such as LangSmith, Opik, or Langfuse.
• Familiarity with AI coding agents and development tools such as Cursor, Claude Code, or Codex.
• Previous experience delivering projects within government, defense, aerospace, utilities, or oil and gas industries.
• Strong technical leadership, problem-solving, and client-facing communication skill.

The Company offers the following benefits for this position, subject to applicable eligibility requirements: medical insurance, dental insurance, vision insurance, 401(k) retirement plan, life insurance, long-term disability insurance, short-term disability insurance, paid parking/public transportation, paid time off, paid sick and safe time, hours of paid vacation time, weeks of paid parental leave, and paid holidays annually – as applicable.