Senior Data Engineer for AI Document Processing & Ontology

NTT DATA; Charlotte (NC); Hybrid; USD 140,000 - 190,000; Full time

Job summary

NTT DATA is seeking a Senior Data Engineer to design and optimize data pipelines using Databricks, Python, PySpark, and SQL. You will build document ingestion and extraction work streams, integrate LLMs into processing workflows, and implement robust data quality controls.

The role involves designing ontology-based data models, mapping attributes to ontology concepts, and ensuring scalable, reliable delivery for underwriting workflows. Collaboration with cross-functional teams is key.

Qualifications

• 7+ years of experience in data engineering, software development, or production data pipeline development.
• 4+ years of hands-on experience with Databricks and scalable data-processing pipelines.
• 5+ years of experience with Python, PySpark, and SQL.
• 3+ years of experience processing unstructured and semi-structured data (PDFs, emails, Excel, JSON).
• 2+ years of experience integrating LLMs or AI models into data-processing pipelines.
• 3+ years of experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build document ingestion, classification, extraction, validation, and transformation pipelines for various documents.
• Integrate LLMs and AI models into data-processing workflows to extract structured information.
• Implement document classification, source-specific extraction rules, data validation, reconciliation, and QC mechanisms.
• Transform extracted information into standardized JSON structures for downstream applications.
• Design ontology-based data models and map attributes to ontology concepts and metadata.

Skills

Databricks

Python

PySpark

SQL

LLMs integration

Data modeling

PostgreSQL

Cloud architectures

API development

Data pipelines

Education

Bachelor's degree in Computer Science / Information Technology / Data Engineering

Tools

Databricks Workflows

Delta Lake

Lakehouse

APIs

SQL tooling

Job description

NTT DATA is seeking a Senior Data Engineer to design and optimize data pipelines using Databricks, Python, PySpark, and SQL. You will build document ingestion and extraction work streams, integrate LLMs into processing workflows, and implement robust data quality controls.

The role involves designing ontology-based data models, mapping attributes to ontology concepts, and ensuring scalable, reliable delivery for underwriting workflows. Collaboration with cross-functional teams is key.