Senior Data Engineer: AI Doc Processing & Ontology

NTT DATA, Inc.; Raleigh (NC); On-site; USD 130,000 - 180,000; Full time

Job summary

NTT DATA, Inc. is seeking a Senior Data Engineer in Charlotte, NC to design and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.

You will build document ingestion, classification, extraction, and validation pipelines for PDFs, emails, Excel files, and other business documents, integrating AI models to extract structured data. Responsibilities include ontology-based data modeling, mapping to standardized concepts, and API development for downstream applications, while

Qualifications

• 7+ years in data engineering/software development/pipeline work
• 4+ years with Databricks and scalable pipelines
• 5+ years with Python, PySpark, SQL
• 3+ years processing unstructured/semi-structured data (PDFs, emails, Excel, JSON)
• 2+ years integrating LLMs/AI models into pipelines
• 3+ years implementing structured data extraction, classification, validation, reconciliation, data quality controls
• 2+ years designing JSON outputs, APIs, data products for downstream consumption
• 2+ years PostgreSQL/Lakebase or similar relational DB
• 2+ years troubleshooting production performance/scalability/concurrency
• Strong cloud data architecture, distributed processing, and pipeline optimization
• Excellent analytical, problem-solving, communication, cross-functional collaboration

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build document ingestion, classification, extraction, validation, and transformation pipelines for various file types.
• Integrate LLMs and AI models into data-processing workflows for structured extraction from documents.
• Troubleshoot false positives, metadata issues, and incorrect field values in extraction pipelines.
• Implement document classification, extraction rules, data validation, reconciliation, and quality controls.
• Transform extracted information into standardized JSON structures for downstream apps and underwriting workflows.
• Design ontology-based data models representing insurance entities and relationships.
• Map extracted attributes to standardized ontology concepts and metadata structures.
• Optimize processing of large Excel/XLSX and SOV files for performance and concurrency.
• Persist data using PostgreSQL, Lakebase, or related databases; develop APIs and data products for integration.
• Collaborate with brokerage, underwriting, AI, and data teams to align outputs with business needs.
• Monitor production pipelines and continually improve reliability and data quality.

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field

Tools

Databricks

Python

PySpark

SQL

PostgreSQL

Lakebase

Delta Lake

Databricks Workflows

Delta Live Tables

AWS

Azure

Google Cloud

Job description

NTT DATA, Inc. is seeking a Senior Data Engineer in Charlotte, NC to design and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.

You will build document ingestion, classification, extraction, and validation pipelines for PDFs, emails, Excel files, and other business documents, integrating AI models to extract structured data. Responsibilities include ontology-based data modeling, mapping to standardized concepts, and API development for downstream applications, while