Senior Data Engineer: AI-Powered Doc Pipelines (Databricks)

NTT Ltd; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

NTT Ltd. is seeking a Senior Data Engineer in Charlotte, NC to design and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.

The role focuses on structured extraction from unstructured content and ontology-based modeling to deliver standardized JSON outputs for underwriting workflows. You will integrate LLMs into data-processing workflows, troubleshoot metadata issues, and collaborate with underwriting and AI teams to ensure data quality and scalable performance.

Qualifications

• Bachelor’s degree in a relevant field or equivalent experience.
• 7+ years in data or production data pipelines.
• 4+ years with Databricks and scalable pipelines.
• 5+ years with Python, PySpark, SQL.
• 2+ years integrating LLMs/AI models into workflows.
• Experience building structured data extraction and data quality controls.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build ingestion, extraction, classification, validation, and transformation pipelines for documents (PDFs, emails, Excel files).
• Integrate LLMs and AI models into data-processing workflows to extract structured information.
• Troubleshoot false positives and incorrect metadata from extraction pipelines.
• Implement classification, validation, reconciliation, and data-quality controls.
• Transform extracted information into standardized JSON structures for underwriting workflows.
• Design ontology-based data models representing insurance entities and relationships.
• Map extracted attributes to ontology concepts and metadata structures.
• Optimize processing for large Excel/SOV files; ensure performance and concurrency.
• Persist data using PostgreSQL, Lakebase, or similar; develop APIs and data products.
• Collaborate with brokerage, underwriting, AI, and data teams; monitor pipelines and improve reliability.

Skills

Data modeling

Cross-functional collaboration

Problem solving

Structured extraction

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equivalent.

Tools

Databricks

Python

PySpark

SQL

LLMs

AI models

PostgreSQL

Lakebase

JSON

APIs

Excel/XLSX

PDFs

Emails

Delta Lake

Lakehouse

Databricks Workflows

Delta Live Tables

AWS

Azure

Google Cloud

Job description

NTT Ltd. is seeking a Senior Data Engineer in Charlotte, NC to design and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.

The role focuses on structured extraction from unstructured content and ontology-based modeling to deliver standardized JSON outputs for underwriting workflows. You will integrate LLMs into data-processing workflows, troubleshoot metadata issues, and collaborate with underwriting and AI teams to ensure data quality and scalable performance.