Senior Data Engineer

NTT Ltd; Charlotte (NC); Presencial; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

NTT Ltd. is seeking a Senior Data Engineer in Charlotte, NC to design and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.

The role focuses on structured extraction from unstructured content and ontology-based modeling to deliver standardized JSON outputs for underwriting workflows. You will integrate LLMs into data-processing workflows, troubleshoot metadata issues, and collaborate with underwriting and AI teams to ensure data quality and scalable performance.

Formación

• Bachelor’s degree in a relevant field or equivalent experience.
• 7+ years in data or production data pipelines.
• 4+ years with Databricks and scalable pipelines.
• 5+ years with Python, PySpark, SQL.
• 2+ years integrating LLMs/AI models into workflows.
• Experience building structured data extraction and data quality controls.

Responsabilidades

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build ingestion, extraction, classification, validation, and transformation pipelines for documents (PDFs, emails, Excel files).
• Integrate LLMs and AI models into data-processing workflows to extract structured information.
• Troubleshoot false positives and incorrect metadata from extraction pipelines.
• Implement classification, validation, reconciliation, and data-quality controls.
• Transform extracted information into standardized JSON structures for underwriting workflows.
• Design ontology-based data models representing insurance entities and relationships.
• Map extracted attributes to ontology concepts and metadata structures.
• Optimize processing for large Excel/SOV files; ensure performance and concurrency.
• Persist data using PostgreSQL, Lakebase, or similar; develop APIs and data products.
• Collaborate with brokerage, underwriting, AI, and data teams; monitor pipelines and improve reliability.

Conocimientos

Data modeling

Cross-functional collaboration

Problem solving

Structured extraction

Educación

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equivalent.

Herramientas

Databricks

Python

PySpark

SQL

LLMs

AI models

PostgreSQL

Lakebase

JSON

APIs

Excel/XLSX

PDFs

Emails

Delta Lake

Lakehouse

Databricks Workflows

Delta Live Tables

AWS

Azure

Google Cloud

Descripción del empleo

The Senior Data Engineer role at NTT Ltd focuses on designing and optimizing scalable data pipelines with Databricks and related technologies to support AI-driven document processing. The position emphasizes structured extraction from unstructured and semi-structured documents, ontology-based modeling, and delivery of standardized JSON outputs for downstream underwriting workflows.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Python, PySpark, and SQL.
• Build ingestion, extraction, classification, validation, and transformation pipelines for documents including PDFs, emails, attachments, and Excel/XLSX files, as well as insurance applications and Statements of Values (SOVs).
• Integrate LLMs and AI models into data-processing workflows to extract structured information from unstructured and semi-structured content.
• Troubleshoot false positives, incorrect metadata, and inaccurate field values produced by LLM-based extraction pipelines.
• Implement document classification, source-specific extraction rules, data validation, reconciliation, and quality-control mechanisms.
• Transform extracted information into standardized JSON structures for downstream applications and underwriting workflows.
• Design and implement ontology-based data models representing insurance entities, business concepts, and relationships.
• Map extracted document attributes to standardized ontology concepts and metadata structures.
• Optimize processing for large and complex Excel/XLSX and SOV files, including performance, scalability, and concurrency challenges.
• Implement reliable data persistence using PostgreSQL, Lakebase, or related database technologies.
• Develop and maintain APIs and data products to support downstream application integration.
• Collaborate with brokerage, underwriting, AI, and data engineering teams to align extraction outputs with business requirements.
• Monitor production pipelines, troubleshoot data-processing issues, and continuously improve reliability, performance, and data quality.

Required Qualifications

• Minimum 7+ years of experience in data engineering, software development, or production data pipeline development.
• Minimum 4+ years of hands‑on experience with Databricks and scalable data-processing pipelines.
• Minimum 5+ years of experience with Python, PySpark, and SQL.
• Minimum 3+ years of experience processing unstructured and semi-structured data, including PDFs, emails, Excel files, JSON, and business documents.
• Minimum 2+ years of experience integrating LLMs or AI models into document-processing or data-extraction pipelines.
• Minimum 3+ years of experience implementing structured data extraction, document classification, validation, reconciliation, and data-quality controls.
• Strong understanding of data modeling, schemas, metadata, entity relationships, and ontology or semantic modeling concepts.
• Minimum 2+ years of experience designing JSON outputs, APIs, and data products for downstream consumption.
• Minimum 2+ years of experience with PostgreSQL, Lakebase, or similar relational database technologies.
• Minimum 2+ years of experience troubleshooting production performance, scalability, concurrency, and data-processing issues.
• Strong understanding of cloud-based data architectures, distributed processing, and pipeline optimization.
• Strong analytical, problem‑solving, communication, and cross‑functional collaboration skills.

Technologies

Databricks, Python, PySpark, SQL, LLMs, AI models, PostgreSQL, Lakebase, JSON, APIs, Excel/XLSX, PDFs, emails

Education

Bachelor’s degree in Computer Science, Information Technology, Data Engineering, Engineering, or a related field, or equivalent.

Location

Charlotte, North Carolina, United States (onsite)

Travel

NA

Nice to Have

• Experience in insurance, commercial insurance, brokerage, underwriting, or risk management.
• Experience processing insurance applications, Statements of Values (SOVs), policy documents, and underwriting submissions.
• Experience with LLM frameworks, prompt engineering, structured outputs, and AI evaluation techniques.
• Experience building custom ontologies, knowledge graphs, or semantic data layers.
• Experience with document AI, OCR, information extraction, and document intelligence platforms.
• Experience with Delta Lake, Lakehouse architecture, Databricks Workflows, and Delta Live Tables.
• Experience with AWS, Azure, or Google Cloud.
• Experience optimizing large-scale document ingestion and distributed processing workloads.