Lead Data Engineer

Jobtailor; South Carolina; On-site; USD 120,000 - 170,000; Full time

Job summary

Jobtailor is seeking a senior data engineer to design, develop, and support reference data applications and APIs across client, counterparty, issuer, and legal entity domains.

You will build cloud-native data pipelines using AWS services and mentor junior engineers while collaborating with stakeholders and architecture teams to ensure data quality and governance.

Qualifications

• 7+ years in data or software engineering.
• 2+ years in technical lead or people leadership.
• Experience with cloud-native apps, APIs, and data platforms.
• Experience with event-driven architectures and data pipelines.
• Bachelor’s degree in related field or equivalent.

Responsibilities

• Lead design, development, and support of reference data applications, APIs, and data services.
• Partner with stakeholders, architects, and engineering teams to deliver reference data solutions.
• Develop and maintain cloud-native data pipelines using AWS Glue/Lambda/EventBridge/S3/RDS/Kafka.
• Support modernization of legacy reference data platforms via cloud migrations.
• Integrate third-party reference data sources (D&B, S&P Global) for enrichment/validation/compliance.
• Develop and support APIs and data services for CRM, compliance, reporting, operations, and trading.
• Implement data quality controls, validation, matching, deduplication, survivorship, and lineage.
• Collaborate with data governance and architecture teams on standards and metadata.
• Participate in Agile planning, backlog refinement, estimation, sprints with Jira.
• Provide technical leadership and mentorship to junior engineers and project staff.

Skills

Cloud-native data eng

Technical leadership

API and data services

Agile / Jira

Data governance

Mentoring

Education

Bachelor’s degree in CS/IT/Engineering

Tools

AWS Glue

AWS Lambda

AWS EventBridge

Amazon S3

Amazon RDS

Kafka / MSK

Dun & Bradstreet

S&P Global

Job description

Responsibilities

• Lead the design, development, and support of party reference data applications, APIs, and data services.
• Partner with business stakeholders, architects, and engineering teams to deliver reference data solutions supporting client, counterparty, issuer, and legal entity domains.
• Develop and maintain cloud-native data pipelines utilizing AWS services such as Glue, Lambda, EventBridge, S3, RDS, and Kafka/MSK.
• Support the modernization of legacy reference data platforms through migration to scalable cloud-based solutions.
• Integrate third‑party reference data sources including Dun & Bradstreet, S&P Global, and other data providers to support enrichment, validation, and compliance requirements.
• Develop and support APIs and data services consumed by CRM, compliance, reporting, operations, and trading platforms.
• Implement data quality controls, validation rules, matching and deduplication processes, survivorship logic, and lineage tracking capabilities.
• Collaborate with data governance and architecture teams to support reference data standards, metadata management, and best practices.
• Participate in Agile planning, backlog refinement, estimation, sprint execution, and delivery activities using Jira.
• Provide technical leadership and mentorship to junior engineers and project team members.

Requirements

• Minimum of 7 years of experience in data engineering, software engineering, cloud engineering, or related technical disciplines.
• Minimum of 2 years of experience serving as a technical lead, project lead, or people leader.
• Experience developing and supporting cloud-native applications, APIs, and data platforms.
• Experience developing event‑driven architectures, data pipelines, and distributed data services.
• Bachelor’s degree in Computer Science, Information Technology, Engineering, Mathematics, or a related technical field (or equivalent combination of education and relevant work experience).

Core Competencies

Demonstrates expertise in designing and developing cloud‑native data applications and APIs, with a strong focus on data quality, governance, and integration of third‑party data sources. Proven ability to lead technical teams and collaborate with stakeholders to deliver scalable reference data solutions.