Databricks Data Architect & Engineer

Lincoln Financial Group; Radnor (Delaware County); Hybrid; USD 97,000 - 176,000; Full time

Benefits offered by this job

Career tracks

Leadership training

Financial counseling

Tuition assistance

Flexible work arrangements

Tech tools & training

Job summary

Lincoln Financial Group is seeking a Risk Data Engineer and Architect to lead design and implementation of cloud-based data solutions for the CRO risk analytics team. You will own technical roadmaps, champion SDLC best practices, and build production-grade systems powering analytics and cross-functional risk management.

You will work on a centralized cloud data platform (AWS, Databricks) to design scalable data pipelines, implement data governance, and migrate legacy systems to cloud platforms

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Engineering or related field.
• 5+ years of experience in data engineering or similar role implementing with a variety of on-premises and cloud data management, integration, and analytical technologies.
• Production experience with relational databases such as MySQL, and SQL Server, including designing schemas, writing complex queries, optimizing performance, and ensuring data integrity for a variety of business applications.
• Strong experience with end-to-end data architecture implementing systems that use AWS services related to data storage and management, must have advanced proficiency in SQL, Python, YAML and Bash.
• Thorough understanding of the Software Development Life Cycle (SDLC), including DevOps practices, CI/CD processes, application resiliency, and security measures.
• Excellent communication skills with ability to translate technical concepts for business audiences.
• Data governance experience: catalog tools, data quality frameworks, lineage tracking, and compliance controls is a plus.
• AWS certifications such as SAA, Associate Developer, Data Analytics Specialty, or Databricks certification is a plus.

Responsibilities

• Architect cloud-native data solutions using AWS services (S3, Redshift, Glue, EMR, Lambda, Athena) and Databricks (Unity Catalog, Workflows, MLflow) or similar technology to support risk analytics at scale.
• Build and optimize data pipelines using Python, PySpark, and SQL to integrate structured and unstructured data from internal and external sources.
• Drive modernization initiatives by migrating legacy on-premises systems to cloud platforms while ensuring zero disruption to critical business services.
• Implement data governance frameworks including cataloging, lineage tracking, quality monitoring, and access controls aligned with internal controls and regulatory requirements.
• Collaborate cross-functionally with data scientists, risk managers, and business stakeholders to translate requirements into scalable technical solutions, drive change management and process improvements.
• Develop production-grade code following DevOps practices including CI/CD, automated testing, code reviews, and infrastructure-as-code.
• Manage and optimize relational databases (MySQL, Aurora PostgreSQL, Redshift) for performance, cost efficiency, and data integrity.
• Integrate structured and unstructured data from various internal and external sources.
• Mentor and influence technical standards and architectural decisions across the Risk, Investments Technology organization.
• Monitor and troubleshoot production systems, implementing observability practices and proactive performance tuning.

Skills

Python

PySpark

SQL

AWS

Databricks

Data governance

Education

Bachelor's degree in Computer Science, Information Systems, Engineering or related field

Tools

Unity Catalog

MLflow

Databricks

S3

Redshift

Glue

Job description

Lincoln Financial Group is seeking a Risk Data Engineer and Architect to lead design and implementation of cloud-based data solutions for the CRO risk analytics team. You will own technical roadmaps, champion SDLC best practices, and build production-grade systems powering analytics and cross-functional risk management.

You will work on a centralized cloud data platform (AWS, Databricks) to design scalable data pipelines, implement data governance, and migrate legacy systems to cloud platforms