Lead Data Engineer (Databricks)

Euclid Innovations; Charlotte (NC); On-site; USD 140,000 - 190,000; Full time

Job summary

Euclid Innovations in Charlotte, NC seeks an experienced Databricks Lead/Architect to design scalable data platforms and steer CCAR regulatory data workflows. You will lead cross-functional teams, shape data models, and ensure governance and security across enterprise Databricks environments.

The role demands 10+ years in data engineering, deep Spark expertise, and strong collaboration with Risk, Finance, and Regulatory Reporting to deliver reliable data solutions.

Qualifications

• 10+ years in data engineering/architecture.
• Hands-on Databricks and Apache Spark experience.
• Experience with CCAR or regulatory reporting.
• Proficiency in Python and SQL.
• Experience with cloud platforms (AWS/Azure/GCP).
• Strong data warehousing and data lake knowledge.
• Excellent stakeholder management skills.

Responsibilities

• Design and architect enterprise Databricks solutions.
• Lead data engineering teams and pipelines.
• Build ETL/ELT processes using Databricks, Spark, and Python.
• Collaborate with Risk, Finance, and Regulatory Reporting teams.
• Design data models and data quality frameworks for regulatory reporting.
• Optimize performance, scalability, and reliability of Databricks workloads.
• Mentor junior engineers and provide technical leadership.
• Ensure compliance with banking regulatory and governance standards.

Skills

Databricks

Apache Spark

Data Architecture

Leadership

Regulatory reporting knowledge

Python

SQL

Cloud platforms

CCAR knowledge

Stakeholder mgmt

Tools

Delta Lake

Unity Catalog

CI/CD pipelines

Job description

We are seeking an experienced Databricks Lead/Architect with strong expertise in CCAR (Comprehensive Capital Analysis and Review) and modern data engineering platforms. The ideal candidate will lead the design, development, and implementation of scalable data solutions using Databricks while supporting regulatory reporting and risk management initiatives within the banking and financial services domain.

Key Responsibilities:

• Design and architect enterprise-scale data solutions using Databricks.
• Lead data engineering teams in developing and optimizing data pipelines.
• Build and maintain ETL/ELT processes using Databricks, Spark, and Python.
• Work closely with Risk, Finance, and Regulatory Reporting teams to support CCAR requirements.
• Design data models and data quality frameworks for regulatory reporting.
• Optimize performance, scalability, and reliability of Databricks workloads.
• Collaborate with business stakeholders, architects, and development teams.
• Ensure compliance with banking regulatory and governance standards.
• Mentor junior engineers and provide technical leadership.

Required Skills:

• 10+ years of Data Engineering/Data Architecture experience.
• Strong hands-on experience with Databricks and Apache Spark.
• Experience with CCAR, Risk Management, Regulatory Reporting, or Finance Data.
• Proficiency in Python, SQL, and ETL development.
• Experience with cloud platforms (AWS, Azure, or GCP).
• Strong understanding of Data Warehousing and Data Lake architectures.
• Excellent communication and stakeholder management skills.

Preferred Skills:

• Experience with Delta Lake, Unity Catalog, and CI/CD pipelines.
i>Knowledge of Basel, Stress Testing, or other regulatory reporting frameworks.c>
• Banking or Financial Services industry experience.