AI-First Data Engineer Lead, Databricks Lakehouse

Capital Group; Los Angeles (CA); On-site; USD 202,000 - 342,000; Full time

Benefits offered by this job

Time-away from day one

Flexible work options

2-for-1 matching gifts

Professional development resources

Job summary

Capital Group is seeking a Data Engineer Lead in the Capital Solutions Group Technology (CSGT) to set technical direction for the data platform and data products powering portfolio construction and investment research. This senior IC role involves hands-on coding, architecture ownership, and close collaboration with investment professionals and product management.

You will lead the design and delivery of large-scale data pipelines, enable AI-first engineering, and build reusable patterns for

Qualifications

• Ten-plus years of data or software engineering experience with technical leadership.
• Strong Python and SQL skills.
• Production experience with Databricks on AWS (PySpark, Delta Lake, Unity Catalog, Databricks Jobs, Databricks SQL, Asset Bundles).
• Experience building and maintaining Airflow pipelines with reliable retries and dependencies.
• Strong data modeling, governance, and software quality practices (CI/CD, tests).
• Experience enabling AI/NL-SQL tools and responsible AI patterns.
• Ability to lead architecture discussions and mentor engineers.

Responsibilities

• Set data engineering strategy and roadmap for the team (Lakehouse on Databricks/AWS).
• Design and build data ingestion, transformation, and serving pipelines.
• Analyze datasets and fit them into data domains and subject areas.
• Lead cross-team data initiatives from requirements to production support.
• Advance AI-first engineering practices and agent-driven workflows.
• Develop reusable agent patterns for profiling, mapping, testing, and incident investigation.
• Ensure data understandability and governance with metadata, lineage, and access controls.
• Define testing strategy across platform layers and integrate quality metrics into CI/CD.
• Partner with investment professionals to align data capabilities with business outcomes.
• Mentor engineers, lead reviews, and raise engineering bar.

Skills

Python

SQL

Databricks AWS

Airflow

dbt

PySpark

Delta Lake

Unity Catalog

Data modeling

Governance

CI/CD

AI-first

Leadership

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks

AWS

Job description

Capital Group is seeking a Data Engineer Lead in the Capital Solutions Group Technology (CSGT) to set technical direction for the data platform and data products powering portfolio construction and investment research. This senior IC role involves hands-on coding, architecture ownership, and close collaboration with investment professionals and product management.

You will lead the design and delivery of large-scale data pipelines, enable AI-first engineering, and build reusable patterns for