Lead Data Engineer: Scalable Pipelines & Architecture

Talent Software Services, Inc.

Charleston (SC)

On-site

USD 120,000 - 180,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

TALENT Software Services is seeking a Senior Data Engineer to lead the design and implementation of robust data solutions across domains. You will mentor peers, shape best practices, and influence data architecture to improve systems, reliability, and data strategy in partnership with product and business stakeholders.

The role requires hands-on experience with SQL, Python, Azure, and Databricks, including Delta Live Tables and Unity Catalog.

Qualifications

  • 5-9 years in data engineering, data modeling, and pipeline development.
  • Expert in SQL and Python for scalable data pipelines.
  • Hands-on experience with Azure and Databricks, including Delta Live Tables and Unity Catalog.
  • Skilled with data integration/orchestration tools such as SnapLogic and Azure Data Factory.
  • Strong use of infrastructure-as-code tools like Terraform for deployment pipelines.
  • Experience with API integration in data pipelines.
  • Familiar with data quality observability tools (e.g., Soda).
  • Proficient in GitHub-based CI/CD workflows.
  • Advanced understanding of dimensional modeling and data warehousing.
  • Comfortable leading agile teams with ownership and collaboration.

Responsibilities

  • Design and implement robust data pipelines across multiple domains.
  • Mentor peers on performance optimization and scalability.
  • Lead design and optimization of data models and storage architectures.
  • Partner with stakeholders to translate business requirements into data structures.
  • Contribute to delivery planning, manage dependencies, and escalate delivery challenges.
  • Standardize data validation frameworks and testing strategies.
  • Lead root cause analysis and data quality improvements.
  • Define SLAs and quality metrics with stakeholders.
  • Automate, monitor, and scale production-grade data pipelines.
  • Develop resilient workflows with retry logic and resource optimization.
  • Document pipelines, data models, and system integrations; enforce standards.
  • Mentor junior engineers in clear coding and documentation habits.

Skills

SQL
Python
Data modeling
Pipeline development
Agile leadership

Tools

Azure Databricks
Delta Live Tables
Unity Catalog
SnapLogic
Azure Data Factory
Jenkins
Terraform
GitHub

Job description

TALENT Software Services is seeking a Senior Data Engineer to lead the design and implementation of robust data solutions across domains. You will mentor peers, shape best practices, and influence data architecture to improve systems, reliability, and data strategy in partnership with product and business stakeholders.

The role requires hands-on experience with SQL, Python, Azure, and Databricks, including Delta Live Tables and Unity Catalog.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer: Scalable Data Lakes & Pipelines
Lead Data Engineer: Scalable Data Lakes & Pipelines

Tectu • United States

Remote
USD 110,000 - 160,000
Lead Data Platform Engineer - Architect & Scale Pipelines
Lead Data Platform Engineer - Architect & Scale Pipelines

Mirion Technologies • United States

On-site
USD 120,000 - 160,000
Data Engineer: Scalable Pipelines & AI Data Platforms
Data Engineer: Scalable Pipelines & AI Data Platforms

Resolve Tech Solutions • Irving (TX)

On-site
USD 110,000 - 150,000
Lead Data Engineer: Scalable Pipelines & AI-Driven Insights
Lead Data Engineer: Scalable Pipelines & AI-Driven Insights

Code and Theory, LLC • Chicago (IL)

On-site
USD 120,000 - 160,000
Senior Data Engineer, DataOps & Pipelines Lead
Senior Data Engineer, DataOps & Pipelines Lead

Insight Global • Atlanta (GA)

On-site
USD 120,000 - 170,000
Senior Data Engineer - Scalable Pipelines & Insights
Senior Data Engineer - Scalable Pipelines & Insights

Partner Company • United States

Remote
USD 90,000 - 130,000
Data Engineer: Build Scalable Pipelines & Insights
Data Engineer: Build Scalable Pipelines & Insights

Compunnel Inc. • Orlando (FL)

On-site
USD 85,000 - 110,000
Lead Data Engineer: Architect Scalable Pipelines
Lead Data Engineer: Architect Scalable Pipelines

Compunnel Inc. • Columbus (OH)

On-site
USD 100,000 - 130,000
Senior Data Engineer - Scalable Pipelines & Cloud Analytics
Senior Data Engineer - Scalable Pipelines & Cloud Analytics

Initvalue • Fremont (CA)

Hybrid
USD 130,000 - 190,000
Senior Data Engineer: Scalable Pipelines & Data Quality
Senior Data Engineer: Scalable Pipelines & Data Quality

TechDigital Group • Burbank (CA)

On-site
USD 120,000 - 150,000