Data Engineer IV (Lead)

Dexian; Charlotte (NC); On-site; USD 140,000 - 180,000; Full time

Job summary

Dexian is seeking a highly skilled Technical Lead in Data Engineering to steer and shape our data platform, partnering with architects and senior engineers to design scalable, reliable data solutions for batch and streaming workloads.

You will guide the team using AWS Glue (PySpark), Python, Kafka, AWS DMS, AWS Lambda, and Aurora PostgreSQL, ensuring architectural integrity, reliability, and data trust. Dexian is a global Talent + Technology firm with 70+ locations and 10,000+ professionals.

Qualifications

• 8+ years of experience in data engineering or backend data platform development.
• 3+ years in a senior or technical leadership role with shared architectural responsibility.
• Expert-level experience with AWS Glue (PySpark), Python, and distributed data processing.
• Proven experience with Kafka and streaming architectures.
• Hands-on AWS DMS pipelines for batch loads and CDC.
• Advanced SQL and data modeling with Amazon Aurora PostgreSQL.
• Serverless, event-driven architectures using AWS Lambda.
• Strong collaboration, communication, and documentation skills.

Responsibilities

• Provide hands-on leadership within the data engineering team and define platform standards.
• Collaborate with peers to ensure consistency in data ingestion, transformation, streaming, and consumption patterns.
• Contribute to technical roadmaps, design reviews, and architecture discussions.
• Lead development of complex ETL/ELT processes using AWS Glue (PySpark) and Python.
• Build, optimize, and manage Kafka-based streaming pipelines.
• Implement and fine-tune AWS DMS pipelines for full-load and CDC.
• Design and optimize database schemas and performance for Aurora PostgreSQL.
• Mentor data engineers and promote knowledge sharing.
• Engage with analytics, BI, and business teams to translate requirements.

Skills

AWS Glue

PySpark

Python

Kafka

Data modeling

SQL

Communication

Leadership

Tools

AWS DMS

AWS Lambda

Aurora PostgreSQL

Job description

We are looking for a highly skilled and hands-on Technical Lead in Data Engineering to steer and shape our data platform. This leadership role combines deep technical expertise with a collaborative approach to architecture, best practices, and delivery accountability. You will partner closely with existing technical leaders, including architects and senior engineers, to design scalable, reliable data solutions across both batch and streaming environments. Your contributions will help ensure the platform's architectural integrity, operational excellence, and data trustworthiness. This role involves significant direct involvement with technologies such as AWS Glue (PySpark), Python, Kafka, AWS DMS, AWS Lambda, data warehouses, and relational databases, making your technical guidance pivotal within a broader data engineering team.

Key Responsibilities

Position Summary

We are looking for a highly skilled and hands-on Technical Lead in Data Engineering to steer and shape our data platform. This leadership role combines deep technical expertise with a collaborative approach to architecture, best practices, and delivery accountability. You will partner closely with existing technical leaders, including architects and senior engineers, to design scalable, reliable data solutions across both batch and streaming environments. Your contributions will help ensure the platform's architectural integrity, operational excellence, and data trustworthiness. This role involves significant direct involvement with technologies such as AWS Glue (PySpark), Python, Kafka, AWS DMS, AWS Lambda, data warehouses, and relational databases, making your technical guidance pivotal within a broader data engineering team.

Technical Leadership & Collaboration

• Provide hands-on leadership within the data engineering team, working with technical leads and architects to define platform standards and architectural direction.
• Collaborate with peers to ensure consistency in data ingestion, transformation, streaming, and consumption patterns.
• Contribute to technical roadmaps, design reviews, and architecture discussions.
• Offer technical guidance and support escalation while fostering shared ownership and collective decision-making.

Architecture & Engineering Standards

• Assist in designing, evolving, and maintaining scalable, fault-tolerant architectures for both batch and streaming data pipelines.
• Establish and enforce standards for pipeline design, code quality, testing, performance, cost efficiency, security, and operational best practices.
• Drive alignment on ingestion strategies, change data capture patterns, streaming vs. batch processing, and data modeling.

Hands-On Data Engineering

• Lead development of complex ETL/ELT processes using AWS Glue (PySpark) and Python.
• Build, optimize, and manage Kafka-based streaming pipelines, including topic design, partitioning, and consumer patterns.
• Implement and fine-tune AWS DMS pipelines for full-load and change data capture (CDC).
• Develop AWS Lambda functions for automation, orchestration, monitoring, and event-driven workflows.
• Design and optimize database schemas, queries, and performance tuning for data stores like Amazon Aurora PostgreSQL.

Platform Reliability, Performance & Cost

• Share responsibility for platform reliability, scalability, and performance improvements.
• Identify and resolve bottlenecks in Spark jobs, streaming consumers, and database workloads.
• Implement robust error handling, retries, idempotency, and recovery mechanisms.
• Collaborate on cost optimization strategies, capacity planning, and infrastructure enhancements.

Data Quality & Trust

• Partner with data quality engineers to embed validation checks across pipelines.
• Ensure pipelines support reconciliation, auditability, and observability.
• Enforce data freshness, completeness, and accuracy SLAs for downstream consumers and analytics tools.

Mentorship & Team Enablement

• Mentor data engineers through reviews, design discussions, and technical coaching.
• Contribute to growing team capabilities by supporting hiring, onboarding, and skills development.
• Promote knowledge sharing via documentation, demonstrations, and internal forums.

Stakeholder & Cross-Team Engagement

• Work closely with analytics, BI, and business teams to translate requirements into scalable, reliable data solutions.
• Ensure curated datasets are ready for analytics tools.
• Communicate technical decisions, risks, and tradeoffs clearly and collaboratively.

Required Qualifications

• 8+ years of experience in data engineering or backend data platform development.
• 3+ years in a senior or technical leadership role with shared architectural responsibility.
• Expert-level experience with AWS Glue (PySpark), Python, and distributed data processing frameworks.
• Proven experience working with Kafka and streaming architectures.
• Hands-on experience designing, implementing, and supporting AWS DMS pipelines for batch loads and CDC.
• Advanced SQL skills and familiarity with data modeling using relational data stores, particularly Amazon Aurora PostgreSQL.
• Experience developing serverless, event-driven architectures using AWS Lambda.
• Deep understanding of data modeling techniques and design patterns for batch and streaming data processing.
• Strong collaborative skills with technical peers, along with excellent communication and documentation abilities.

Preferred Qualifications

• Experience working with orchestration tools such as Step Functions or Airflow.
• Knowledge of data governance, lineage, cataloging, and metadata management practices.
• Proven ability to optimize Spark workloads for performance and cost-effectiveness.
• Experience operating within enterprise or regulated environments, following compliance standards.

Why This Opportunity May Be Appealing

This role offers the chance to lead and influence the technical evolution of a complex data platform, working with cutting-edge cloud and data technologies. You will collaborate with a team of talented engineers and technical leaders, contribute to shaping architectural standards, and deliver impactful solutions that support critical business initiatives. The position provides a pathway to deepen your expertise in cloud-native data engineering while mentoring others and driving operational excellence.

Dexian stands at the forefront of Talent + Technology solutions with a presence spanning more than 70 locations worldwide and a team exceeding 10,000 professionals. As one of the largest technology and professional staffing companies and one of the largest minority-owned staffing companies in the United States, Dexian combines over 30 years of industry expertise with cutting-edge technologies to deliver comprehensive global services and support. Dexian connects the right talent and the right technology with the right organizations to deliver trajectory-changing results that help everyone achieve their ambitions and goals. To learn more, please visit https://dexian.com/. Dexian is an Equal Opportunity Employer that recruits and hires qualified candidates without regard to race, religion, sex, sexual orientation, gender identity, age, national origin, ancestry, citizenship, disability, or veteran status.