Lead Data Engineer

Bartech Staffing; Charlotte (NC); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Bartech Staffing in Charlotte, NC is seeking a Lead Data Engineer to own end-to-end data initiatives on AWS. You will design, implement, and govern data pipelines across data lake and warehouse environments, while mentoring engineers and driving architectural decisions.

The role emphasizes governance, observability, and scalable cloud-native architectures, with a hybrid work model based in Charlotte and a preference for local candidates.

Qualifications

• 8+ years in Data Engineering with 5+ years on AWS-based platforms.
• Expertise with S3, EMR, Glue Jobs, Lambda, Athena, CloudWatch, Step Functions.
• Experience building data lakes and data warehouses (Lake Formation, Redshift, Athena).
• Kafka-based streaming experience (Confluent Kafka preferred).
• Strong SQL, data modeling (dimensional, data vault) and large-scale DW design.
• Hands-on Python and PySpark; Terraform for IaC; CI/CD with GitHub Actions.
• Security, governance, and observability practices across data platforms.
• Ability to lead hands-on development while mentoring teams and driving architecture.

Responsibilities

• Lead design, architecture, and delivery of enterprise data solutions on AWS.
• Collaborate with stakeholders to define technical strategy and scalable solutions.
• Provide technical leadership and mentorship across Data Engineers and teams.
• Drive architectural decisions for scalable, secure, reliable data platforms.
• Design data warehouse and data lake solutions balancing usability and performance.
• Establish standards for data modeling, ETL, monitoring, and operational excellence.
• Oversee end-to-end delivery ensuring alignment with enterprise architecture and regs.
• Manage production support and optimization of AWS-based data platforms.
• Champion data quality, governance, observability, and data stewardship.

Skills

Advanced SQL

Python

PySpark

Data modeling

ETL design

Cloud computing

Leadership

Mentoring

Data governance

Observability

Tools

AWS

Lake Formation

Amazon Redshift

Apache Airflow

Terraform

GitHub Actions

Confluent Kafka

Kafka

PySpark

Databricks

Python

SQL

S3/EMR

LLMs / Generative AI

Job description

The Lead Data Engineer will take technical ownership of enterprise data engineering initiatives, supporting end-to-end design, architecture, and delivery across AWS-based platforms. This hybrid role is based in Charlotte, NC, with a preference for local candidates.

Key Responsibilities

• Lead the design, architecture, and implementation of enterprise data engineering solutions across the AWS ecosystem
• Collaborate with Lead Developers, Data Scientists, Architects, Product Owners, and business stakeholders to define technical strategy and scalable solutions
• Provide technical leadership and mentorship to Data Engineers and development teams, promoting engineering excellence and best practices
• Drive key architectural decisions with Data Architects and Solution Architects to support scalability, security, reliability, and maintainability
• Design and oversee data warehouse and data lake solutions balancing business usability, performance, and long-term sustainability
• Establish engineering standards for data modeling, ETL frameworks, pipeline reliability, monitoring, and operational excellence
• Lead end-to-end solution delivery, ensuring alignment with business requirements, enterprise architecture standards, and regulatory requirements
• Oversee production support and operational management of AWS-based data platforms, including root-cause analysis and performance optimization
• Champion data quality, governance, observability, and data stewardship practices across platforms and teams
• Identify opportunities to modernize data architecture and improve operational efficiency through automation and cloud-native technologies

Required Qualifications

• 8+ years of experience in Data Engineering, including at least 5+ years working extensively within AWS ecosystems
• Expert-level experience with AWS services including S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, and Step Functions
• Experience creating AI applications with AWS Bedrock
• Strong experience designing and implementing enterprise-scale data lake and data warehouse solutions using Lake Formation, Amazon Redshift, and Amazon Athena
• Extensive experience with Kafka-based streaming architectures, preferably Confluent Kafka
• Advanced SQL and data modeling expertise, including dimensional modeling, data vault, and large-scale data warehousing solutions
• Deep experience designing, developing, and optimizing scalable, resilient data pipelines within AWS environments
• Strong understanding of distributed data processing frameworks, particularly PySpark and EMR
• Expert knowledge of database management principles, performance tuning, and data architecture best practices
• Advanced Python development skills with extensive hands-on experience using PySpark
• Expertise in Infrastructure as Code using Terraform
• Experience designing and implementing CI/CD frameworks using GitHub and GitHub Actions
• Deep knowledge of AWS IAM roles, policies, governance, and security best practices
• Strong experience with workflow orchestration tools such as AWS Step Functions, Apache Airflow, or equivalent platforms
• Experience leading cloud migration, modernization, and enterprise data platform initiatives
• Strong understanding of data governance, metadata management, data quality frameworks, and observability principles
• Ability to lead hands-on development while providing technical direction, code reviews, and engineering oversight across multiple initiatives
• Experience establishing development environments, infrastructure standards, security controls, and migration strategies across multiple AWS accounts and environments
• Ability to architect, develop, and govern enterprise-scale data pipelines, ETL processes, data ingestion frameworks, and orchestration workflows
• Proven ability to identify data gaps, define strategic remediation plans, and implement scalable automation solutions that improve analytical capabilities across the organization
• Ability to design and implement highly reliable data pipelines with a focus on data quality, observability, resiliency, and operational supportability
• Experience building and optimizing large-scale data warehousing solutions that prioritize business user experience and system performance
• Ability to drive technical decision-making, influence architectural direction, and communicate complex technical concepts to technical and non-technical stakeholders
• Proven track record mentoring engineers, fostering technical growth, and building high-performing data engineering teams
• Experience leading cross-functional initiatives involving data engineering, analytics, architecture, platform engineering, and business stakeholders
• Strong problem-solving and leadership skills, including managing competing priorities in fast-paced enterprise environments
• Experience building AI-ready data pipelines and ML workflows, including feature engineering and MLOps
• Proficiency in Python, SQL, Spark, and Generative AI technologies for scalable data and analytics solutions
• Hands‑on experience with AWS, Azure, or Databricks and cloud-based AI/data platforms
• Knowledge of LLMs, RAG, and vector databases to support AI-powered applications and intelligent search
• Strong understanding of data governance, quality, security, and responsible AI practices

Preferred Skills

• AWS Certified Data Engineer, AWS Solutions Architect, or equivalent cloud certifications
• Experience implementing enterprise data governance and metadata management platforms
• Experience with real-time analytics, event‑driven architectures, and streaming data platforms
• Knowledge of modern data architecture patterns including Data Mesh, Lakehouse, and domain‑oriented design
• Experience leading large‑scale cloud transformation or enterprise data modernization programs

Technology Focus

AWS, including S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, and Step Functions; AWS Bedrock; Lake Formation; Amazon Redshift; Kafka and Confluent Kafka; PySpark, Python, SQL; dimensional modeling and data vault; Terraform; GitHub and GitHub Actions; AWS IAM; Apache Airflow; ETL and CI/CD; Generative AI; Azure and Databricks; LLMs, RAG, vector databases, and MLOps; Spark.

Location and Role Summary

Charlotte, NC (Hybrid). Lead Data Engineer role designed for candidates with deep AWS data engineering experience and the ability to lead architecture, delivery, and mentorship across enterprise initiatives.