Data Engineering Manager: AWS/PySpark Lead

Relha LLC; Charlotte, Northern (NC, KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Deloitte Consulting LLP in Charlotte, NC seeks a Data Engineering Manager to lead the PySpark-based data processing framework and drive cloud modernization. You will guide architecture, oversee development, and collaborate with clients to transform on-premise projects to AWS data lake ecosystems.

Responsibilities include building data lineage, designing metadata-driven processing, and implementing ingestion and orchestration with Talend, Airflow, and Databricks.

Qualifications

• 8+ years of AWS cloud services and data lake build & support.
• Strong experience in PySpark-based data engineering and ETL development.
• SQL (Snowflake) pipeline build, data analysis, and data validation.
• Requirements gathering and stakeholder coordination.
• Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience.

Responsibilities

• Lead solution architecture for a PySpark-based data processing framework.
• Migrate major projects from on-premises platforms to AWS-based data lake ecosystems.
• Design and implement a metadata-driven processing framework using PySpark, Python, Databricks, and AWS.
• Build data lineage capabilities and data observability PoCs to speed delivery.
• Develop ingestion frameworks and orchestration with Airflow to improve onboarding.
• Perform performance tuning of data quality rules to reduce costs.

Skills

AWS cloud

PySpark

Snowflake SQL

Stakeholder coordination

Education

Bachelor's degree in CS/IT/Eng

Tools

Talend

Apache Airflow

Databricks

Python

Job description

Deloitte Consulting LLP in Charlotte, NC seeks a Data Engineering Manager to lead the PySpark-based data processing framework and drive cloud modernization. You will guide architecture, oversee development, and collaborate with clients to transform on-premise projects to AWS data lake ecosystems.

Responsibilities include building data lineage, designing metadata-driven processing, and implementing ingestion and orchestration with Talend, Airflow, and Databricks.