Data Engineering Manager: AWS-PySpark Platform Lead

PowerToFly; North Carolina; On-site; USD 150,000 - 190,000; Full time

Job summary

Deloitte is seeking a Data Engineering Manager to lead architecture for a PySpark-based data processing framework and migrate major projects to an AWS data lake. You will design a metadata-driven processing stack using PySpark, Python, Databricks and AWS, while building data lineage and observability capabilities.

The role involves ingestion framework development with Talend, orchestration migration to Airflow, and optimizing data quality rules across the SDLC with client collaboration and

Qualifications

• 8+ years of AWS cloud services and data lake build & support.
• Strong experience in PySpark-based data engineering and ETL development.
• SQL (Snowflake) pipeline build, data analysis, and data validation.
• Requirements gathering and stakeholder coordination.
• Bachelor's degree, preferably in Computer Science, Information Technology, CS, or related IT discipline; or equivalent experience.
• Limited immigration sponsorship may be available.
• Ability to travel 10%, on average, based on the work you do and the clients/industries served.

Responsibilities

• Lead solution architecture for a PySpark-based enterprise data processing framework across programs.
• Migrate major projects from on-premises to AWS-based cloud data lake ecosystems.
• Design and implement a metadata-driven processing framework using PySpark, Python, Databricks, and AWS.
• Build data lineage capabilities and run data observability PoCs to accelerate delivery.
• Develop ingestion frameworks in Talend and migrate orchestration to Apache Airflow.
• Performance-tune complex data quality rules to improve processing efficiency and reduce costs.
• Engage in full SDLC activities including client discussions, requirements, architecture design, and deployment.

Skills

Independent work

Communication skills

Attention to detail

Relationship building

Project leadership

Task prioritization

Interpersonal skills

Deadline orientation

Guidance to others

Education

Bachelor's degree in CS/IT/related

Tools

PySpark

Talend

Apache Airflow

Databricks

AWS

Snowflake

Job description

Deloitte is seeking a Data Engineering Manager to lead architecture for a PySpark-based data processing framework and migrate major projects to an AWS data lake. You will design a metadata-driven processing stack using PySpark, Python, Databricks and AWS, while building data lineage and observability capabilities.

The role involves ingestion framework development with Talend, orchestration migration to Airflow, and optimizing data quality rules across the SDLC with client collaboration and