Data Engineering Manager

PowerToFly; North Carolina; On-site; USD 150,000 - 190,000; Full time

Job summary

Deloitte is seeking a Data Engineering Manager to lead architecture for a PySpark-based data processing framework and migrate major projects to an AWS data lake. You will design a metadata-driven processing stack using PySpark, Python, Databricks and AWS, while building data lineage and observability capabilities.

The role involves ingestion framework development with Talend, orchestration migration to Airflow, and optimizing data quality rules across the SDLC with client collaboration and

Qualifications

• 8+ years of AWS cloud services and data lake build & support.
• Strong experience in PySpark-based data engineering and ETL development.
• SQL (Snowflake) pipeline build, data analysis, and data validation.
• Requirements gathering and stakeholder coordination.
• Bachelor's degree, preferably in Computer Science, Information Technology, CS, or related IT discipline; or equivalent experience.
• Limited immigration sponsorship may be available.
• Ability to travel 10%, on average, based on the work you do and the clients/industries served.

Responsibilities

• Lead solution architecture for a PySpark-based enterprise data processing framework across programs.
• Migrate major projects from on-premises to AWS-based cloud data lake ecosystems.
• Design and implement a metadata-driven processing framework using PySpark, Python, Databricks, and AWS.
• Build data lineage capabilities and run data observability PoCs to accelerate delivery.
• Develop ingestion frameworks in Talend and migrate orchestration to Apache Airflow.
• Performance-tune complex data quality rules to improve processing efficiency and reduce costs.
• Engage in full SDLC activities including client discussions, requirements, architecture design, and deployment.

Skills

Independent work

Communication skills

Attention to detail

Relationship building

Project leadership

Task prioritization

Interpersonal skills

Deadline orientation

Guidance to others

Education

Bachelor's degree in CS/IT/related

Tools

PySpark

Talend

Apache Airflow

Databricks

AWS

Snowflake

Job description

Data Engineering Manager

Deloitte Consulting's technology professionals help clients identify and solve their most critical information and technological challenges. We provide advisory through end-to-end implementation services as well as outsourcing services and are recognized in the marketplace for capabilities across the spectrum.

Recruiting for this role ends on 10/1/2026

Our experienced technology professionals offer deep technical experience in their area of focus and are committed to delivering efficient, technology-based solutions to our clients. Our professionals are also aligned to industry sectors. By combining our technical capabilities with our industry experience, we create unmatched market offerings to solve our clients' business issues. Additionally, we have long-term partnerships with many of the world's leading technological companies, allowing us to understand solution alternatives and recommend and support the most appropriate solution for our clients. By leveraging these elements, we can help our clients convert leading edge ideas into tangible results.

Work you'll do

A Data Engineering Manager will lead solution architecture for a PySpark-based enterprise data processing framework, improving scalability and standardizing transformation patterns across programs. Migrate major projects from on-premises platforms to AWS-based cloud data lake ecosystems, improving platform flexibility and modernization readiness. Design and implement a metadata-driven processing framework using PySpark, Python, Databricks, and AWS, reducing project go-live timelines. Build data lineage capabilities and conducted POCs in data observability, improving change impact analysis and accelerating enhancement delivery. Develop ingestion frameworks in Talend and later migrated orchestration to Apache Airflow, improving data onboarding efficiency. Performance tuning complex data quality rules, improving processing efficiency and reducing run-cost and maintenance overhead. Work across the full SDLC, including client discussions, requirements gathering, architecture design, development oversight, and production deployment. The successful candidate would possess these skills:

• Ability to work independently and collaborate as part of a team
• Effective written and verbal communication skills
• Meticulous attention to detail and quality of work product
• Ability to build and sustain professional relationships
• Ability to lead projects or workstreams
• Ability to manage and prioritize multiple tasks in a fast-paced and dynamic environment
• Strong interpersonal skills and professional demeanor
• Ability to meet deadlines
• Ability to provide clear guidance to others

The Team

AI & Data offers a full spectrum of solutions for designing, developing, and operating cutting-edge Data and AI platforms, products, insights, and services. Our offering helps clients innovate, and enhance their data, AI, and analytics capabilities, ensuring they can mature and scale effectively.

Required Qualifications

• 8+ years of AWS cloud services and data lake build & support
• Strong experience in PySpark-based data engineering and ETL development
• SQL (Snowflake) pipeline build, data analysis, and data validation
• Requirements gathering and stakeholder coordination
• Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
• Limited immigration sponsorship may be available.
• Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve

Preferred

• Strong analytical, problem-solving, communication, stakeholder-management, offshore coordination, documentation, and knowledge-transfer skills
• Analytical ability to manage multiple projects and prioritize tasks into manageable work product
• Can operate independently or with minimum supervision
• Excellent Written and Communication Skills
• Ability to deliver technical demonstrations

The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Deloitte, it is not typical for an individual to be hired at or near the top of the range for their role and compensation decisions are dependent on the facts and circumstances of each case.

You may also be eligible to participate in a discretionary annual incentive program, subject to the rules governing the program, whereby an award, if any, depends on various factors, including, without limitation, individual and organizational performance.