IN_Senior Associate_Java Full Stack_Data and Analytics_Advisory_Bangalore

PwC South Africa; Bengaluru; On-site; Confidential; Full time

Job summary

PwC South Africa is seeking a Senior Associate to design, develop, and maintain scalable data pipelines using AWS services and Snowflake. You will implement ETL/ELT processes, craft PySpark jobs, and optimize data models in a dynamic analytics environment.

Responsibilities include building batch and near-real-time pipelines, ensuring data quality, and collaborating in Agile teams. Strong Python, SQL, and Spark skills are essential for delivering data-driven insights to clients.

Qualifications

• 4–8 years of experience in Data Engineering.
• Strong hands-on experience with AWS Data Engineering.
• Strong experience with Snowflake.
• Hands-on experience with Apache Airflow and DAG development.
• Strong programming experience in Python.
• Strong hands-on experience with PySpark / Apache Spark.
• Advanced SQL skills.
• Strong understanding of ETL/ELT and data pipeline development.
• Experience with AWS S3 and AWS Glue.
• Good understanding of data warehousing and dimensional data modeling.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using AWS services.
• Build and orchestrate data pipelines using Apache Airflow, including DAG development, scheduling, monitoring, retries, dependencies, and error handling.
• Develop data processing and transformation solutions using Python and PySpark.
• Design and implement data warehouse solutions using Snowflake.
• Develop complex SQL queries, stored procedures, views, CTEs, and data transformations.
• Work with AWS services such as S3, Glue, Lambda, EMR, Athena, Redshift, and IAM.
• Build batch and, where required, near-real-time data ingestion pipelines.
• Implement data ingestion from APIs, databases, files, and other source systems into AWS/Snowflake.
• Perform Snowflake performance and cost optimization, including warehouse sizing, query optimization, clustering, partitioning, and efficient data loading.
• Implement Snowflake features such as Snowpipe, Streams, Tasks, stages, file formats, and secure data sharing.

Skills

AWS

Snowflake

Python

PySpark

Apache Airflow

SQL

ETL/ELT

Git/CI/CD

Education

Bachelor's or Master's in CS/IT/Engineering

Tools

AWS

Snowflake

Airflow

Python

Job description

Line of Service Advisory Industry/Sector Not Applicable Specialism Data, Analytics & AI Management Level Senior Associate Job Description & Summary

At PwC, our people in data and analytics focus on leveraging data to drive insights and make informed business decisions. They utilise advanced analytics techniques to help clients optimise their operations and achieve their strategic goals. In data analysis at PwC, you will focus on utilising advanced analytical techniques to extract insights from large datasets and drive data-driven decision-making. You will leverage skills in data manipulation, visualisation, and statistical modelling to support clients in solving complex business problems.

Why PWC

At PwC, you will be part of a vibrant community of solvers that leads with trust and creates distinctive outcomes for our clients and communities. This purpose-led and values-driven work, powered by technology in an environment that drives innovation, will enable you to make a tangible impact in the real world. We reward your contributions, support your wellbeing, and offer inclusive benefits, flexibility programmes and mentorship that will help you thrive in work and life. Together, we grow, learn, care, collaborate, and create a future of infinite experiences for each other.

Learn more about us

At PwC, we believe in providing equal employment opportunities, without any discrimination on the grounds of gender, ethnic background, age, disability, marital status, sexual orientation, pregnancy, gender identity or expression, religion or other beliefs, perceived differences and status protected by law. We strive to create an environment where each one of our people can bring their true selves and contribute to their personal growth and the firm’s growth. To enable this, we have zero tolerance for any discrimination and harassment based on the above considerations.

Job Description & Summary

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using AWS services.
• Build and orchestrate data pipelines using Apache Airflow, including DAG development, scheduling, monitoring, retries, dependencies, and error handling.
• Develop data processing and transformation solutions using Python and PySpark.
• Design and implement data warehouse solutions using Snowflake.
• Develop complex SQL queries, stored procedures, views, CTEs, and data transformations.
• Work with AWS services such as S3, Glue, Lambda, EMR, Athena, Redshift, and IAM.
• Build batch and, where required, near-real-time data ingestion pipelines.
• Implement data ingestion from APIs, databases, files, and other source systems into AWS/Snowflake.
• Perform Snowflake performance and cost optimization, including warehouse sizing, query optimization, clustering, partitioning, and efficient data loading.
• Implement Snowflake features such as Snowpipe, Streams, Tasks, stages, file formats, and secure data sharing.
• Develop scalable Spark/PySpark jobs and optimize transformations, joins, partitioning, caching, and resource utilization.
• Implement data quality checks, validation, reconciliation, and monitoring mechanisms.
• Troubleshoot pipeline failures, data issues, performance bottlenecks, and production incidents.
• Follow best practices for data security, governance, access control, and PII-sensitive data handling.
• Use Git and CI/CD practices for source control, automated testing, and deployment of data pipelines.
• Collaborate with cross-functional teams in an Agile/Scrum environment.
• Create technical documentation for data pipelines, workflows, data models, and operational procedures.

Mandatory Skill Sets

• 4–8 years of experience in Data Engineering.
• Strong hands-on experience with AWS Data Engineering.
• Strong experience with Snowflake.
• Hands-on experience with Apache Airflow and DAG development.
• Strong programming experience in Python.
• Strong hands-on experience with PySpark / Apache Spark.
• Advanced SQL skills.
• Strong understanding of ETL/ELT and data pipeline development.
• Experience working with AWS S3 and AWS Glue.
• Good understanding of data warehousing and dimensional data modeling.
• Experience with data pipeline monitoring, debugging, and performance optimization.
• Good understanding of Git and CI/CD.

Preferred Skill Sets

• AWS Lambda, EMR, Athena, Redshift, Kinesis, Step Functions, IAM.
• Snowflake Snowpipe, Streams, Tasks, Dynamic Tables, Time Travel and performance tuning.
• Experience with dbt.
• Experience with Kafka or other streaming technologies.
• Experience with Terraform / Infrastructure as Code.
• Experience with data quality tools such as Great Expectations.
• Knowledge of Lakehouse / Medallion Architecture.
• Experience with Databricks.
• Snowflake certification such as SnowPro Core.
• Exposure to Docker/Kubernetes is a plus.

Technical Skills Category Required Skills

• Cloud AWS AWS Services S3, Glue, Lambda, EMR, Athena, Redshift, IAM
• Data Warehouse Snowflake
• Programming Python
• Big Data PySpark, Apache Spark
• Orchestration Apache Airflow
• Database SQL, Relational Databases
• Data Engineering ETL/ELT, Data Pipelines, Data Integration
• Data Modeling Star Schema, Snowflake Schema, Dimensional Modeling
• DevOps Git, CI/CD
• Optional Kafka, dbt, Terraform, Databricks

Years of Experience Required: 4–8 years of experience in Data Engineering.

Education Qualification Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Additional Information

Degrees/Field of Study required: Bachelor of Technology
Degrees/Field of Study preferred: Certifications (if blank, certifications not specified)
Required Skills Java (Programming Language)
Optional Skills Accepting Feedback, Accepting Feedback, Active Listening, Algorithm Development, Alteryx (Automation Platform), Analytical Thinking, Analytic Research, Big Data, Business Data Analytics, Communication, Complex Data Analysis, Conducting Research, Creativity, Customer Analysis, Customer Needs Analysis, Dashboard Creation, Data Analysis, Data Analysis Software, Data Collection, Data-Driven Insights, Data Integration, Data Integrity, Data Mining, Data Modeling, Data Pipeline {+ 38 more} Desired Languages (If blank, desired languages not specified)