EY - GDS Consulting - AI And DATA - Data Engineer-Senior

EY; Pune District; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

EY is seeking a Senior Data Engineer in India to design and optimize scalable data solutions that drive business insights. You will build data pipelines, map data, and deploy Data & AI solutions using Python, PySpark, and SQL in an Agile setting.

You’ll work with AWS services, Databricks, and DBT, delivering production-grade code and adhering to governance and quality standards across large data warehouses.

Qualifications

• Extensive experience in building data ingestion frameworks and scalable data pipelines.
• Proven expertise in AWS cloud services including S3, EC2, Glue, Lambda, and Secrets Manager for secure and efficient data processing.
• Hands-on experience with Databricks for unified analytics and collaborative data engineering.
• Proficient in DBT for data transformation and modeling within modern data stacks.
• Skilled in orchestrating workflows using Apache Airflow to automate complex data pipelines.
• Strong programming skills in Python, PySpark, Spark, and Scala for big data processing.
• Experience with version control and CI/CD tools such as Git, Jenkins, and Subversion.
• Familiarity with Big Data ecosystems including EMR, Hadoop, Databricks, Hive, and PySpark.
• Minimum 3 years of experience in very large data warehousing environments.
• Excellent analytical skills with deep knowledge of SQL and data querying.

Responsibilities

• Design, develop, and manage robust CI/CD pipelines to streamline the product development lifecycle and ensure seamless deployment of data solutions.
• Perform data mapping and integration between diverse source and target systems to support analytics and business intelligence needs.
• Build and optimize Data & AI solutions, including data engineering pipelines, data modeling, and performance tuning, with 5-8 years of relevant experience.
• Develop high-quality production-level code primarily in Python, PySpark, and SQL, ensuring maintainability and scalability.
• Implement enterprise-scale production deployments, applying DevOps best practices, with hands-on experience in Git version control.
• Collaborate effectively within Agile teams, contributing to sprint planning, reviews, and continuous improvement.
• Apply strong knowledge of Data Governance and Data Quality standards to maintain data integrity and compliance.

Skills

Python

PySpark

SQL

Spark

Scala

Databricks

Airflow

Git

Jenkins

Subversion

ETL/ELT

Hadoop

Hive

EMR

Java

Tools

AWS S3

AWS EC2

AWS Glue

AWS Lambda

Secrets Manager

Databricks

DBT

Apache Airflow

Git

Jenkins

Subversion

Job description

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

Senior Data Engineer – Merck Role

The Opportunity:

Join our dynamic team as a Senior Data Engineer, where you will play a critical role in designing, building, and optimizing scalable data solutions that drive business insights and innovation. You will work with cutting-edge technologies in a collaborative Agile environment, contributing to the full data lifecycle from ingestion to transformation and deployment.

• Design, develop, and manage robust CI/CD pipelines to streamline the product development lifecycle and ensure seamless deployment of data solutions.
• Perform data mapping, and integration between diverse source and target systems to support analytics and business intelligence needs.
• Build and optimize Data & AI solutions, including data engineering pipelines, data modeling, and performance tuning, with 5-8 years of relevant experience.
• Develop high-quality, production-level code primarily in Python, PySpark, and SQL, ensuring maintainability and scalability.
• Implement enterprise-scale production deployments, applying DevOps best practices, with hands-on experience in Git version control.
• Collaborate effectively within Agile teams, contributing to sprint planning, reviews, and continuous improvement.
• Apply strong knowledge of Data Governance and Data Quality standards to maintain data integrity and compliance.

Technical Skills & Experience:

• Extensive experience in building data ingestion frameworks and scalable data pipelines.
• Proven expertise in AWS cloud services including S3, EC2, Glue, Lambda, and Secrets Manager for secure and efficient data processing.
• Hands-on experience with Databricks for unified analytics and collaborative data engineering.
• Proficient in DBT (Data Build Tool) for data transformation and modeling within modern data stacks.
• Skilled in orchestrating workflows using Apache Airflow to automate complex data pipelines.
• Strong programming skills in Python, PySpark, Spark, and Scala for big data processing.
• Experience with software version control and CI/CD tools such as Git, Jenkins, and Apache Subversion.
• AWS / Databricks certifications or equivalent professional technical certifications are highly desirable.
• Familiarity with cloud and enterprise integration technologies.
• Demonstrated ability to write efficient Spark jobs and optimize performance in large-scale environments.
• Excellent analytical skills with deep knowledge of SQL and data querying.
• Minimum 3 years of experience working in very large data warehousing environments.
• Strong communication skills, both written and verbal, to effectively collaborate with cross-functional teams.
• At least 3 years of experience with data warehouse architectures, ETL/ELT processes, and reporting/analytics tools.
• Experience in Python and/or Java development in data engineering contexts.
• Familiarity with Big Data ecosystems including EMR, Hadoop, Databricks, Hive, and Pyspark.

EY | Building a better working world

EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets. Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate. Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.