Principal Data Engineer

Amgen; India; On-site; INR 4,200,000 - 6,400,000; Full time

Job summary

Amgen in India seeks a highly motivated Principal Data Engineer to own the design and development of complex data pipelines, data integration frameworks and metadata-driven architectures enabling enterprise-wide data access and analytics.

You will lead Databricks, Spark and Delta Lake based pipelines, drive performance tuning, governance, and observability while mentoring engineers and collaborating with analytics, governance, and DevOps teams to align with strategic data goals.

Qualifications

• Hands-on data engineering with Databricks, PySpark, SparkSQL, AWS, Python and SQL.
• Strong skills in workflow orchestration and performance tuning for big data.
• Solid knowledge of AWS services and cloud data platforms.
• Experience with Data Fabric or Data Mesh architectures.
• Ability to learn and apply new technologies quickly.
• Strong problem-solving, communication and collaboration skills.
• Experience with SAFe, Agile delivery and DevOps practices.

Responsibilities

• Architect and maintain robust, scalable data pipelines using Databricks, Spark and Delta Lake.
• Lead adoption of new technologies to improve productivity and data delivery.
• Optimize Spark performance, resource usage, job scheduling and queries.
• Implement data ingestion, transformation, lineage, and observability frameworks.
• Build metadata-driven data engineering frameworks for reuse and consistency.
• Foster technical excellence and continuous improvement within the team.
• Collaborate with platform, architecture, analytics and governance teams.
• Define and uphold SLOs, monitoring standards and data quality KPIs.
• Translate business needs into scalable data products with cross-functional teams.
• Mentor engineers and promote modern engineering practices.
• Align data engineering strategy with enterprise goals with DevOps collaboration.
• Stay current with emerging data technologies and best practices.

Skills

Databricks

PySpark

SparkSQL

Apache Spark

AWS

Python

SQL

Scaled Agile (SAFe)

Agile delivery

DevOps practices

Education

AWS Certified Data Engineer

Databricks Certificate

Scaled Agile SAFe certification

Tools

Git

Subversion

CI/CD (Jenkins, Maven)

Job description

ABOUT THE ROLE

Role Description: Let's do this. Let's change the world. We are looking for highly motivated expert Principal Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management.

Roles & Responsibilities:

• Architect and maintain robust, scalable data pipelines using Databricks, Spark, and Delta Lake, enabling efficient batch and real-time processing.
• Lead efforts to evaluate, adopt, and integrate emerging technologies and tools that enhance productivity, scalability, and data delivery capabilities.
• Drive performance optimization efforts, including Spark tuning, resource utilization, job scheduling, and query improvements.
• Identify and implement innovative solutions that streamline data ingestion, transformation, lineage tracking, and platform observability.
• Build frameworks for metadata-driven data engineering, enabling reusability and consistency across pipelines.
• Foster a culture of technical excellence, experimentation, and continuous improvement within the data engineering team.
• Collaborate with platform, architecture, analytics, and governance teams to align platform enhancements with enterprise data strategy.
• Define and uphold SLOs, monitoring standards, and data quality KPIs for production pipelines and infrastructure.
• Partner with cross-functional teams to translate business needs into scalable, governed data products.
• Mentor engineers across the team, promoting knowledge sharing and adoption of modern engineering patterns and tools.
• Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
• Stay up to date with emerging data technologies and best practices, ensuring continuous improvement of Enterprise Data Fabric architectures.

Must-Have Skills:

• Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies.
• Proficiency in workflow orchestration, performance tuning on big data processing.
• Strong understanding of AWS services
• Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
• Ability to quickly learn, adapt and apply new technologies
• Strong problem-solving and analytical skills
• Excellent communication and teamwork skills
• Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.

Good-to-Have Skills:

• Good to have deep expertise in Biotech & Pharma industries
• Experience in writing APIs to make the data available to the consumers
• Experienced with SQL/NOSQL database, vector database for large language models
• Experienced with data modeling and performance tuning for both OLAP and OLTP databases
• Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops

Education and Professional Certifications

• 12 to 17 years of experience in Computer Science, IT or related field
• AWS Certified Data Engineer preferred
• Databricks Certificate preferred
• Scaled Agile SAFe certification preferred

Soft Skills:

• Excellent analytical and troubleshooting skills.
• Strong verbal and written communication skills
• Ability to work effectively with global, virtual teams
• High degree of initiative and self-motivation.
• Ability to manage multiple priorities successfully.
• Team-oriented, with a focus on achieving team goals.
• Ability to learn quickly, be organized and detail oriented.
• Strong presentation and public speaking skills.

.