Principal Data Engineer

Biopharma Careers; Hyderabad; On-site; INR 4,000,000 - 9,000,000; Full time

Job summary

Biopharma Careers is seeking a Principal Data Engineer to own the design and development of complex data pipelines, data integration frameworks, and governance architectures for enterprise analytics.

The role emphasizes scalable, metadata-driven data platforms, real-time processing, and governance to enable AI-driven insights across the organization. You will mentor engineers and collaborate with cross-functional teams in a biotech context.

Qualifications

• 12 to 17 years of experience in Computer Science, IT or related field.
• Hands-on experience with Databricks, PySpark, SparkSQL, AWS, Python, SQL.
• Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
• Experience with SAFe, Agile delivery practices, and DevOps practices.
• Excellent analytical and troubleshooting skills.

Responsibilities

• Architect and maintain robust, scalable data pipelines using Databricks, Spark, and Delta Lake, enabling efficient batch and real-time processing.
• Lead efforts to evaluate, adopt, and integrate emerging technologies to enhance data delivery capabilities.
• Drive performance optimization, including Spark tuning, resource scheduling, and query improvements.
• Identify and implement solutions for data ingestion, transformation, lineage tracking, and platform observability.
• Build frameworks for metadata-driven data engineering to ensure reusability and consistency.
• Mentor engineers and promote best practices in modern data engineering.
• Collaborate with platform, architecture, analytics, and governance teams to align data strategy with enterprise goals.
• Define and uphold SLOs and data quality KPIs for production pipelines and infrastructure.
• Partner with cross-functional teams to translate business needs into scalable data products.

Skills

Databricks

PySpark

SparkSQL

AWS

Python

SQL

SAFe

DevOps

Data Governance

Scala

Education

Bachelors in CS/IT or related field

AWS Certified Data Engineer

Databricks Certificate

Scaled Agile SAFe certification

Tools

Airflow

Delta Lake

Kubernetes

Job description

Career Category

Engineering

Job Description

ABOUT THE ROLE

Role Description:

Let's do this. Let's change the world. We are looking for highly motivated expert Principal Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management.

Roles & Responsibilities:

• Architect and maintain robust, scalable data pipelines using Databricks, Spark, and Delta Lake, enabling efficient batch and real-time processing.
• Lead efforts to evaluate, adopt, and integrate emerging technologies and tools that enhance productivity, scalability, and data delivery capabilities.
• Drive performance optimization efforts, including Spark tuning, resource utilization, job scheduling, and query improvements.
• Identify and implement innovative solutions that streamline data ingestion, transformation, lineage tracking, and platform observability.
• Build frameworks for metadata-driven data engineering, enabling reusability and consistency across pipelines.
• Foster a culture of technical excellence, experimentation, and continuous improvement within the data engineering team.
• Collaborate with platform, architecture, analytics, and governance teams to align platform enhancements with enterprise data strategy.
• Define and uphold SLOs, monitoring standards, and data quality KPIs for production pipelines and infrastructure.
• Partner with cross-functional teams to translate business needs into scalable, governed data products.
• Mentor engineers across the team, promoting knowledge sharing and adoption of modern engineering patterns and tools.
• Collaborate with cross-functional teams, including data architects, business analysts, and DevOps teams, to align data engineering strategies with enterprise goals.
• Stay up to date with emerging data technologies and best practices, ensuring continuous improvement of Enterprise Data Fabric architectures.

Must-Have Skills:

• Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies.
• Proficiency in workflow orchestration, performance tuning on big data processing.
• Strong understanding of AWS services
• Experience with Data Fabric, Data Mesh, or similar enterprise-wide data architectures.
• Ability to quickly learn, adapt and apply new technologies
• Strong problem-solving and analytical skills
• Excellent communication and teamwork skills
• Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.

Good-to-Have Skills:

• Good to have deep expertise in Biotech & Pharma industries
• Experience in writing APIs to make the data available to the consumers
• Experienced with SQL/NOSQL database, vector database for large language models
• Experienced with data modeling and performance tuning for both OLAP and OLTP databases
• Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops

Education and Professional Certifications

• 12 to 17 years of experience in Computer Science, IT or related field
• AWS Certified Data Engineer preferred
• Databricks Certificate preferred
• Scaled Agile SAFe certification preferred

Soft Skills:

• Excellent analytical and troubleshooting skills.
• Strong verbal and written communication skills
• Ability to work effectively with global, virtual teams
• High degree of initiative and self-motivation.
• Ability to manage multiple priorities successfully.
• Team-oriented, with a focus on achieving team goals.
• Ability to learn quickly, be organized and detail oriented.
• Strong presentation and public speaking skills.