Senior Data Engineer, Python

Jobtailor; Bengaluru; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Jobtailor in Bengaluru, India seeks a senior Data Engineer to design and maintain scalable data pipelines using Palantir Foundry and AWS. You will collaborate with data scientists and analysts to translate data requirements into robust solutions, build reusable data assets, and ensure data quality and governance.

The role requires 5-12 years of experience, strong Python/SQL skills, and experience with data modeling and ETL in cloud data warehouses.

Qualifications

• 5-12 years of experience in data engineering, analytics, or software engineering roles.
• Strong hands-on experience with Pyspark, Palantir Foundry, AWS, including ontology modelling, pipelines, data fusion, and transformation logic.
• Proficient in Python, SQL, Snowflake and data processing frameworks.
• Solid understanding of data integration, ETL workflows, and data modelling principles.
• Experience working with relational databases, data lakes, cloud data warehouses (AWS, Azure, GCP, Snowflake, Web-services).
• Strong problem-solving skills and ability to debug complex data workflows.
• Familiarity with CI/CD pipelines and version control tools like Git.
• Excellent communication skills and ability to work collaboratively with cross-functional teams.

Responsibilities

• Design, implement, and maintain scalable and reliable data pipelines using Palantir Foundry, AWS.
• Solid experience in web front-end development, web-services.
• Some exposure to business analytics, financial forecasting, or a related domain would be preferable.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Build reusable data assets, integrate data from multiple sources, and ensure data quality, integrity, and security.
• Develop and maintain transformation workflows, ontology models, and data governance frameworks within Foundry.
• Optimize data pipelines for performance, scalability, and cost efficiency.
• Troubleshoot, monitor, and resolve data issues and pipeline failures in real-time.
• Implement best practices in data versioning, lineage, and observability.
• Assist in defining data architecture strategies and tooling roadmaps.
• Mentor junior engineers and contribute to team knowledge sharing and documentation.

Skills

Pyspark

Palantir Foundry

AWS

ontology modelling

pipelines

data fusion

transformation logic

Python

SQL

Snowflake

data processing frameworks

Data integration

ETL

Data modelling

Tools

Git

CI/CD

Job description

Responsibilities

• Design, implement, and maintain scalable and reliable data pipelines using Palantir Foundry, AWS
• Solid experience in web front-end development, web-services.
• Some exposure to business analytics, financial forecasting, or a related domain would be preferable.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Build reusable data assets, integrate data from multiple sources, and ensure data quality, integrity, and security.
• Develop and maintain transformation workflows, ontology models, and data governance frameworks within Foundry.
• Optimize data pipelines for performance, scalability, and cost efficiency.
• Troubleshoot, monitor, and resolve data issues and pipeline failures in real-time.
• Implement best practices in data versioning, lineage, and observability.
• Assist in defining data architecture strategies and tooling roadmaps.
• Mentor junior engineers and contribute to team knowledge sharing and documentation.

Requirements

• 5-12 years of experience in data engineering, analytics, or software engineering roles
• Strong hands-on experience with Pyspark, Palantir Foundry, AWS, including ontology modelling, pipelines, data fusion, and transformation logic
• Proficient in Python, SQL, Snowflake and data processing frameworks
• Solid understanding of data integration, ETL workflows, and data modelling principles
• Experience working with relational databases, data lakes, cloud data warehouses (AWS, Azure, GCP, Snowflake, Web-services)
• Strong problem-solving skills and ability to debug complex data workflows
• Familiarity with CI/CD pipelines and version control tools like Git
• Excellent communication skills and ability to work collaboratively with cross-functional teams.

Core Competencies

Demonstrates expertise in designing and maintaining scalable data pipelines using Palantir Foundry and AWS, with strong proficiency in Pyspark, Python, SQL, and data integration principles. Capable of collaborating with cross-functional teams to translate data requirements into effective technical solutions while ensuring data quality and governance.