Senior Data Engineer

AppSierra; India; On-site; INR 3,500,000 - 5,200,000; Full time

Job summary

AppSierra is seeking a senior data engineer to design, build, and maintain scalable data pipelines across AWS regions. You will assemble large datasets, automate workflows, and re-architect infrastructure for performance and reliability.

You will lead a small team, collaborate with cross-functional partners, and implement data governance, security, and high-availability practices. Proficiency in Python, Spark, Kafka, and Databricks is expected.

Qualifications

• 6+ years in a Data Engineering role with scalable data systems.
• Strong experience with Spark and Kafka for streaming/ batch data.
• Hands-on with AWS EC2 and RDS and cloud data infra.
• Proficient in building ETL pipelines using Airflow and Databricks.
• Experience with SQLMesh for data workflow management.
• Experience with REST APIs, GraphQL, and SFTP integrations.
• Excellent communication and leadership abilities.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Aggregate large datasets meeting functional and non-functional requirements.
• Automate workflows and improve data infra for scalability and reliability.
• Develop ETL infra enabling high-quality data across sources.
• Create analytics tools to derive insights on key metrics.
• Collaborate with Exec, Product, Data, and Design teams on data needs.
• Ensure data integrity, security across data centers and AWS regions.
• Build data tools to empower analytics and data science teams.
• Lead and mentor a small data engineering team.
• Provide code reviews, standards, and career development guidance.

Skills

Big data

Spark

Kafka

Python

Airflow

SQL

Data modeling

Leadership

Cloud AWS

REST APIs

GraphQL

Databricks

SQLMesh

Tools

Airflow

Databricks

Kafka

Spark

SQLMesh

PostgreSQL

MongoDB

Redis

REST APIs

GraphQL

SFTP

NestJS/TypeScript

Job description

Key Responsibilities

• Design, build, and maintain robust and scalable data pipeline architectures.


• Assemble large, complex datasets that meet both functional and non-functional business requirements.


• Identify, design, and implement internal process improvements — including automation of manual workflows, optimization of data delivery, and re-architecting infrastructure for greater scalability and reliability.


• Design, build, and optimize ETL infrastructure to enable scalable, high-quality data workflows across diverse sources, leveraging SQL and modern data processing frameworks.


• Build analytics tools that utilize the data pipeline to deliver actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.


• Collaborate with stakeholders across Executive, Product, Data, and Design teams to resolve data-related technical issues and ensure their data infrastructure needs are met.


• Ensure data integrity, separation, and security across multiple data centers and AWS regions.


• Create data tools and frameworks to empower analytics and data science teams in building and optimizing products that drive innovation and establish market leadership.


• Lead and mentor a small team of data engineers, fostering a culture of technical excellence, collaboration, and continuous improvement.


• Provide technical guidance, set coding standards, conduct code reviews, and support career development for team members.


• Work closely with data and analytics experts to continually enhance the functionality, reliability, and scalability of our data systems.

Data Engineering And Infrastructure

• 6+ years of experience in a Data Engineering role, designing, building, and managing scalable and reliable data systems.


• Proficient with big data and stream-processing technologies such as Spark and Kafka.


• Hands-on experience with cloud platforms, particularly AWS services like EC2 and RDS.


• Skilled in building and orchestrating data pipelines using tools like Airflow.


• Experience with Databricks for scalable data processing and advanced analytics.


• Strong knowledge of SQLMesh for modern data workflow management.


• Extensive experience integrating and working with external data sources via REST APIs, GraphQL endpoints, and SFTP servers.


• Strong communication skills and leadership capabilities are required.

Databases And Data Management

• Expertise with relational and NoSQL databases, including Postgres and MongoDB.


• Solid understanding of data modeling, data governance, and data security best practices.

Programming And Development

• Proficient in Python for data engineering, automation, and workflow scripting.


• Familiarity with software engineering best practices, including version control, testing, and CI/CD pipelines for data workflows.


• Experience with JavaScript and TypeScript is a plus.

Analytics, Visualization, And BI

• Skilled in implementing and supporting self-service BI tools to enable business teams with accessible, actionable insights.


• Experience with Streamlit for building interactive data visualizations is a plus.

Blockchain And Financial Data Expertise

• Knowledge of blockchain technology and the cryptocurrency ecosystem is a nice-to- have, with a strong interest in staying up to date with emerging trends.


• Experience working with financial datasets and financial engineering concepts is considered a strong advantage.

Our Stack

We work with a modern and evolving technology stack, including but not limited to:

• Cloud Infrastructure: AWS for cloud services and infrastructure management


• Databases: PostgreSQL for relational data, MongoDB for non-relational (NoSQL) data, and Redis for caching and real-time data management


• Backend: NestJS (Node.js, TypeScript) and Python for building scalable backend services


• Frontend: React for web applications, Streamlit for building interactive data visualizations


• Data Engineering: Airflow and SQLMesh for data pipeline orchestration and modern workflow management


• Big Data & Processing: Databricks and Kafka for scalable data processing, analytics, and streaming


• Integrations & APIs: Extensive use of REST APIs, GraphQL, SFTP, and Slack integrations to enable seamless data exchange and operational workflows

Messaging&EventStreaming:

Kafkaforreal-timedatapipelinesandevent-driven architectures