SDE 2 Pyspark Data Engineer

Astar Data; Hyderabad, Bengaluru; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

Astar Data is seeking a Software Development Engineer II (SDE II) for Bengaluru, Karnataka, India, to join our engineering teams focusing on scalable distributed systems and big data platforms. The role emphasizes end-to-end ownership across the lifecycle, real-time data pipelines, and enterprise-grade development practices.

Ideal candidates have 6+ years of hands-on software development experience with Java/Scala/Python/C++, strong knowledge of data structures, and exposure to cloud platforms

Qualifications

• 6+ years of strong hands-on software development experience.
• Proficiency in Java, Scala, Python, or C++, with strong coding and debugging skills.
• Experience in building enterprise-scale applications.
• Strong understanding of SDLC, Agile methodologies, and DevOps practices.

Responsibilities

• Own the end-to-end lifecycle including requirements, design, development, testing, deployment, and production support.
• Design and build scalable, high-performance applications and distributed data pipelines.
• Develop reusable frameworks, APIs, and integration patterns for enterprise and distributed computing.
• Implement robust data ingestion, transformation, and processing solutions in parallel environments.
• Ensure code quality, version control, CI/CD, automated testing, and secure deployment.

Skills

Java/Scala/Python/C++

Job description

Job Description Software Development Engineer II (SDE II)-Pyspark

Location: Bengaluru, Karnataka, India
Experience: 4 - 5 Years
Function: Engineering / Technical / Analytics

About the Role

We are looking for a highly skilled and motivated Software Development Engineer III to join our high-impact engineering teams supporting strategic data and analytics initiatives for leading global financial institutions. The role demands strong ownership across the entire software development lifecycle, with a focus on scalable distributed systems, Big Data engineering, and enterprise-grade development practices.

This position offers an opportunity to work on large-scale data platforms, real-time data pipelines, and next-generation AI-driven solutions, while collaborating with global stakeholders and driving innovation.

Key Responsibilities

Engineering & Delivery

• Own the end-to-end lifecycle including requirements, design, development, testing, deployment, and production support.
• Design and build scalable, high-performance applications and distributed data pipelines.
• Develop reusable frameworks, APIs, and integration patterns to solve complex enterprise and distributed computing problems.
• Implement robust data ingestion, transformation, and processing solutions in parallel processing environments.
• Ensure strong software engineering practices such as code quality, version control, CI/CD, automated testing, and secure deployment.

Big Data & Distributed Systems

• Provide technical expertise in Big Data and distributed computing technologies, including:
• Apache Spark, Kafka, Flink
• Hadoop ecosystem (MapReduce, HDFS, Hive, HBase, Flume, Sqoop)
• NoSQL databases such as Cassandra and HBase
• Build and optimize real-time and batch data pipelines.
• Explore and evaluate new technologies to improve scalability, performance, and cost efficiency.

Technical Leadership & Innovation

• Drive adoption and contribution to open-source technologies.
• Provide technical mentorship and leadership to junior engineers.
• Promote best practices in system design, coding, and performance optimization.
• Align engineering solutions with strategic client initiatives including AI, automation, and data governance.

Client Collaboration

• Interface daily with global stakeholders to understand business and technical requirements.
• Translate strategic business needs into scalable engineering solutions.
• Ensure delivery excellence and measurable business impact.

AI-First & Continuous Learning

• Stay updated with emerging technologies such as Generative AI, data platforms, and cloud ecosystems.
• Contribute to innovation initiatives including automation, intelligent workflows, and next-generation analytics.

Required Skills & Experience

• 6+ years of strong hands-on software development experience.
• Proficiency in Java, Scala, Python, or C++, with strong coding and debugging skills.
• Deep understanding of Data Structures, Algorithms, OOPS, and design patterns.
• Experience in building enterprise-scale applications.
• Strong understanding of SDLC, Agile methodologies, and DevOps practices.
• Experience with application integration, data management, and production support.
• Ability to work independently and adapt quickly in fast-paced environments.

Preferred Qualifications

• Hands-on experience in Core Java or Scala.
• Exposure to functional programming paradigms.
• Experience with third-party libraries, frameworks, and open-source ecosystems.
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Experience in financial services, compliance, or regulated environments is a plus.
• Understanding of data governance, security, and regulatory controls.

Culture & Mindset

We are looking for individuals who:

• Demonstrate strong ownership and accountability.
• Are strategic, analytical, and data-driven.
• Can think creatively and solve problems unconventionally.
• Thrve in a fast-growing, entrepreneurial, and agile environment.
• Are both leaders and hands-on doers.
• Show high energy, curiosity, and commitment to excellence.