Data Engineer with Scala

Prophecy Technologies; Hyderabad; On-site; INR 2,000,000 - 4,000,000; Full time

Job summary

Prophecy Technologies is seeking a highly skilled Data Engineer with 6+ years of experience to design, develop, and maintain scalable data platforms and pipelines, using Apache Spark, Scala, Python, SQL, and cloud technologies (Azure/AWS). You will own end-to-end data engineering solutions.

Collaborate with business stakeholders, data scientists, and engineering teams to build reliable data ecosystems for analytics and reporting, while ensuring performance, data quality, and security.

Qualifications

• 6+ years of hands-on experience in Data Engineering, Data Warehousing, and Big Data technologies.
• Strong experience with Spark, Scala, and Python in enterprise environments.
• Experience with Azure and/or AWS cloud platforms for data storage and processing.
• Advanced SQL knowledge including complex queries, data modeling, and optimization.
• Experience designing end-to-end ETL/ELT workflows for large-scale data processing.
• Ability to own and deliver data solutions from requirements to production.
• Strong troubleshooting and problem-solving for data quality and performance.
• Experience working in Agile/Scrum teams with cross-functional stakeholders.

Responsibilities

• Design, develop, and maintain large-scale batch and real-time data pipelines using Spark, Scala, and Python.
• Build scalable data processing frameworks to ingest, transform, and integrate data from multiple sources.
• Develop and optimize complex SQL queries, data models, and analytics-ready datasets.
• Design and implement cloud-based data solutions using Azure and AWS.
• Create and manage ETL/ELT workflows with cloud-native tools (ADF, Data Lake, Glue, S3).
• Collaborate with stakeholders, data analysts, and data scientists to deliver data solutions.
• Monitor, troubleshoot, and optimize pipelines for performance and data quality.
• Implement data governance, security, and operational excellence practices.
• Participate in code reviews and architectural discussions.
• Support CI/CD and automation of data engineering workflows.
• Work with distributed processing systems and platform modernization.
• Mentor junior team members and share knowledge within the team.

Skills

Apache Spark

Scala

Python

SQL

Azure

AWS

CI/CD

DevOps

Agile/Scrum

Mentoring

Data Modeling

Tools

Delta Lake

Databricks

Kafka

Azure Data Factory

Azure Synapse Analytics

AWS Glue

Job description

We are seeking a highly skilled and motivated Data Engineer with 6+ years of experience in designing, developing, and maintaining scalable data platforms and pipelines. The ideal candidate will possess strong expertise in Apache Spark, Scala, Python, SQL, and Cloud Technologies (Azure/AWS), with the ability to independently own and deliver end-to-end data engineering solutions. The role requires working closely with business stakeholders, data scientists, and engineering teams to build reliable, high-performance, and scalable data ecosystems that support analytics, reporting, and advanced data-driven initiatives.

Key Responsibilities:

• Design, develop, and maintain large-scale batch and real-time data pipelines using Apache Spark, Scala, and Python.
• Build scalable and reliable data processing frameworks for ingesting, transforming, and integrating data from multiple sources.
• Develop and optimize complex SQL queries, stored procedures, and data models to support reporting and analytics requirements.
• Design and implement cloud-based data solutions using Azure and/or AWS services.
• Create and manage ETL/ELT workflows using cloud-native tools such as Azure Data Factory, Azure Data Lake, AWS Glue, and Amazon S3.
• Collaborate with business stakeholders, data analysts, and data scientists to understand requirements and deliver data solutions.
• Monitor, troubleshoot, and optimize data pipelines to ensure performance, reliability, and data quality.
• Implement best practices for data governance, security, scalability, and operational excellence.
• Participate in code reviews, architecture discussions, and technical design sessions.
• Support CI/CD implementation and automation of data engineering workflows.
• Work with distributed data processing systems and contribute to platform modernization initiatives.
• Mentor junior team members and contribute to knowledge-sharing activities within the team.

Experience Required:

• 6+ years of hands-on experience in Data Engineering, Data Warehousing, and Big Data technologies.
• Strong experience developing scalable data pipelines using Apache Spark, Scala, and Python in enterprise environments.
• Proven experience working with cloud platforms such as Microsoft Azure and/or AWS, including data storage, processing, and integration services.
• Advanced knowledge of SQL, including complex query development, performance tuning, data modeling, and query optimization.
• Experience designing and implementing end-to-end ETL/ELT workflows for large-scale data processing and analytics.
• Demonstrated ability to independently own and deliver data engineering solutions from requirements gathering through deployment and production support.
• Strong troubleshooting and problem-solving skills with experience resolving complex data quality, performance, and scalability challenges.
• Experience working within Agile/Scrum teams and collaborating effectively with cross-functional stakeholders to deliver business-critical data solutions.

Preferred to Have Skills:

• Experience with Azure Data Factory (ADF), Azure Synapse Analytics, or AWS Glue.
• Hands-on experience with Apache Kafka or other real-time streaming platforms.
• Knowledge of Delta Lake, Databricks, or Lakehouse architectures.
• Experience with CI/CD pipelines and DevOps practices for Data Engineering.
• Familiarity with Data Governance, Data Quality, and Metadata Management frameworks.
• Exposure to Generative AI, Machine Learning data pipelines, or Analytics platforms is an added advantage.