Lead Data Engineer with ML Experience

Enable Data Incorporated; Hyderabad; On-site; INR 1,600,000 - 2,500,000; Full time

Job summary

Enable Data Incorporated is seeking a Senior Data Engineer in Hyderabad, India, to design and maintain robust cloud-based data solutions using Apache Spark and Databricks. The ideal candidate will have over 10 years of experience and strong skills in cloud technologies, data integration, and SQL. You will collaborate with cross-functional teams, gather data requirements, and optimize processing pipelines. This position offers an exciting opportunity to work with cutting-edge technologies and deliver high-quality solutions, contributing to ongoing improvements in data engineering practices.

Qualifications

• 10+ years of experience as a Data Engineer or similar role focused on cloud-based solutions.
• Experience designing and implementing data processing pipelines with Spark and Databricks.
• Strong knowledge of data governance and compliance frameworks.

Responsibilities

• Design, develop, and maintain scalable data solutions in the cloud using Spark and Databricks.
• Gather and analyze data requirements and identify insights opportunities.
• Ensure data quality, integrity, and security throughout the data lifecycle.

Skills

Cloud technologies

Apache Spark

Databricks

SQL

Data integration

ETL processes

Problem-solving

Collaboration

Education

Bachelor's or Master's degree in computer science, engineering or a related field

Tools

Apache Airflow

Azure

Job description

Enable Data Incorporated is seeking a highly skilled Senior Data Engineer with expertise in cloud technologies, Spark, and Databricks to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem.

As a Senior Data Engineer, you will play a crucial role in designing, developing, and maintaining scalable and efficient data solutions in the cloud. You will work closely with cross-functional teams to gather data requirements, design and implement data processing pipelines using Apache Spark and Databricks, and ensure data quality and integrity. Additionally, you will collaborate with clients to optimize their data infrastructure and provide insights using advanced analytics.

This is an excellent opportunity for a talented Data Engineer with strong cloud engineering skills and a deep understanding of Spark and Databricks. If you are passionate about leveraging cutting-edge technologies to solve complex data challenges and have a track record of delivering high-quality solutions, we would love to hear from you.

Responsibilities

• Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and Databricks.
• Gather and analyze data requirements from business stakeholders and identify opportunities for data-driven insights.
• Build and optimize data pipelines for data ingestion, processing, and integration using Spark and Databricks.
• Ensure data quality, integrity, and security throughout all stages of the data lifecycle.
• Collaborate with cross-functional teams to design and implement data models, schemas, and storage solutions.
• Optimize data processing and analytics performance by tuning Spark jobs and leveraging Databricks features.
• Provide technical guidance and expertise to junior data engineers and developers.
• Stay up-to-date with emerging trends and technologies in cloud computing, big data, and data engineering.
• Contribute to the continuous improvement of data engineering processes, tools, and best practices.

• Bachelor's or Master's degree in computer science, engineering, or a related field.
• 10+ years of experience as a Data Engineer, Software Engineer, or similar role, with a focus on building cloud-based data solutions.
• Strong experience with cloud platforms such as Azure
• Proficiency in Apache Spark and Databricks for large-scale data processing and analytics.
• Experience in designing and implementing data processing pipelines using Spark and Databricks.
• Strong knowledge of SQL and experience with relational and NoSQL databases.
• Experience with data integration and ETL processes using tools like Apache Airflow or cloud-native orchestration services.
• Good understanding of data modeling and schema design principles.
• Experience with data governance and compliance frameworks.
• Excellent problem-solving and troubleshooting skills.
• Strong communication and collaboration skills to work effectively in a cross-functional team.
• Relevant certifications in cloud platforms, Spark, or Databricks are a plus.