Senior Data Engineer

Mags Consulting; Chennai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Mags Consulting in Chennai, India, is seeking a data engineer to design, develop, and maintain scalable data pipelines and implement robust data processing solutions.

You will work with Apache Spark / PySpark, handle ingestion and integration from multiple sources, and leverage AWS data services to enable analytics.

You will ensure data quality, troubleshoot issues, and optimize performance while collaborating with stakeholders.

Qualifications

• Strong/advanced SQL skills and complex query optimization.
• Experience with Spark and distributed data processing.
• Experience with data ingestion, transformation, and integration.
• Familiarity with JSON, logs, PDFs, and other data formats.

Responsibilities

• Design, develop, and maintain scalable data pipelines and high-performance data processing solutions.
• Build distributed data processing using Apache Spark / PySpark.
• Develop ingestion, transformation and integration from multiple data sources and AWS services.
• Implement data validation, quality checks, and monitoring processes.

Skills

SQL

Apache Spark

PySpark

Data ingestion

Data transformation

Data integration

JSON

Logs

PDFs

Relational databases

NoSQL databases

Data quality

Performance optimization

Troubleshooting

Tools

AWS

Apache Spark

PySpark

Job description

• Design, develop, and maintain scalable and high-performance data pipelines .
  
• Build distributed data processing solutions using Apache Spark / PySpark .
  
• Develop data ingestion, transformation, and integration processes from multiple data sources.
  
• Work with AWS data engineering and cloud processing services .
  
• Process structured, semi-structured, and unstructured data including JSON, logs, PDFs, and other file formats .
  
• Develop complex and optimized SQL queries for data processing and analysis.
  
• Work with both relational and non-relational databases .
  
• Implement data validation, quality checks, and monitoring processes.
  
• Troubleshoot complex data issues and perform root-cause analysis.
  
• Optimize data pipelines, queries, and distributed processing jobs for performance and scalability.
  
• Collaborate with stakeholders to understand business requirements and translate them into scalable data solutions.
  
• Take ownership of data engineering initiatives and deliver solutions independently.
  
• Stay updated with emerging AWS, Big Data, and Data Engineering technologies and best practices .
  

Requirements

• Strong/advanced SQL, including complex queries and optimization
  
• Apache Spark and distributed data processing
  
• Data ingestion, transformation and integration
  
• Structured, semi-structured and unstructured data — JSON, logs, PDFs, etc.
  
• Relational and non-relational databases
  
• Data quality, validation, performance tuning and optimization
  
• Deep-dive data analysis and troubleshooting