Data Engineer - Scala/PySpark

Forward Eye Technologies; Dadri; On-site; INR 700,000 - 1,200,000; Part time

Job summary

Forward Eye Technologies is seeking a data engineer for a 3-month contract to design and implement scalable data pipelines using PySpark/Scala, Spark, Hadoop, and Kafka. The role focuses on data modeling, integration, and performance optimization in a cloud environment.

You will collaborate with DevOps and security teams to ensure data quality, compliance, and robust infrastructure, while documenting architectures and sharing knowledge with cross-functional groups.

Qualifications

• Data engineering role requiring experience with Spark, Hadoop, and SQL.
• Proficiency in streaming tech and ETL tooling.
• Hands-on experience with cloud-based data platforms and data security.

Responsibilities

• Data Pipeline Development: Design, implement, and maintain scalable data pipelines to collect, process, and store large volumes of structured and unstructured data.
• Data Modeling: Develop and maintain data models, schemas, and metadata to support data initiatives and ensure integrity.
• Data Integration: Integrate data from various sources, including databases, data warehouses, APIs, and streaming platforms, ensuring quality.
• Performance Optimization: Optimize data pipelines for performance, scalability, and reliability; resolve bottlenecks.
• Data Quality Assurance: Implement data quality checks and monitoring.
• Infrastructure Management: Collaborate with DevOps to provision and manage infrastructure for data processing.
• Security and Compliance: Implement data security practices and ensure regulatory compliance.
• Documentation and Communication: Document pipelines and architectures; communicate with cross-functional teams.
• Continuous Learning: Stay updated on emerging data engineering technologies and approaches.

Skills

PySpark/Scala with Spark

Spark Architecture

Hadoop

SQL

Kafka

Airflow

S3

StreamSets

AWS IAM

AWS EMR

Snowflake

StreamSets

Tools

Kafka

Airflow

AWS S3

AWS IAM

AWS EMR

Snowflake

Spark

StreamSets

Job description

Contract Duration : 3 months

Skills :

• - PySpark or Scala with Spark, Spark Architecture, Hadoop, SQL
• - Streaming Technologies like Kafka etc.
• - Proficiency in Advanced SQL (Window functions)
• - Airflow, S3, and Stream Sets or similar ETL tools.
• - Basic Knowledge on AWS IAM, AWS EMR and Snowflake.

Responsibilities :

• - Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines to collect, process, and store large volumes of structured and unstructured data.
• - Data Modeling: Develop and maintain data models, schemas, and metadata to support the organization's data initiatives. Ensure data integrity and optimize data storage and retrieval processes.
• - Data Integration: Integrate data from various sources, including databases, data warehouses, APIs, and streaming platforms, ensuring compatibility, consistency, and quality.
• - Performance Optimization: Optimize data pipelines and processing systems for performance, scalability, and reliability. Identify and resolve bottlenecks and inefficiencies in data workflows.
• - Data Quality Assurance: Implement data quality checks, validation rules, and monitoring mechanisms to ensure the accuracy, completeness, and consistency of data across different systems.
• - Infrastructure Management: Collaborate with DevOps and IT teams to provision, configure, and manage infrastructure components such as databases, clusters, and cloud services required for data processing and storage.
• - Security and Compliance: Implement data security best practices and compliance standards to protect sensitive information and ensure regulatory compliance (e.g., GDPR, HIPAA). Monitor and audit data access and usage to prevent unauthorized activities.
• - Documentation and Communication: Document data pipelines, systems architecture, and technical processes. Communicate effectively with cross-functional teams to gather requirements, provide updates, and address issues.
• - Continuous Learning: Stay updated on emerging technologies, tools, and techniques in data engineering and related fields. Evaluate and recommend new technologies and approaches to improve data infrastructure and workflows.