Data Engineer

Uplers; Kamrup Metropolitan; Remote; INR 21,25,000 - 25,00,000; Full time

Job summary

A technology company is offering a remote position requiring 3+ years of experience in designing and optimizing ETL pipelines. Candidates should be skilled in Apache Hudi, Flink, Iceberg, and have proficiency in Python and SQL. The role includes deploying data infrastructure on cloud platforms like AWS, along with other responsibilities related to data processing and governance. Competitive salary and performance bonuses are offered.

Benefits

Competitive salary

Performance bonus

Opportunity for rapid career advancement

Responsibilities

• Design, build, and optimize scalable ETL pipelines using Apache Airflow or similar frameworks.
• Utilize Spark (PySpark), Kafka, Flink, or similar tools for distributed data processing.
• Deploy, manage, and optimize data infrastructure on cloud platforms like AWS.
• Design and implement robust data models ensuring consistency, integrity, and performance.
• Enhance query performance through indexing, partitioning, and tuning techniques.
• Manage cloud-based storage solutions ensuring data governance and security.
• Work closely with data scientists and engineers to support data-driven decision-making.
• Strong proficiency in Python and SQL, with experience in Java or Scala.
• Hands-on experience with frameworks like Spark (PySpark) and Kafka.
• Familiarity with cloud platforms for managing data infrastructure.
• Strong understanding of data warehousing concepts.
• Experience with ETL tools like Apache Airflow.
• Proficiency in working with data lakes and cloud storage solutions.
• Expertise in Git for version control and collaborative coding.
• Expertise in performance tuning for large-scale data processing.

Skills

Apache Hudi

Flink

Iceberg

AWS

Python

PySpark

Batch Processing

Apache Spark

Job description

Opportunity Details

Experience: 3+ years

Salary: INR 2500000.00 / year (based on experience)

Expected Notice Period: 30 Days

Shift: (GMT+05:30) Asia/Kolkata (IST)

Opportunity Type: Remote

Placement Type: Full Time Permanent position (Payroll and Compliance to be managed by: NA)

Must have skills

• Apache Hudi
• Flink
• Iceberg
• AWS
• Python
• PySpark
• Batch Processing
• Apache Spark

NomuPay opportunity overview

• Opportunity in a company with a solid track record of performance
• Opportunity to work with diverse, global teams
• Rapid career advancement with opportunities to learn
• Competitive salary and Performance bonus

Responsibilities

• Design, build, and optimize scalable ETL pipelines using Apache Airflow or similar frameworks to process and transform large datasets efficiently.
• Utilize Spark (PySpark), Kafka, Flink, or similar tools to enable distributed data processing and real-time streaming solutions.
• Deploy, manage, and optimize data infrastructure on cloud platforms such as AWS, GCP, or Azure, ensuring security, scalability, and cost-effectiveness.
• Design and implement robust data models, ensuring data consistency, integrity, and performance across warehouses and lakes.
• Enhance query performance through indexing, partitioning, and tuning techniques for large-scale datasets.
• Manage cloud-based storage solutions (Amazon S3, Google Cloud Storage, Azure Blob Storage) and ensure data governance, security, and compliance.
• Work closely with data scientists, analysts, and software engineers to support data-driven decision-making, while maintaining thorough documentation of data processes.
• Strong proficiency in Python and SQL, with additional experience in languages such as Java or Scala.
• Hands-on experience with frameworks like Spark (PySpark), Kafka, Apache Hudi, Iceberg, Apache Flink, or similar tools for distributed data processing and real-time streaming.
• Familiarity with cloud platforms like AWS, Google Cloud Platform (GCP), or Microsoft Azure for building and managing data infrastructure.
• Strong understanding of data warehousing concepts and data modeling principles.
• Experience with ETL tools such as Apache Airflow or comparable data transformation frameworks.
• Proficiency in working with data lakes and cloud based storage solutions like Amazon S3, Google Cloud Storage, or Azure Blob Storage.
• Expertise in Git for version control and collaborative coding.
• Expertise in performance tuning for large-scale data processing, including partitioning, indexing, and query optimization.

How to apply for this opportunity

• Step 1: Click On Apply! And Register or Login on our portal.
• Step 2: Complete the Screening Form & Upload updated Resume
• Step 3: Increase your chances to get shortlisted & meet the client for the Interview