Data Engineer

Hoonartek; Pune District; On-site; INR 900,000 - 1,600,000; Full time

Job summary

Hoonartek is seeking a Data Engineer to design, develop, and optimize data pipelines using Databricks and PySpark. You will implement ETL processes, ensure data quality, and integrate diverse data sources with Delta Lake for reliable storage and analytics.

Required 3-4 years of experience with Databricks and PySpark, plus familiarity with big data tech. Strong collaboration and communication skills are essential in a fast-paced, delivery-driven environment.

Qualifications

• Bachelor's degree in CS/IT or related field.
• 3-4+ years of experience with Databricks and PySpark.
• Familiarity with big data technologies and distributed computing.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks and PySpark.
• Implement ETL processes with efficient extraction, transformation, and loading of data.
• Establish data quality checks and monitor data quality metrics.

Skills

Databricks

PySpark

ETL

Azure Databricks

Delta Lake

Education

Bachelor's degree in Computer Science

Tools

Spark

SQL Server

Job description

We empower enterprises globally through intelligent, creative, and insightful services for data integration, data analytics and data visualization.

Hoonartek is a leader in enterprise transformation, data engineering and an acknowledged world-class Ab Initio delivery partner.

Using centuries of cumulative experience, research and leadership, we help our clients eliminate the complexities & risk of legacy modernization and safely deliver big data hubs, operational data integration, business intelligence, risk & compliance solutions and traditional data warehouses & marts.

At Hoonartek, we work to ensure that our customers, partners and employees all benefit from our unstinting commitment to delivery, quality and value. Hoonartek is increasingly the choice for customers seeking a trusted partner of vision, value and integrity

How We Work?

Define, Design and Deliver (D3) is our in-house delivery philosophy. It’s culled from agile and rapid methodologies and focused on ‘just enough design’. We embrace this philosophy in everything we do, leading to numerous client success stories and indeed to our own success.

We embrace change, empowering and trusting our people and building long and valuable relationships with our employees, our customers and our partners. We work flexibly, even adopting traditional/waterfall methods where circumstances demand it. At Hoonartek, the focus is always on delivery and value.

Job Description

• Utilize Databricks and PySpark to design, develop, and maintain data processing pipelines.
• Implement ETL (Extract, Transform, Load) processes, ensuring efficient data extraction, transformation, and loading.
• Deep knowledge and hands-on of Databricks features and services - Feature Engineering, Lakeflow, Lakebase, Spark

2. Data Quality Implementation:

• Establish data quality checks within Azure Databricks.
• Ensure data accuracy, consistency, and adherence to business rules.
• Monitor data quality metrics and address anomalies promptly.

3. Source System Integration:

• Integrate Azure Databricks with various source systems (databases, data lakes, APIs).
• Efficiently ingest data from diverse sources into Azure Databricks.
• Handle schema evolution and changes in source data.

4. Pyspark Coding and Optimization:

• Write efficient PySpark code for data transformations, aggregations, and analytics.
• Optimize Spark jobs for performance and resource utilization.
• Troubleshoot and debug PySpark scripts as needed.

5. Collaboration and Communication:

• Work closely with data scientists, engineers, and other stakeholders.
• Gather requirements, understand business needs, and deliver effective solutions.
• Participate in cross-functional teams to ensure successful project outcomes.
• Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes.
• Implement Delta Lake tables for reliable data storage and versioning.

7 Stored Procedure Conversion in Databricks:

• Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code.
• Optimize and enhance stored procedures for better performance within Databricks

Job Requirement

Education: Bachelor's degree in Computer Science, Information Technology, or related field.

• Minimum 3-4 years of experience working with Databricks and PySpark.
• Familiarity with big data technologies and distributed computing.

Certifications (preferred):

• Microsoft Certified: Azure Data Engineer Associate or similar.
• Databricks Certified Associate Developer for Apache Spark.

• Strong problem-solving abilities and quick troubleshooting skills.
• Excellent communication and collaboration skills.
• Adaptability to work in a dynamic, team-oriented environment.