Data Engineer

Netlink; Gurgaon; On-site; Full time

Job description

Responsibilities

• Design, develop, and maintain efficient and scalable data pipelines using Python, PySpark, and Databricks on AWS.
• Work closely with data scientists, analysts, and other engineers to ensure smooth data integration and high-quality data processing.
• Build and optimize complex ETL (Extract, Transform, Load) workflows to handle large datasets.
• Implement data ingestion and transformation logic using PySpark on Databricks to improve data processing performance.
• Ensure data quality, accuracy, and consistency across all data pipelines.
• Collaborate with cross-functional teams to identify and resolve bottlenecks in data systems.
• Utilize AWS cloud services (e.g., S3, Redshift, RDS, EMR) for data storage and management.
• Write SQL queries to extract, manipulate, and transform data stored in relational databases.
• Develop, deploy, and maintain APIs for integrating data systems and enabling data access.
• Monitor and troubleshoot data pipelines, ensuring they operate smoothly with minimal downtime.
• Stay up to date with the latest industry tools, technologies, and best practices in cloud data engineering and big data processing.