Data Engineer

Speridian; Bengaluru; On-site; INR 1,200,000 - 2,500,000; Full time

Job summary

A leading data solutions company is seeking an experienced Data Engineer based in Bengaluru, India. The ideal candidate will have 8 to 12 years of experience with profound knowledge in Azure Data Engineering, especially using Azure Data Factory and Databricks. Responsibilities include developing and maintaining complex data pipelines, designing data storage solutions, and collaborating with teams for effective data solutions. The position requires strong problem-solving skills and familiarity with data warehousing. Competitive compensation and benefits provided.

Qualifications

• 8 to 12 years of experience as a data engineer.
• Ability to optimize and troubleshoot data pipelines for performance and scalability.
• Experience working in an Agile environment.

Responsibilities

• Develop and maintain data pipelines and ETL processes using Azure Data Factory (ADF) and Databricks.
• Design and implement data storage solutions on Azure.
• Collaborate with cross-functional teams to gather and analyze business requirements.

Skills

Azure Data Engineering

Azure Data Factory (ADF)

Databricks

SQL

NoSQL databases

Data warehousing

ETL/ELT processes

Git

Azure DevOps

Cloud computing

Data security and privacy

Problem-solving

Collaboration

Agile environment

Education

Bachelor's or Master's degree in Computer Science, Engineering, or a related field

Job description

The candidate for this data engineer position brings with them 8 to 12 years of extensive experience in the field, making them a highly skilled and knowledgeable professional. They have a strong expertise in Azure Data Engineering, having worked with the platform for several years and gaining a deep understanding of its capabilities and functionalities. Their proficiency in Azure Data Engineering is complemented by their additional skills in ADF (Azure Data Factory) and Databricks, making them a well-rounded data engineer with a diverse set of technical skills. They have a proven track record of designing, building, and maintaining complex data pipelines and systems using these technologies, and have a thorough understanding of data warehousing, data modeling, and ETL processes. With their extensive experience and expertise, the candidate is well-equipped to handle any data engineering challenges and deliver high-quality solutions to drive business success.

Responsibilities

• Develop and maintain data pipelines and ETL processes using Azure Data Factory (ADF) and Databricks
• Design and implement data storage solutions on Azure, including Azure SQL Database, Azure Data Lake Storage, and Azure Blob Storage
• Create and manage data models and data warehouses using Azure Synapse Analytics
• Collaborate with cross-functional teams to gather and analyze business requirements and translate them into technical solutions
• Troubleshoot and optimize data processes to ensure high performance and data quality
• Implement data security measures to protect sensitive information
• Stay updated with the latest data engineering technologies and best practices
• Lead and mentor junior data engineers in the team
• Provide technical guidance and support to other teams and stakeholders
• Participate in code reviews and ensure adherence to coding standards and best practices
• Collaborate with DevOps teams to deploy and manage data pipelines and processes
• Conduct data analysis and provide insights to support business decisions
• Develop and maintain documentation for data processes and solutions
• Identify and troubleshoot data-related issues and provide timely resolutions

Skills

• 8 to 12 years of experience as a data engineer
• Proficient in designing, developing, and maintaining data pipelines in Azure
• Expertise in using Azure Data Factory (ADF) for data integration and orchestration
• Experience with Azure Databricks for data processing and analytics
• In-depth knowledge of SQL and NoSQL databases
• Experience with data warehousing and data modeling
• Familiarity with ETL/ELT processes and tools
• Ability to optimize and troubleshoot data pipelines for performance and scalability
• Experience with version control and deployment tools such as Git and Azure DevOps
• Strong understanding of cloud computing concepts and architecture
• Experience with data security and privacy best practices
• Excellent problem-solving and analytical skills
• Strong communication and collaboration skills
• Experience working in an Agile environment
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field