Welspun GCC - Lead Data Engineer

Welspun GCC; Ahmedabad; On-site; INR 8,00,000 - 15,00,000; Full time

Job summary

An innovative firm is seeking a Data Engineering Lead to manage a talented team and drive the development of scalable data management systems. This role involves designing and implementing data pipelines, ensuring data quality, and collaborating with cross-functional teams to deliver impactful data solutions. The ideal candidate will possess strong analytical skills and be proficient in tools like Azure Databricks and SQL. Join this dynamic environment where you can leverage your expertise to influence strategic decisions and contribute to the company's growth. If you're passionate about data and leadership, this opportunity is perfect for you.

Qualifications

• Strong understanding of database structures and data management systems.
• Experience in using Python and SQL for data analysis and processing.

Responsibilities

• Lead the data engineering team and ensure high performance.
• Develop and implement data pipelines and ETL processes.
• Collaborate with data scientists on various projects.

Skills

Data Engineering

Data Visualization

SQL

Python

Azure Databricks

DevOps

Data Pipeline

ETL

Microsoft Fabric

Job description

Responsibilities of Data Engineering:

1. Lead and manage the data engineering team, ensuring high performance and smooth workflow.
2. Design, construct, install, test, and maintain highly scalable data management systems.
3. Develop and implement data pipelines, ensuring robustness and data quality.
4. Collaborate with data scientists and architects on several projects.
5. Use Azure Databricks for big data analytics and processing.
6. Implement ETL processes and work closely with the DevOps team to automate them.
7. Use SQL and Python for data querying and analysis.
8. Implement data visualization tools and techniques to present data in a way that is easily understandable for non-technical team members.
9. Use Microsoft Fabric for data integration and processing.
10. Apply strong analytical thinking to interpret data and make strategic decisions.
11. Ensure data privacy and comply with all relevant data protection regulations.
12. Continually improve ongoing reporting and analysis processes, automating or simplifying self-service support for secondary datasets.
13. Provide technical expertise and recommendations in assessing new IT software projects and initiatives to support and enhance our existing systems.
14. Conduct team meetings and provide training to team members as necessary.
15. Stay updated with the latest industry trends and technologies to ensure the company's data infrastructure is up-to-date.

The Data Engineering Lead should be a strategic thinker, excellent team leader, and a problem-solver with a strong analytical mindset. The ability to manage and interpret large amounts of data is crucial for this role.

Functional Competencies:

1. Data Engineering: The candidate should have a strong understanding of database structures, theories, principles, and practices. They should be able to design, construct, install, test, and maintain highly scalable data management systems.
2. Data Visualization: The candidate should be proficient in transforming data into readable, goal-driven reports for continued innovation and growth. They should be able to use data visualization tools to present data in a clear, concise manner.
3. SQL: The candidate should have a strong understanding of SQL database coding and management. They should be able to write and optimize SQL statements for data access and retention.
4. Python: The candidate should have experience in using Python for creating high-performing data processing scripts. They should be proficient in Python libraries for data analysis.
5. Azure Databricks: The candidate should have experience in using Azure Databricks for big data analytics and artificial intelligence solutions.
6. DevOps: The candidate should have a good understanding of DevOps principles and tools, and should be able to work in a fast-paced environment with a focus on continuous delivery.
7. Data Pipeline: The candidate should have experience in building and maintaining reliable and scalable data pipelines to meet business needs.
8. ETL: The candidate should have experience in designing, building, and maintaining ETL pipelines.

Behavioural Competencies:

1. Team Management: The candidate should have strong leadership skills and the ability to manage a team effectively. They should be able to delegate tasks, motivate team members, and resolve conflicts.
2. Analytical Thinking: The candidate should have strong analytical skills and the ability to make data-driven decisions. They should be able to analyze complex data sets and derive meaningful insights.
3. Data Analysis: The candidate should have strong data analysis skills. They should be able to analyze large amounts of raw information to find patterns and use data-driven insights to help the company make business decisions.

Good to have skills:

1. Microsoft Fabric: The candidate should have a deep understanding of Microsoft Fabric and its use in building scalable and reliable applications. They should be able to use Microsoft Fabric to design and implement high-quality solutions.