Senior Data Engineer

The Opportunity

We are seeking a skilled Data Engineer to join our Data & Digital team. This role focuses on enhancing and maintaining data platform architecture, operationalizing Generative AI solutions, and implementing automation solutions. This individual must be self-directed and comfortable in addressing the data needs of multiple teams, systems, and products. The right candidate will be enthusiastic about the opportunity to optimize or even redesign the data architecture to support the next generation of products and data initiatives. The expertise of the Data Engineer is crucial in building a robust data infrastructure so that all stakeholders have seamless access to reliable and accurate data.

Your Role

1. Delivering solutions using advanced techniques in data modeling, ETL, data visualization, and more to derive insights from both structured and unstructured data.
2. Maintaining, reviewing, and supporting the development of automation solutions.
3. Integrating, consolidating, cleansing, and structuring large data sets to facilitate business insights and reporting.
4. Designing, implementing, and maintaining data pipelines for data ingestion, processing, and transformation in Azure Cloud.
5. Proactively understanding business requirements and translating them into superior software implementations using an agile approach.
6. Collaborating with data scientists and analysts to understand data needs and operationalize GenAI use cases and automation.
7. Utilizing Databricks, Azure Data Factory, or similar technologies to create and maintain ETL operations, including data quality checks.
8. Enhancing the scalability, efficiency, and cost-effectiveness of data pipelines.
9. Monitoring and resolving data pipeline issues to ensure data consistency and availability, while working towards improving data quality.
10. Defining and implementing data governance processes.

Your Skills and Experience

1. Approximately 6+ years of experience in a Data Engineer role with a degree in Computer Science, Data Science, Information Systems, or a related field.
2. Advanced working knowledge of SQL and experience with relational databases.
3. Experience in building and optimizing 'big data' pipelines, architectures, and datasets.
4. Proven history of manipulating, processing, and extracting value from large datasets, including building processes for data transformation, data structures, metadata, dependency, and workload management.
5. Working knowledge of Master Data and Reference Data Management.
6. Working knowledge of Data Modeling.
7. Experience in developing and integrating Power BI reports and dashboards.
8. Experience in supporting and collaborating with cross-functional teams in a dynamic environment.
9. Working knowledge of building user interfaces using web technologies.
10. Experience with the following software/tools:
11. Azure cloud services- Microservices, APIs, Function, Databricks, ADF, LogicApp.
12. Relational SQL and NoSQL databases.
13. Data pipeline and workflow management tools: Control-M, Airflow.
14. Basic understanding of stream-processing systems: Storm, Spark-Streaming.
15. Object-oriented/ scripting languages: Python, pySpark, Java, Scala.
16. Web technologies: Streamlit, React.js, HTML, etc.
17. Basic knowledge of Data Science and LLMs.
18. Strong problem-solving abilities.
19. Excellent verbal and written communication skills.