Gen AI Data Engineer - Azure AI Foundry - Senior Associate

PwC; Pune District, Bengaluru, Delhi; Híbrido; INR 1 200 000 - 2 400 000; Tempo integral

Descrição da oferta de emprego

Responsibilities

• Design, develop, and maintain data pipelines and ETL processes for GenAI projects.
• Collaborate with data scientists and software engineers to implement machine learning models and algorithms.
• Optimize data infrastructure and storage solutions to ensure efficient and scalable data processing.
• Implement event-driven architectures to enable real-time data processing and analysis.
• Utilize containerization technologies like Kubernetes and Docker for efficient deployment and scalability.
• Develop and maintain data lakes for storing and managing large volumes of structured and unstructured data.
• Implement and integrate LLM frameworks (Langchain, Semantic Kernel) for advanced language processing and analysis.
• Collaborate with cross-functional teams to design and implement solution architectures for GenAI projects.
• Utilize cloud computing platforms such as Azure or AWS for data processing, storage, and deployment.
• Monitor and troubleshoot data pipelines and systems to ensure smooth and uninterrupted data flow.
• Stay up-to-date with the latest advancements in GenAI technologies and recommend innovative solutions to enhance data engineering processes.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Document data engineering processes, methodologies, and best practices.
• Maintain solution architecture certificates and stay current with industry best practices.

Requirements

• Python Proficiency: Minimum 3 years of hands-on experience building applications with Python.
• Scalable System Design: Solid understanding of designing and architecting scalable Python applications, particularly for Gen AI use cases, with a strong understanding of various components and systems architecture patterns to make cohesive and decoupled, scalable applications.
• Web Frameworks: Familiarity with Python web frameworks (Flask, FastAPI) for building web applications around AI models.
• Modular Design & Security: Demonstrated ability to design applications with modularity, reusability, and security best practices in mind (session management, vulnerability prevention, etc.,).
• Cloud-Native Development: Familiarity with cloud-native development patterns and tools (e.g., REST APIs, microservices, serverless functions).
• Cloud Deployments: Experience deploying and managing containerized applications on Azure/AWS (Azure Kubernetes Service, Azure Container Instances, or similar).
• Version Control (Git): Strong proficiency in Git for effective code collaboration and management.
• CI/CD: Knowledge of continuous integration and deployment (CI/CD) practices on cloud platforms.
• 3-5 years of relevant technical/technology experience, with a focus on GenAI projects.
• Strong programming skills in Python.
• Experience with data processing frameworks like Apache Spark or similar.
• Proficiency in SQL and database management systems.

Preferred Skills

• Gen AI Frameworks: Experience with LLM frameworks or tools for interacting with LLMs such as LangChain, Semantic Kernel, LlamaIndex
• Data Pipelines: Experience in setting up data pipelines for model training and real-time inference.