Expert Data Engineer (Databricks)

Codvo.ai; Pune District; On-site; INR 800,000 - 1,200,000; Full time

Job summary

Codvo.ai is seeking a Data Engineer_Expert in Pune District, India, responsible for designing, building, and maintaining Databricks data pipelines. The role includes operationalizing machine learning models and collaborating with data scientists while ensuring data quality and reliability.

The ideal candidate will have strong experience with the Databricks platform, Python development for ETL processes, and familiarity with AWS data services. This role emphasizes best practices in DevOps/MLOps.

Qualifications

• Databricks platform experience is essential.
• Strong proficiency in Python for ETL processes.
• Knowledge of Unity Catalog for data management.

Responsibilities

• Design and maintain Databricks data pipelines using Spark/Delta Lake.
• Operationalize machine learning models with robust validation processes.
• Collaborate with data scientists to manage model deployment lifecycles.

Skills

Databricks platform experience

Python development for data processing and ETL pipelines

Unity Catalog knowledge

AWS data services (S3, IAM, VPC, potentially Glue/Lambda)

Data lake/lakehouse architecture patterns

Dashboard building experience

Job description

Data Engineer_Expert

Job Description

• Design, build, and maintain Databricks data pipelines (ETL/ELT) for ingestion, transformation, and orchestration using Spark/Delta Lake/Databricks Workflows.
• Operationalize machine learning models by building inference pipelines that invoke models authored by data scientists (batch or real-time), ensuring consistency between training and inference environments.
• Ensure data reliability, quality, and observability through robust validation, monitoring, alerting, and automated recovery mechanisms.
• Collaborate closely with data scientists to productionize models, manage model deployment lifecycles, and optimize inference performance and cost.
• Implement best‑practice DevOps/MLOps processes such as CI/CD for pipelines, model versioning, environment promotion, and infrastructure‑as‑code.
• Optimize performance and cost across compute clusters, jobs, and storage layers.
• Implement and manage the enterprise data catalog, including schema design, table ownership, lineage, governance, and documentation using Unity Catalog.
• Experience with some Databricks infrastructure.
• Experience with building BI dashboards and visualization.
• Experience with coding agents and best practices (spec‑driven development, etc.).

Must Have Skills

• Databricks platform experience
• Python development for data processing and ETL pipelines
• Unity Catalog knowledge
• AWS data services (S3, IAM, VPC, potentially Glue/Lambda)
• Data lake/lakehouse architecture patterns
• Dashboard building experience

Nice to Have Skills

• RESTful API design and development (Flask, FastAPI, or similar)
• Authentication/authorization patterns (OAuth, API keys, IAM roles)
• Query optimization and performance tuning
• PySpark optimization experience
• ML/AI pipeline experience
• Databricks AI/BI