Senior Software Engineer, Data Engineering

ValGenesis; Chennai; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A leading technology company in Chennai seeks an experienced Data Engineer to design and maintain data pipelines and architectures. Responsibilities include developing ETL processes, optimizing data workflows, and collaborating with data scientists. Candidates should possess 4 to 8 years of experience, strong skills in Python and SQL, and familiarity with Azure Data Lake and Databricks. This role offers opportunities for innovation in data and AI technologies, aligned with a collaborative work culture.

Qualifications

• 4 to 8 years in Data Engineering or equivalent roles.
• Strong programming skills in Python and SQL, and at least one compiled language.
• Experience with both relational and NoSQL databases.

Responsibilities

• Design, develop, and maintain data ingestion and transformation pipelines.
• Optimize data Lakehouse architectures using Azure Synapse and Delta Lake.
• Collaborate with Data Scientists to prepare datasets for ML.

Skills

Python

SQL

Azure Data Lake

Databricks

ETL

Apache Airflow

Kubernetes

Power BI

Apache Kafka

Data Modeling

Tools

Azure Data Factory

Docker

GitHub Actions

Job description

About the Role

Responsibilities

• Design, develop, and maintain data ingestion, transformation, and orchestration pipelines (batch and real‑time).
• Build and optimize data Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
• Integrate and manage structured and unstructured data sources (SQL/NoSQL, files, documents, IoT streams).
• Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.
• Collaborate with Data Scientists to prepare and serve ML‑ready datasets for model training and inference.
• Implement data quality, lineage, and governance frameworks across pipelines and storage layers.
• Work with BI tools (Power BI, Superset, Tableau) to enable self‑service analytics for business teams.
• Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.
• Ensure scalability, performance, and observability of data workflows through effective monitoring and automation.
• Collaborate cross‑functionally with engineering, product, and business teams to translate insights into action.

Requirements

• Experience: 4 to 8 years in Data Engineering or equivalent roles.
• Programming: Strong in Python, SQL, and at least one compiled language (C#, Java, or Scala).
• Databases: Experience with relational (SQL Server, PostgreSQL, MySQL) and NoSQL (MongoDB, Cosmos DB) systems.
• Data Platforms: Hands‑on experience with Azure Data Lake, Databricks.
• ETL/ELT Tools: Azure Data Factory, Apache Airflow, or dbt.
• Messaging & Streaming: Kafka, Event Hubs, or Service Bus for real‑time data processing.
• AI/ML Exposure: Familiarity with ML frameworks (TensorFlow, PyTorch) and MLOps concepts.
• Visualization & Analytics: Power BI, Apache Superset, or Tableau.
• Cloud & DevOps: Azure, Docker, Kubernetes, GitHub Actions, Azure DevOps.
• Best Practices: Solid understanding of data modeling, version control, and CI/CD for data systems.

Preferred Skills (Nice to Have)

• Experience in knowledge graph or semantic search solutions.
• Understanding of LLM‑based data retrieval (RAG) patterns.
• Exposure to data mesh, data fabric, or domain‑oriented data architecture.
• Familiarity with MLflow, Delta Live Tables, or Databricks Unity Catalog.

Soft Skills

• Strong analytical and problem‑solving ability.
• Excellent communication and collaboration skills.
• Attention to detail and ability to work with large, complex datasets.
• Creativity and ability to automate repetitive workflows.
• Passion for continuous learning and innovation in data and AI technologies.

ValGenesis is an equal‑opportunity employer that makes employment decisions on the basis of merit. Our goal is to have the best‑qualified people in every job. All qualified applicants will receive consideration for employment without regard to race, religion, sex, sexual orientation, gender identity, national origin, disability, or any other characteristics protected by local law.