Data Engineer

Coretek; Kondapur; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Coretek is seeking a Data Engineer to design and operate data pipelines and models that power the business. You will own ingestion from source systems through to curated datasets used by analysts, data scientists, and application teams.

This is a hands-on role requiring production code, schema design, and accountability for reliability and cost. You will partner with stakeholders across teams to turn ambiguous requirements into durable data contracts, implement data quality checks, and ensure

Qualifications

• 5+ years building production data pipelines.
• Hands-on Python development with testing and packaging.
• Strong SQL and PySpark skills for large-scale data processing.
• Experience with Azure data services and data governance.

Responsibilities

• Design, build, and maintain batch and streaming data pipelines that are idempotent, observable, and recoverable.
• Model data for analytics (dimensional models, semantic layers, and curated marts).
• Integrate data from operational databases, SaaS APIs, files, and event streams, including handling schema drift.
• Build data quality checks into pipelines and define how failures alert and escape.
• Own pipelines in production: monitoring, on-call rotation for data incidents, root-cause analysis, and backfills.
• Tune performance and cost (partitioning, clustering, file sizing, warehouse and cluster sizing).
• Apply engineering discipline to data: version control, code review, CI/CD, automated testing, and IaC.
• Implement access controls, PII handling, retention, and lineage and audit requirements with security/compliance.
• Partner with analysts, data scientists, and product engineers to turn ambiguous requirements into data contracts.
• Maintain data dictionaries, lineage, and pipeline runbooks so consumers can use datasets correctly.

Skills

Python

PySpark

SQL

CI/CD

Git

Communication skills

Analytical thinking

Consultative approach

Tools

Azure Data Factory

Databricks

Synapse/Fabric

ADLS

Terraform

Bicep

Docker

Kubernetes

GitHub

Azure DevOps

Job description

Job Description:

Coretek is looking for a Data Engineer to build and operate the pipelines and data models that the rest of the business runs on. Youll own ingestion from source systems through to curated, well-documented datasets that analysts, data scientists, and application teams depend on. This is a hands-on engineering role: youll write production code, design schemas, and be accountable for the reliability and cost of what you ship.

Responsibilities

• Design, build, and maintain batch and streaming data pipelines that are idempotent, observable, and recoverable.
• Model data for analytics (dimensional models, semantic layers, and curated marts), balancing query performance against maintainability.
• Integrate data from operational databases, SaaS APIs, files, and event streams, including handling schema drift and late-arriving data.
• Build data quality checks (freshness, volume, uniqueness, referential integrity) into pipelines rather than bolting them on afterward, and define how failures alert and escape.
• Own pipelines in production: monitoring, on-call rotation for data incidents, root-cause analysis, and backfills.
• Tune performance and cost (partitioning, clustering, file sizing, warehouse and cluster sizing) and make the tradeoffs explicit.
• Apply engineering discipline to data: version control, code review, CI/CD, automated testing, and infrastructure as code.
• Implement access controls, PII handling, retention, and lineage and audit requirements in partnership with security and compliance.
• Partner with analysts, data scientists, and product engineers to turn ambiguous requirements into durable data contracts.
• Maintain data dictionaries, lineage, and pipeline runbooks so consumers can find a dataset, understand what each field means and how current it is, and use it correctly without having to ask the team that built it.

• 5+ years building production data pipelines.
• Strong hands-on Python development for data engineering, with real testing, packaging, and code review practice, not scripting alone.
• Working knowledge of PySpark: DataFrame and SQL APIs, joins and aggregations at scale, partitioning and shuffle behavior, and the ability to read a Spark UI to diagnose a slow or failing job.
• Strong SQL: window functions, query plans, and performance tuning, not just SELECTs.
• Hands-on experience with the Azure data platform: Data Factory, Databricks, Synapse/Fabric, and ADLS.
• Solid data modeling fundamentals: normalization, star schemas, slowly changing dimensions.
• Git-based workflow and experience shipping through CI/CD.
• Excellent communication skills, with the ability to debug a failing pipeline end to end and articulate the impact to diverse audiences, including non-technical stakeholders.
• Exceptional analytical and problem-solving skills, with the judgment to find the root cause of a data issue rather than patching the symptom.
• Strong knowledge and experience in working with customers in a consultative approach in a technical environment.

Additional Qualifications

• Streaming experience (Kafka, Event Hubs).
• Lakehouse formats: Delta Lake, Iceberg.
• Infrastructure as code (Terraform, Bicep) and containerization (Docker, Kubernetes).
• Experience in a regulated environment (HIPAA, SOC 2, PCI, GDPR): auditability, encryption, data residency.
• Experience building data platforms for ML or supporting feature pipelines.
• Proven ability to manage multiple client projects and deliver high-quality results on time.
• Experience in Azure DevOps or GitHub for source control and pipelines.