JD17 - Consultant - DBT

Kivitronicsconsulting; Chennai District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Kivitronicsconsulting in Chennai invites a client-facing data engineering consultant to design and deliver secure, scalable data platforms using Python, PySpark, dbt and Kafka. You will own end-to-end engineering across ingestion, transformation, modelling, governance, and CI/CD, partnering with stakeholders to translate requirements into technical solutions.

The role emphasizes architecture work, data governance, production reliability, and mentoring junior engineers while collaborating with BI

Qualifications

• Bachelor's degree in CS/IT or equivalent professional experience.
• 4+ years in data engineering with Python/PySpark, dbt, and Kafka.
• Strong knowledge of data governance, security, and scalable architectures.

Responsibilities

• Lead requirements workshops and translate business needs into user stories and architecture patterns.
• Design secure, scalable data solutions across lakehouse and streaming paradigms.
• Prepare architecture diagrams, dataflow specs, and runbooks.
• Ensure governance and compliance (GDPR/HIPAA/SOC 2) across batch and streaming.
• Collaborate with cross-functional teams and mentor junior engineers.
• Build ingestion pipelines using Python and PySpark for distributed processing.

Skills

Python

PySpark

dbt

Kafka

Data engineering

Education

Bachelor's degree in Computer Science / Information Technology

Tools

Schema Registry

Debezium

Spark Structured Streaming

Delta Lake

Git

Job description

Balamurugan Nagar,Keelkattalai, India | Posted on 01/10/2026

Client-facing consultantresponsible for designing and delivering secure, scalable, and cost-efficientdata platforms using Python, PySpark (Apache Spark), dbt (ELT), and ApacheKafka for streaming. Owns end-to-end engineering across ingestion (batch + streaming),transformation, modelling, governance, performance tuning, CI/CD, andoperations. Partners with stakeholders to translate business requirements intotechnical solutions, produce design artifacts, and ensure reliable delivery.

Duties and responsibilities ofposition:

General areas of responsibilityrelate to:

• Lead/co-lead requirements workshops; translatebusiness needs into user stories and architecture patterns combining PySpark,dbt, and Kafka.
• Design secure, scalable, cost-optimized datasolutions across lakehouse + warehouse and streaming paradigms.
• Prepare HLD/LLD, architecture diagrams, dataflow/mapping specifications, estimations, and runbooks.
• Ensure compliance and governance (e.g., GDPR,HIPAA, SOC 2) and enterprise standards across batch and streaming.
• Support production reliability throughmonitoring, incident response, and continuous improvement.
• Collaborate with cross-functional teams (BI,Cloud Ops, Data Science) and mentor junior engineers.

Requirements

Specific areas of responsibility include but are not limited to:

• Design and implement Kafka producers/consumers(Python or JVM-based) with exactlyonce/at-least-once semantics per usecase.
• Configure topics, partitions, replication,schemas (Avro/JSON/Protobuf) with Schema Registry and compatibility rules.
• Implement stream processing via Kafka Streams,Spark Structured Streaming (PySpark), or Flink (if applicable).
• Build CDC pipelines from databases usingDebezium/CDC connectors; manage offsets, replays, and dead-letter queues.
• Enforce resilience patterns: idempotency,retries/backoff, circuit breakers, exactly-once sinks (Delta/Snowflake/etc.).
• Build ingestion frameworks using Python(files/APIs) and PySpark for distributed processing; parameterizeenvironments.
• Standardize medallion zones (bronze/silver/gold)with file formats (Parquet/Delta/CSV/JSON) and metadata conventions.
• Implement incremental loads and hybridmicro-batch strategies for streaming/batch convergence.
• Develop scalable ETL/ELT using PySpark(DataFrames, Spark SQL) and dbt for modular SQL models (sources → staging →marts).
• PySpark/Delta Lake MERGE with surrogate keys,effective/expiry dates, current flags, audit columns.
• dbt snapshots and incremental models withmaterializations (table/incremental).
• Optimize jobs: partitioning,Z-ordering/bucketing, broadcast joins, predicate pushdown, and shuffletuning.
• Enforce coding standards: modular Pythonpackages, parameterized notebooks/jobs, dbt macros and packages.
• Design dimensional models (star/snowflake) withconformed dimensions and KPI-aligned fact tables.
• Build dbt semantic layers with tests (unique/notnull/relationships), docs, exposures, and data contracts.
• Manage schema evolution and late-arriving data;ensure reproducibility with versioned models.
• Apply RBAC in data platforms; manage secrets viaKey Vault/Secrets Manager and environment variables.
• Implement encryption in transit/at rest, datamasking, and row/column-level policies in target systems.
• Maintain lineage and catalog (dbt docs/manifest;Purview/Collibra/Atrium where available); steward metadata and tagging.
• Profile PySpark jobs (stage/task metrics); fixdata skew, excessive shuffles, small-file problems; tune cluster configs.
• Tune dbt models (query plans,indices/distributions, materializations, incremental strategies).
• Optimize Kafka workloads: topic partitioning,consumer group balancing, backpressure, batch size/linger.ms tuning.
• Version control with Git; implement CI/CD forPython packages, PySpark jobs, dbt projects, and Kafka artifacts(schemas/config).
• Automate quality gates: unit tests, data tests(dbt), schema compatibility checks, linting, and release notes.
• Implement observability: job/app telemetry,logs, metrics dashboards for SLAs/SLOs, freshness, and latency (batch +streaming).
• Set alerts for failures/anomalies; conduct RCAand remediation plans; track data downtime.
• Optimize costs: autoscaling/serverless (whereapplicable), warehouse/cluster rightsizing, storage lifecycle, usagetagging/reporting.

Other duties:

• Lead solution walkthroughs and trade-offanalysis (PySpark vs. dbt, streaming vs. batch, Delta vs. warehouse).
• Build POCs (CDC with Debezium, SCD2 withDelta/dbt snapshots, Kafka throughput/latency benchmarks) to de-riskdelivery.
• Manage risks/assumptions, change requests, andstakeholder communications; maintain decisions & actions log.
• Contribute to proposals, estimations, and SOWsfor Python/PySpark/dbt/Kafka engagements.
• Prepare executive reports on pipeline health,governance posture, streaming reliability, and cost optimization.
• Support integrations with upstream/downstreamplatforms (APIs, event streams, BI tools, warehouses/lakehouses).
• Conduct internal knowledge-sharing; buildaccelerators (dbt macros, PySpark templates, Kafka schema standards); mentorassociates.

Qualifications:

• Bachelor’s degree in Computer Science,Information Technology, or a related field; or equivalent professionalexperience.
• Experience: 4+ years in data engineering/BIconsulting with strong hands-on work in Python, PySpark, dbt, and Kafka.
• PySpark: DataFrame APIs, Spark SQL,performance tuning (partitions, caching, joins, shuffle), structuredstreaming.
• dbt: sources/staging/marts,materializations (table/incremental), snapshots, tests, docs; Jinja/macros;packages.
• Kafka: producers/consumers,topics/partitions/replication, Schema Registry (Avro/JSON/Protobuf),Streams/Structured Streaming, offset management, DLQs.
• CDC/SCD: incremental loads, watermarks,SCD1/SCD2 via Delta MERGE or dbt snapshots/incremental patterns; Debezium/CDCconnectors (nice-to-have).
• Modelling: star/snowflake, conformeddimensions, semantic consistency.
• Storage/Compute: Lakehouse/warehouse(Delta/Parquet; Snowflake/Synapse/BigQuery/Databricks—any preferred).
• Security/Governance: RBAC, secrets,encryption, masking/row policies; catalogue/lineage basics.
• Hands-on experience in data modeling, indexing,and query performance tuning.
• Experience with backup, disaster recovery, andbusiness continuity planning.
• Strong understanding of cloud security risks,including data breaches, malware, and social engineering.
• Excellent collaboration skills for working withcross-functional teams and providing leadership in cloud security projects.
• Experience working in a team, able to lead,prioritize and execute tasks in a high-pressure environment and provide or makesound decisions in emergencies.