JD19 - Consultant - DBT

Kivitronicsconsulting; Chennai District; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Kivitronicsconsulting is seeking a client-facing data engineering consultant to design and deliver secure, scalable data platforms using PySpark, Kafka and dbt for streaming and batch processing.

You will own end-to-end ingestion, transformation, governance, and CI/CD, collaborate with BI and Cloud Ops, and mentor junior engineers while ensuring reliability and cost efficiency.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field.
• 4+ years in data engineering or consulting with hands-on Python, PySpark, dbt, and Kafka.
• Proficiency in PySpark, Kafka, dbt, and data modeling concepts.

Responsibilities

• Lead/work with stakeholders to translate requirements into PySpark, dbt, and Kafka patterns.
• Design secure, scalable data solutions across lakehouse and streaming paradigms.
• Prepare architecture diagrams, dataflow specs, and runbooks; ensure governance and compliance.

Skills

PySpark

Kafka

Python

dbt

Data pipelines

Streaming

SCD2

Education

Bachelor’s degree in Computer Science or related field

Tools

Kafka

Schema Registry

Delta Lake

dbt

Spark

Debezium

Job description

Balamurugan Nagar,Keelkattalai, India | Posted on 01/10/2026

Client-facing consultantresponsible for designing and delivering secure, scalable, and cost-efficientdata platforms using Python, PySpark (Apache Spark), dbt (ELT), and ApacheKafka for streaming. Owns end-to-end engineering across ingestion (batch + streaming),transformation, modelling, governance, performance tuning, CI/CD, andoperations. Partners with stakeholders to translate business requirements intotechnical solutions, produce design artifacts, and ensure reliable delivery.

Duties and responsibilities ofposition:

General areas of responsibilityrelate to:

• Lead/co-lead requirements workshops; translatebusiness needs into user stories and architecture patterns combining PySpark,dbt, and Kafka.
• Design secure, scalable, cost-optimized datasolutions across lakehouse + warehouse and streaming paradigms.
• Prepare HLD/LLD, architecture diagrams, dataflow/mapping specifications, estimations, and runbooks.
• Ensure compliance and governance (e.g., GDPR,HIPAA, SOC 2) and enterprise standards across batch and streaming.
• Support production reliability throughmonitoring, incident response, and continuous improvement.
• Collaborate with cross-functional teams (BI,Cloud Ops, Data Science) and mentor junior engineers.

Requirements

Specific areas of responsibility include but are not limited to:

• Design and implement Kafka producers/consumers(Python or JVM-based) with exactlyonce/at-least-once semantics per usecase.
• Configure topics, partitions, replication,schemas (Avro/JSON/Protobuf) with Schema Registry and compatibility rules.
• Implement stream processing via Kafka Streams,Spark Structured Streaming (PySpark), or Flink (if applicable).
• Build CDC pipelines from databases usingDebezium/CDC connectors; manage offsets, replays, and dead-letter queues.
• Enforce resilience patterns: idempotency,retries/backoff, circuit breakers, exactly-once sinks (Delta/Snowflake/etc.).
• Build ingestion frameworks using Python(files/APIs) and PySpark for distributed processing; parameterizeenvironments.
• Standardize medallion zones (bronze/silver/gold)with file formats (Parquet/Delta/CSV/JSON) and metadata conventions.
• Implement incremental loads and hybridmicro-batch strategies for streaming/batch convergence.
• Develop scalable ETL/ELT using PySpark(DataFrames, Spark SQL) and dbt for modular SQL models (sources → staging →marts).
• PySpark/Delta Lake MERGE with surrogate keys,effective/expiry dates, current flags, audit columns.
• dbt snapshots and incremental models withmaterializations (table/incremental).
• Optimize jobs: partitioning,Z-ordering/bucketing, broadcast joins, predicate pushdown, and shuffletuning.
• Enforce coding standards: modular Pythonpackages, parameterized notebooks/jobs, dbt macros and packages.
• Design dimensional models (star/snowflake) withconformed dimensions and KPI-aligned fact tables.
• Build dbt semantic layers with tests (unique/notnull/relationships), docs, exposures, and data contracts.
• Manage schema evolution and late-arriving data;ensure reproducibility with versioned models.
• Apply RBAC in data platforms; manage secrets viaKey Vault/Secrets Manager and environment variables.
• Implement encryption in transit/at rest, datamasking, and row/column-level policies in target systems.
• Maintain lineage and catalog (dbt docs/manifest;Purview/Collibra/Atrium where available); steward metadata and tagging.
• Profile PySpark jobs (stage/task metrics); fixdata skew, excessive shuffles, small-file problems; tune cluster configs.
• Tune dbt models (query plans,indices/distributions, materializations, incremental strategies).
• Optimize Kafka workloads: topic partitioning,consumer group balancing, backpressure, batch size/linger.ms tuning.
• Version control with Git; implement CI/CD forPython packages, PySpark jobs, dbt projects, and Kafka artifacts(schemas/config).
• Automate quality gates: unit tests, data tests(dbt), schema compatibility checks, linting, and release notes.
• Implement observability: job/app telemetry,logs, metrics dashboards for SLAs/SLOs, freshness, and latency (batch +streaming).
• Set alerts for failures/anomalies; conduct RCAand remediation plans; track data downtime.
• Optimize costs: autoscaling/serverless (whereapplicable), warehouse/cluster rightsizing, storage lifecycle, usagetagging/reporting.

Other duties:

• Lead solution walkthroughs and trade-offanalysis (PySpark vs. dbt, streaming vs. batch, Delta vs. warehouse).
• Build POCs (CDC with Debezium, SCD2 withDelta/dbt snapshots, Kafka throughput/latency benchmarks) to de-riskdelivery.
• Manage risks/assumptions, change requests, andstakeholder communications; maintain decisions & actions log.
• Contribute to proposals, estimations, and SOWsfor Python/PySpark/dbt/Kafka engagements.
• Prepare executive reports on pipeline health,governance posture, streaming reliability, and cost optimization.
• Support integrations with upstream/downstreamplatforms (APIs, event streams, BI tools, warehouses/lakehouses).
• Conduct internal knowledge-sharing; buildaccelerators (dbt macros, PySpark templates, Kafka schema standards); mentorassociates.

Qualifications:

• Bachelor’s degree in Computer Science,Information Technology, or a related field; or equivalent professionalexperience.
• Experience: 4+ years in data engineering/BIconsulting with strong hands-on work in Python, PySpark, dbt, and Kafka.
• PySpark: DataFrame APIs, Spark SQL,performance tuning (partitions, caching, joins, shuffle), structuredstreaming.
• dbt: sources/staging/marts,materializations (table/incremental), snapshots, tests, docs; Jinja/macros;packages.
• Kafka: producers/consumers,topics/partitions/replication, Schema Registry (Avro/JSON/Protobuf),Streams/Structured Streaming, offset management, DLQs.
• CDC/SCD: incremental loads, watermarks,SCD1/SCD2 via Delta MERGE or dbt snapshots/incremental patterns; Debezium/CDCconnectors (nice-to-have).
• Modelling: star/snowflake, conformeddimensions, semantic consistency.
• Storage/Compute: Lakehouse/warehouse(Delta/Parquet; Snowflake/Synapse/BigQuery/Databricks—any preferred).
• Security/Governance: RBAC, secrets,encryption, masking/row policies; catalogue/lineage basics.
• Hands-on experience in data modeling, indexing,and query performance tuning.
• Experience with backup, disaster recovery, andbusiness continuity planning.
• Strong understanding of cloud security risks,including data breaches, malware, and social engineering.
• Excellent collaboration skills for working withcross-functional teams and providing leadership in cloud security projects.
• Experience working in a team, able to lead,prioritize and execute tasks in a high-pressure environment and provide or makesound decisions in emergencies.