Responsibilities
- Advanced SQL and Python for ingestion, transformation, validation, reconciliation, and performance optimization.
- Hands-on GCP experience with BigQuery, Dataflow / Apache Beam, Pub/Sub, Cloud Storage, and Cloud Composer / Airflow.
- Strong Apache Spark and distributed data-processing experience, including partitioning, skew handling, incremental loads, and tuning.
- Data modeling for customer, identity, commerce, campaign, audience, and measurement datasets.
- Experience with data contracts, schema evolution, metadata, lineage, automated data-quality controls, and reconciliation.
- CI/CD using GitHub / GitHub Actions or equivalent, automated tests, and infrastructure-as-code.
- Pipeline observability covering freshness, completeness, accuracy, latency, volume, failures, dashboards, and alerts.
- Secure data engineering using encryption, tokenization / hashing, role-based access, secrets management, and retention controls.
- Working knowledge of identity resolution, identity graphs, customer 360, consent-aware processing, and audience activation.
- Retail media, advertising, marketing technology, or campaign-measurement experience is preferred.
- Design, build, and operate scalable batch and near-real-time pipelines for customer, transaction, product, digital interaction, campaign, and partner data.
- Develop curated data products supporting identity resolution, audience segmentation, activation, analytics, attribution, and measurement.
- Partner with architects, product managers, analysts, data scientists, privacy teams, and application engineers to define data contracts, SLAs, and acceptance criteria.
- Implement controls for schema, uniqueness, referential integrity, completeness, freshness, validity, and source reconciliation.
- Optimize BigQuery, Spark, and pipeline workloads for throughput, latency, reliability, and cost.
- Build unit, integration, regression, and production-validation tests and embed quality gates in CI/CD.
- Establish monitoring, alerting, dashboards, runbooks, incident triage, and permanent corrective actions.
- Protect sensitive identifiers using approved access, encryption, tokenization / hashing, retention, auditability, and least-privilege design.
- Perform code reviews, maintain technical and lineage documentation, and contribute to reusable engineering standards.
- Support releases, backfills, migrations, performance tuning, and controlled recovery from pipeline failures.
Generic Managerial Skills, If any
- Own delivery from design through production stabilization and communicate risks, dependencies, and status clearly.
- Collaborate effectively across onsite and offshore teams in an Agile model.
- Mentor engineers through design reviews, code reviews, and troubleshooting.
- Demonstrate strong problem-solving, stakeholder communication, documentation, and prioritization.
Education
Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, Engineering, or related discipline