Staff Data Platform Engineer – Real-Time Ingestion

News Corp; Austin (TX); On-site; USD 150,000 - 210,000; Full time

Job summary

Realtor.com in Austin, TX, part of News Corp, seeks a hands-on Staff Software Engineer to advance the Data Ingestion Platform. You will design, build, and operate scalable ingestion, streaming, replication, orchestration, and transformation capabilities used by many teams.

You will own platform APIs, libraries, and reference implementations, work with Kafka/MSK, Airflow, dbt, and Snowflake, and improve reliability, developer experience, and security across the data platform.

Responsibilities

• Design, build, and operate scalable ingestion, streaming, replication, orchestration, and transformation capabilities used by many teams.
• Develop and improve platform APIs, services, libraries, templates, and paved paths for streaming, batch, bulk, and external data ingestion.
• Build and maintain ingestion APIs and client libraries in TypeScript, Kotlin, and Python, supporting streaming, batch, bulk, and external data-ingestion use cases.
• Operate and evolve Kafka/MSK infrastructure, including topics, partitions, replication, retention, throughput, consumer groups, access controls, and upgrades.
• Support Kafka Connect and sink integrations for systems such as Snowflake, S3, New Relic, and Elasticsearch; improve connector reliability, provisioning, and troubleshooting.
• Build and maintain Schema Registry and Avro-based data contract workflows, including schema validation, compatibility, versioning, and safe evolution.
• Build and maintain KSQL or equivalent near-real-time transformation patterns where appropriate.
• Improve Airflow/MWAA orchestration patterns, DAG templates, scheduling, backfills, retries, dependency management, and operational reliability.
• Improve dbt platform capabilities, including project conventions, macros, testing, materializations, restricted-data patterns, and Airflow integration.
• Develop reusable TypeScript, Python, and/or Kotlin libraries and reference implementations for producing, consuming, validating, and moving data.
• Contribute to Snowflake integration patterns, data exports, toolsets, and secure access paths without taking ownership of stakeholder-specific data modeling.
• Take pride in ensuring delivery, quality, availability, low latency, cost efficiency, and operational reliability for our platform capabilities.
• Instrument systems with actionable metrics, logs, traces, alerts, runbooks, and service-level objectives.
• Diagnose data issues across the full path from source to serialization, topic, connector, and Snowflake destination.
• Participate in incident response, root-cause analysis, post-incident improvements, and production support rotations.
• Apply least-privilege access, environment separation, secrets management, auditability, and secure-by-default platform design.
• Improve platform health monitoring and standardize operational workflows across Fivetran, Kafka/MSK, Airflow, dbt, connectors, and Snowflake.
• Balance reliability, delivery speed, developer experience, operational simplicity, and cost when making architecture decisions.
• Create clear onboarding paths, templates, examples, documentation, and diagnostics that help teams self-serve.
• Partner closely with internal users to understand their ingestion and transformation needs and guide them to the appropriate platform path.
• Improve discoverability, lineage, data contracts, and metadata through integration with the data catalog.
• Clearly communicate technical requirements, tradeoffs, operating boundaries, and migration plans to engineers and stakeholders.
• Inspire and mentor fellow engineers, raising the technical bar through constructive design reviews, code reviews, pairing, and knowledge sharing.
• Create practical agentic workflows that automate repetitive platform and developer-support work, such as onboarding, schema validation, DAG and dbt scaffolding, connector checks, pipeline diagnostics, incident triage, and documentation updates.
• Build reusable AI skills, tools, and workflows that interact with platform APIs, repositories, tickets, observability systems, and documentation while preserving appropriate access controls.
• Design grounded AI experiences that use authoritative platform knowledge, return structured findings, cite or link to source documentation, and elevate safely when confidence is low.
• Develop agents that can reason across pipeline boundaries, correlate evidence, identify likely failure points, and produce concise remediation guidance or actionable tickets.
• Add automated checks and evaluation harnesses for agent behavior, including correctness, safety, permissions, tool-use reliability, hallucination resistance, and regression testing.
• Use AI to improve engineering throughput without weakening review, change-management, production-safety, or data-governance controls.
• Contribute reusable patterns for prompt/context design, tool calling, workflow orchestration, human approval, audit logging, and continuous improvement of AI-enabled platform operations.

Job description

Realtor.com in Austin, TX, part of News Corp, seeks a hands-on Staff Software Engineer to advance the Data Ingestion Platform. You will design, build, and operate scalable ingestion, streaming, replication, orchestration, and transformation capabilities used by many teams.

You will own platform APIs, libraries, and reference implementations, work with Kafka/MSK, Airflow, dbt, and Snowflake, and improve reliability, developer experience, and security across the data platform.