Senior Engineer, Data Engineering

News Corporation; Bengaluru; On-site; INR 2,600,000 - 4,800,000; Full time

Job summary

News Corp in Bengaluru seeks a Lead Data Engineer to consolidate data across categories and countries into a unified customer view for engagement, measurement and targeting. You will partner with product, data and engineering teams to make data accessible for revenue growth and improved customer experiences.

You will design and maintain scalable data pipelines, integrate upstream systems, and mentor a team of engineers while driving best practices across the data platform.

Qualifications

• BS in Computer Science or other technical discipline.
• 8+ years designing and developing big data processing systems using distributed computing.
• 6+ years experience in SQL.
• 5+ years experience in Python and Spark.
• 6+ years of experience working with cloud platforms such as AWS.
• 4+ years experience in Apache Airow.
• Expert knowledge in optimizing complex SQL queries and Hands on experience with NoSQL (DynamoDB).
• An affinity for automation.
• Familiarity with networking and REST APIs.
• Experience with at least one object oriented language (e.g. Java).
• Strong OO design, data structure, and algorithm design skills.
• Strong interest in emerging technologies: Spark, EKS.

Responsibilities

• Design, implement and maintain high performance big data infrastructure/systems & big data processing pipelines scaling to billions of events daily.
• Design, implement and maintain deep integration with up-stream systems.
• Implement data pipelines to automate ingestion, transformation, and augmentation of data sources; provide best practices for pipeline operations.
• Advanced SQL writing and experience in data mining and using databases with complex datasets.
• Monitor and optimize the data platform for reliability, scalability, and efficiency.
• Ensure data quality across systems with transparency and proactive monitoring.
• Support products with the roadmap and provide technically accurate senior leadership updates.
• Translate business requirements into logical data pipelines and processes.
• Research and prototype new data processing technologies when needed.
• Mentor and guide a team of data engineers and promote best practices.

Skills

SQL

Python

Spark

Java

Data modeling

Big data processing

OO design

Data analysis

Automation

Education

BS in Computer Science

Tools

AWS

Hadoop/Spark stack

Airow

NoSQL DynamoDB

REST APIs

Job description

Job Description

Lead Data Engineer One of the most innovative and high-prole teams in News Corp is looking for a lead data engineer to help accelerate its vision. The role will involve combining all of News Corp’s data across categories, countries and organizations into a singular view of a customer enabling engagement, measurement and targeting. The Dow Jones Lead Data Engineer will partner closely with product, data & engineering teams to help make this data available and accessible to our businesses, teams and partners in order to drive revenue and improve the customer experience.

Responsibilities

• Design, implement and maintain high performance big data infrastructure/systems & big data processing pipelines scaling to billions of structured and unstructured events daily
• Design, implement and maintain deep integration with up-stream systems
• Implement data pipelines to automate the ingestion, transformation, and augmentation of data sources, and provide best practices for pipeline operations.
• Advanced SQL writing and experience in data mining (SQL, ETL, data warehouse, etc.) and using databases in a business environment with complex datasets
• Monitor and optimize the performance of the data platform to ensure reliability, scalability, and efiency.
• Ensure data quality across systems (accuracy, consistency, completeness) by providing transparency and proactive monitoring.
• Support products with the overall roadmap and ensure updates to senior leadership are 100% technically correct.
• Data analysis, understanding of business requirements and translation into logical pipelines & processes
• Conduct timely and effective research in response to specific requests (e.g. data collection, summarization, analysis, and synthesis of relevant data and information)
• Evaluate and prototype new technologies in the area of data processing
• Think quickly, communicate clearly and work collaboratively with product, data, engineering, QA and operations teams
• Data Engineering & Data Modelling: Creating data architectures and logical/physical data models for databases and data warehouses.
• Providing training on data engineering concepts and techniques to other members of the team
• Writing documentation for data pipelines and data models
• Mentor and guide a team of data engineers, fostering a culture of continuous improvement, innovation, and knowledge sharing to enhance technical expertise and collaboration across the team.
• Promote a high-performance team environment, ensuring strong work ethic, accountability, and enthusiasm to drive successful project delivery.

Technologies we use

• AWS (EMR, EC2, Glue, Athena, Redshift, Lambda, S3, DynamoDB, SNS)
• Python, Spark (PySpark)
• Kubernetes, Docker
• Airow
• Git for source code management
• Jira

Qualifications

• BS in Computer Science or other technical discipline
• 8+ years of experience designing and developing big data processing systems using distributed computing
• 6+ years experience in SQL
• 5+ years experience in Python and Spark
• 6+ years of experience working with cloud platforms such as AWS
• 4+ years experience in Apache Airow
• Expert knowledge in optimizing complex SQL queries and Hands on experience with NoSQL (DynamoDB)
• An affinity for automation
• Familiarity with networking and network application programming, including HTTP/HTTPS, JSON, and REST APIs
• Experience with at least one object oriented language (ex: Java)
• Strong OO design, data structure, and algorithm design skills
• Strong interest in emerging technologies: Spark, EKS

Nice to Have

• Experience in the digital advertising and marketing industry
• Contribution to Open Source projects

Equal Opportunity Employer

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, national origin, protected veteran status, disability status or any other protected characteristic.

Reasonable Accommodation

We are committed to providing reasonable accommodation for qualified individuals with disabilities in our job application and/or interview process. If you need assistance or accommodation in completing your application or participating in an interview due to a disability, email us at [e-mail] . Please put "Reasonable Accommodation" in the subject line and provide a brief description of the type of assistance you need. This inbox will not be monitored for application status updates.

Please refer to the privacy notice at the bottom of this page for submitting any data access, deletion, or other data subject rights requests, where permitted under your local laws and regulations.

About Us

News Corp (Nasdaq: NWS, NWSA; ASX: NWS, NWSLV) is a global, diversified media and information services company focused on creating and distributing authoritative and engaging content and other products and services. The company comprises businesses across a range of media, including: information services and news, digital real estate services and book publishing. Headquartered in New York, News Corp operates primarily in the United States, Australia and the United Kingdom, and its content and other products and services are distributed and consumed worldwide. More information is available at: http://newscorp.com.

If you are a current employee in Workday, please navigate to the Jobs Hub and select Browse Jobs to apply internally.