Senior Software Engineer - Data Engineering | Skillman, NJ, USA

Bloomberg; Montgomery Township (NJ); On-site; USD 120,000 - 180,000; Full time

Job summary

An established industry player is seeking a seasoned Data Engineer to enhance their data management and analytics capabilities. This role involves designing and developing complex data engineering solutions, building robust ETL frameworks, and leveraging cutting-edge technologies like Pyspark and Hadoop. You'll work collaboratively with diverse teams to drive data analytics initiatives, ensuring data quality and governance. If you are passionate about data and innovation, this is a fantastic opportunity to make a significant impact in a dynamic environment.

Qualifications

• 7+ years of experience in data engineering solutions.
• 5+ years of experience with ETL frameworks using Python and Hadoop.
• Strong understanding of data warehousing methodologies.

Responsibilities

• Drive the vision for Data Analytics across business areas.
• Build scalable data engineering applications for self-service analytics.
• Collaborate with cross-functional teams to understand business needs.

Skills

Data Engineering Solutions

ETL Frameworks

Python/Pyspark

Hadoop/Hive

Real-time Streaming Frameworks

SQL Proficiency

Data Modeling

Communication Skills

Education

BA, BS, MS, or PhD in Computer Science

Engineering or related technology field

Tools

Pyspark

Hadoop

Airflow

Kafka

Flink

Informatica/IDMC

QlikSense

Tableau

PowerBI

Job description

Our Team:
Bloomberg runs on data. As the Data Management & Analytics team within Engineering, we support our organization's needs around managing data efficiently and enabling everyone across the company to make informed decisions using self-service data analytics tools. We are responsible for ingesting and preparing massive amounts of data and ensuring that the data is properly modeled, governed, accurate, and protected for reporting and advanced analytics. A key objective of this role is to help build and support company-wide data analytics programs leveraging data engineering technologies such as Hadoop, Pyspark, Flink, Airflow, and traditional MPP Data Warehouse/ETL technologies.

We'll trust you to:

• Proactively drive the vision for Data Analytics across multiple business areas, and define and execute on a plan to achieve that vision.
• Build high-quality, scalable, and robust data engineering applications with the goal of providing end users with self-service analytics.
• Be proficient at developing, architecting, standardizing, and supporting technology platforms using open source technologies such as Pyspark, Airflow, and industry-leading ETL solutions.
• Build data pipelines by ingesting petabytes of data using real-time streaming tools like Kafka and Flink. Maintain and support these data pipelines and ensure they are scalable and highly available.
• Incorporate exceptional handling, logging, and auditing procedures within the data pipelines codebase.
• Leverage open source frameworks around workflow automation, orchestration, and job scheduling.
• Integrate data management capabilities such as data modeling, data quality, and data anonymization throughout the data lifecycle — data in transit and data at rest.
• Collaborate and build cross-functional relationships with Product Owners, Data Scientists, and Software Engineers to understand business needs and deliver on those.
• Embrace an agile framework with iterative product development and a continuous improvement mindset.
• Stay up to date with market trends, bring new ideas onboard, and evaluate tooling for future needs.
• Be highly motivated to drive innovations across engineering.

You'll need to have:

• 7+ years of experience designing and developing complex data engineering solutions.
• 5+ years of experience building ETL frameworks using Python/Pyspark and Hadoop/Hive.
• 2+ years of experience with industry-standard orchestration tools, e.g., Tidal, Airflow, Autosys.
• 3+ years of experience in distributed parallel computing databases for Data Warehousing needs.
• 3+ years of experience using real-time streaming frameworks such as Kafka and Flink.
• Strong understanding of data warehousing methodologies, ETL processing, and dimensional data modeling.
• Advanced SQL proficiency and data modeling experience. Knowledge of database design techniques.
• Demonstrated experience working with business users to gather requirements and manage scope.
• Strong communication, presentation, problem-solving, and troubleshooting skills.
• BA, BS, MS, or PhD in Computer Science, Engineering, or a related technology field.

We'd love to see:

• Experience working with extremely large data volumes, large databases, and data warehouse implementations.
• Understanding of VLDB performance aspects, such as table partitioning, sharding, table distribution, and optimization techniques.
• Prior working experience with ETL tools such as Informatica/IDMC.
• Knowledge of reporting tools such as QlikSense, Tableau, PowerBI.

What is Bloomberg? Our teams unleash the power of information and technology to organize, understand, and improve our world. Our 325,000+ global customers rely on us to provide innovative solutions that make a difference in their lives and businesses.