Sr. Data Engineer

Beacon Hill Staffing Group, LLC; Greenwood Village (CO); On-site; USD 120,000 - 170,000; Full time

Benefits offered by this job

Medical, dental, vision benefits

Paid time off

Job summary

Beacon Hill Technologies seeks a Sr Data Engineer to design, build, and maintain scalable ETL pipelines using Spark on AWS. You will ensure data availability and quality for analytics across enterprise networks.

You will lead data integration efforts, collaborate with BI and analytics teams, and implement monitoring, CI/CD for data applications. A Bachelor's degree and strong SQL, Spark, Python are expected.

Qualifications

• Must have strong English communication skills.
• Expertise in data engineering languages (Scala preferred) or Java with Python.
• Experience with Spark and Hadoop ecosystems for data processing.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines.
• Implement monitoring and alerting for data pipelines.
• Develop deployment pipelines for CI/CD of data solutions.
• Manage and optimize data storage across distributed file systems and databases.
• Collaborate with BI/analytics teams and stakeholders.

Skills

Advanced SQL

ETL

Scala

Java

Python

Spark

Hadoop

AWS

S3

Lambda

Redshift

EMR

GenAI

Education

Bachelor's degree in CS/Math/Stats/Engineering

Tools

SQL

BigData tools

Job description

Beacon Hill was founded to set a new standard in search, career placement and flexible staffing.

Requirements

Top 3 Requirements

• Advanced SQL skills (ETL)
• Expertise in data engineering languages such as Scala (preferred) or Java, with proficiency in Python
• Experience with BigData tools, particularly Spark, and familiarity with Hadoop ecosystems
• Experience with cloud platforms like AWS, including services such as S3, Lambda, Redshift, and EMR
• Plusses: Understanding of AI technologies and their application (kiro, claude, genAI tools)

Job Description

As part of the Network Analytics team, the Sr Data Engineer role is crucial in building and maintaining the data infrastructure that supports the clients enterprise networks. This position focuses on constructing robust ETL pipelines using Spark to ensure data availability and quality for analytics tasks. The role requires expertise in data engineering, leveraging AWS for scalable solutions and enhancing operational efficiency. Additionally, the position demands the ability to prioritize efforts, synthesize actionable insights, and effectively communicate findings to audiences with diverse agendas and varying levels of technical expertise.

Major Duties and Responsibilities

• Design, develop, and maintain scalable ETL pipelines to ensure data quality and availability
• Implement monitoring and alerting solutions to ensure data pipeline reliability and performance
• Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions
• Manage and optimize data storage solutions, including: Distributed file systems, Relational databases, Flat files, External sources accessed via API
• Implement data integration solutions to support analytics and reporting needs
• Collaborate across teams to ensure data solutions align with existing production architectures and business requirements
• Provide technical leadership and guidance on data engineering best practices and methodologies
• Document and communicate data engineering processes and standards to business intelligence, data, and analytics professionals with varied backgrounds
• Continuously evaluate and improve data engineering tools and approaches to enhance performance and efficiency
• Perform other duties as required

REQUIRED QUALIFICATIONS

Skills/Abilities and Knowledge

• Ability to read, write, speak, and understand English
• Strong communication and collaboration skills
• Expertise in data engineering languages such as Scala (preferred) or Java, with proficiency in Python
• Experience with BigData tools, particularly Spark, and familiarity with Hadoop ecosystems
• Proficiency in building and managing ETL pipelines
• Experience with cloud platforms like AWS, including services such as S3, Lambda, Redshift, and EMR
• Strong understanding of relational databases and SQL, and familiarity with NoSQL databases
• Knowledge of data architecture, data warehousing, and data marts
• Experience in implementing monitoring solutions and deployment pipelines for data applications
• Ability to work with distributed systems and manage data storage solutions
• Demonstrated ability and desire to continually expand skill set, and learn from and teach others

PREFERRED QUALIFICATIONS

• Experience with Linux-based operating systems
• Knowledge of other relevant techniques such as text analysis and text mining

Education

Bachelors degree (or relevant experience) in computer science, mathematics, statistics, physics, operations research, or other quantitatively-focused fields.

California residents: Qualified applications with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.

Company Profile

Beacon Hill Technologies, a premier National Information Technology Staffing Group, provides world class technology talent across all industries utilizing a complete suite of staffing services. Beacon Hill Technologies' dedicated team of recruiting and staffing experts consistently delivers quality IT professionals to solve our customers' technical and business needs. Beacon Hill Technologies covers a broad spectrum of IT positions, including Project Management and Business Analysis, Programming/Development, Database, Infrastructure, Quality Assurance, Production/Support and ERP roles.

Learn more about Beacon Hill and our specialty divisions, Beacon Hill Associates, Beacon Hill Financial, Beacon Hill HR, Beacon Hill Legal, Beacon Hill Life Sciences and Beacon Hill Technologies by visiting www.bhsg.com.

Benefits Information

• Beacon Hill offers a robust benefit package including, but not limited to, medical, dental, vision, and federal and state leave programs as required by applicable agency regulations to those that meet eligibility.
• Upon successfully being hired, details will be provided related to our benefit offerings.

We look forward to working with you.Beacon Hill.

Employing the Future