Lead Data Engineer

Diverse Lynx; Dublin (OH); Remote; USD 100,000 - 130,000; Full time

Job summary

A leading company in data services seeks a Lead Data Engineer to design scalable data solutions and mentor junior data engineers. The ideal candidate will possess significant experience in AWS, SQL, and big data technologies, with a passion for innovative data solutions. This role promises to enhance your skills in a collaborative environment while contributing to high-impact projects.

Qualifications

• 5+ years of hands-on experience in AWS and SQL.
• 3-4 years of experience working with Hadoop stack.
• Strong analytical and problem-solving abilities.

Responsibilities

• Design and build reusable components for analytics products.
• Collaborate on new data management projects and architecture.
• Lead and mentor a team of data engineers.

Skills

AWS

SQL

Python

Spark

Pyspark

Education

Bachelor's degree in IT or related field

Tools

Apache Airflow

Jenkins

Docker

Kubernetes

GitHub

Snowflake

Job description

Job Description
Role: Lead Data Engineer
Duration: 11 Months
Location: Dublin OH (Remote till further notice)

Skill set: Hands on exp on :Spark , Python , SQL , AWS, Pyspark, DW concepts

Responsibilities:
(The primary tasks, functions and deliverables of the role)
Design and build reusable components, frameworks and libraries at scale to support analytics products
Design and implement product features in collaboration with business and Technology stakeholders
Identify and solve issues concerning data management to improve data quality
Clean, prepare and optimize data for ingestion and consumption
Collaborate on the implementation of new data management projects and re-structure of the current data architecture
Implement automated workflows and routines using workflow scheduling tools
Build continuous integration, test-driven development and production deployment frameworks
Collaboratively review design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards
Analyze and profile data for designing scalable solutions
Troubleshoot data issues and perform root cause analysis to proactively resolve product and operational issues
Lead and mentor team of data engineers
Trouble shoot key issues and ensure team member performance

Requirements:

Experience:
Strong understanding of data structures and algorithms
Strong understanding of solution and technical design
Has a strong problem solving and analytical mindset?
Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders
Able to quickly pick up new programming languages, technologies, and frameworks
Advanced experience building cloud scalable, real time and high-performance data lake solutions
In-depth understanding of micro service architecture
Strong understanding of developing complex data solutions
Experience working on end-to-end solution design
Able to lead others in solving complex problems by taking a broad perspective to identify innovative solutions
Willing to learn new skills and technologies
Has a passion for data solutions

Required and Preferred Skill Sets:
3+ years of hands-on expert level experience in AWS - EMR [Hive, Pyspark], S3, Athena or any other equivalent cloud
3+ years of hands-on expert level experience Spark Batch Processing and some familiarity with Spark Structured Streaming
3-4 years experience Working experience with Hadoop stack dealing huge volumes of data in a scalable fashion
5+ years of hands-on experience with SQL, ETL, data transformation and analytics functions
5+ years of hands-on Python experience including Batch scripting, data manipulation, distributable packages
5+ years experience working with batch orchestration tools such as Apache Airflow or equivalent
5+ years working with code versioning tools such as GitHub or BitBucket; expert level understanding of repo design and best practices
5+ years working with deployment automation tools such as Jenkins and familiarity with containerization concepts such as Docker and Kubernetes
5+ years of hands-on experience designing and building ETL pipelines; expert with data ingest, change data capture, data quality; hand on experience with API development; some exposure to Nifi or Kafka
5+ years designing and developing relational database objects; knowledgeable on logical and physical data modelling concepts; some experience with Snowflake
3+ years of experience supporting Tableau or Cognos use cases; familiarity with tools and capabilities
Familiarity with Agile; working experience preferred


Education:
Bachelor's degree in IT or related field
One of the following alternatives/optional may be accepted:
o PhD or Law + 3 yrs;
o Masters + 4 yrs;
o Associates + 6 yrs;







Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.

About the company

We provide Consulting Services for our customer’s project and staffing needs. We also provide Data Management for Clinical Trials.

Notice

Talentify is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.

Talentify provides reasonable accommodations to qualified applicants with disabilities, including disabled veterans. Request assistance at [e-mail] or 407-000-0000.

Federal law requires every new hire to complete Form I-9 and present proof of identity and U.S. work eligibility.

An Automated Employment Decision Tool (AEDT) will score your job-related skills and responses. Bias-audit & data-use details: www.talentify.io/bias-audit-report . NYC applicants may request an alternative process or accommodation at [e-mail] or 407-000-0000.