Senior Lead Software Engineer – Data Engineering

Clarivate; United States; Hybrid; USD 100,000 - 125,000; Full time

Job summary

Join a forward-thinking company as a Senior Lead Data Engineer, where you will leverage your extensive experience in pySpark, Python, and Java to lead the development of scalable data processing solutions. This innovative role involves collaborating with business and IT teams to design a robust data lake platform, implementing best practices for Big Data management, and ensuring data quality in ETL processes. With a hybrid work model and a supportive environment, you'll thrive in a team that values continuous improvement and technological advancement. If you're passionate about data and technology, this opportunity is perfect for you.

Qualifications

• 7+ years of software development experience.
• Expertise in building scalable data processing solutions.
• Strong background in database design and development.

Responsibilities

• Provide technical leadership and evaluate technologies for business needs.
• Collaborate with teams to design and deliver a data lake platform.
• Ensure solutions meet SLAs and are scalable and reliable.

Skills

pySpark

Python

Java

PostgreSQL

ElasticSearch

AWS Glue

Apache Spark

Cassandra

Hadoop

Apache Hive

Snowflake

Jupyter Notebook

Databricks

Education

Bachelor's degree in Computer Science or related field

Tools

AWS

Agile Scrum

Kanban

Automated database migration tools

Job description

We are currently seeking an experienced Senior Lead Data Engineer with demonstrated expertise in pySpark, Python, and Java technologies to join our global Technology team in India. If you are skilled in Query engines such as Postgres/SQL/AWS/Java/Kotlin, we would love to connect with you.

About You – experience, education, skills, and accomplishments:

1. At least 7+ years of experience in software development
2. At least 4+ years of experience in building massively scalable distributed data processing solutions
3. At least 6+ years of experience in database design and development
4. At least 4 years of experience with Apache Spark and ElasticSearch
5. Experience with Spark, ElasticSearch, Cassandra, Hadoop, Apache Hive, Snowflake, Jupyter Notebook, Databricks stack
6. At least 4 years of experience with PostgreSQL or Oracle DB (Oracle 11g+), PostgresSQL 9+, AWS RDS
7. Experience with AWS technologies such as AWS Glue and Lambda
8. Experience in Master Data Management, designing CMSes, or evaluating third-party CMS products
9. Familiarity with Agile Scrum or Kanban methodologies and deploying solutions using Continuous Delivery practices
10. Proficiency with automated database migration tools and strong opinions on version control best practices for SQL scripts

Preferred additional skills:

1. Experience in Master Data Management and CMS design or evaluation
2. Experience working in Agile Scrum or Kanban teams with Continuous Delivery
3. Use of automated database migration tools and version control for SQL scripts

Role responsibilities:

1. Provide technical leadership and evaluate different technologies to meet business needs and cost efficiencies
2. Collaborate with Business and IT teams to design and deliver a data lake platform
3. Develop and maintain the overall solution design for the Data Lake Platform
4. Implement data strategies, assisting in solution design and architecture
5. Establish best practices for Big Data management, including software selection, architecture, and implementation
6. Create documentation, guidelines, and training materials
7. Ensure solutions meet SLAs and are scalable and reliable
8. Contribute to continuous improvement by staying updated on technological advancements
9. Focus on data quality in ETL processes, data pipelines, storage, and security

About the team:

The team is distributed across India, with strong skills in PySpark, AWS Glue, and Databricks, among other tools and technologies.

Work hours and environment:

This is a permanent role based on IST hours, with a hybrid work model, flexible schedule, and a supportive work environment.

We are committed to equal employment opportunities and adhere to all applicable non-discrimination laws.

About Clarivate

Founded in 2016 with offices in Philadelphia and London, Clarivate was formed following the acquisition of Thomson Reuters' Intellectual Property and Science Business by Onex Corporation and Baring Private Equity Asia.