Data Engineer - Databricks

Open Systems Technologies; New York (NY); Remote; USD 80,000 - 100,000; Full time

Job summary

An innovative financial firm is seeking a skilled Data Engineer specializing in Databricks to join their remote team. This role offers the opportunity to design and build large-scale solutions, focusing on cloud databases and data pipelines. The ideal candidate will have extensive experience in SQL, data modeling, and cloud technologies, along with a strong understanding of data governance and warehousing concepts. Join a forward-thinking organization that values expertise and creativity, and contribute to impactful projects that shape the future of data engineering in finance.

Qualifications

• 7+ years of experience in designing and building large-scale solutions.
• Expertise in building Cloud databases and managing data pipelines.

Responsibilities

• Design and develop data pipelines with a focus on security, scale, and reliability.
• Build and manage Cloud databases in Databricks Lakehouse.

Skills

Cloud Database Management

SQL Expertise

Data Modeling

Scripting Languages (Python, Bash)

Data Warehousing Concepts

Agile Methodologies

Data Governance

Education

Bachelor's Degree in Computer Science

Bachelor's Degree in Management Information Systems

Bachelor's Degree in Computer Engineering

Tools

Databricks

AWS RDS

DynamoDB

Talend

Informatica

SparkSQL

Hadoop

Kafka

Redshift

BigQuery

Job description

A financial firm is looking for a Data Engineer - Databricks to join their team. This role is remote.

Pay: $100-130/hr

US Citizens or GC Holders Only
No c2c - W2 Only

Qualifications:

1. A Bachelor's Degree in Computer Science, Management Information Systems, Computer Engineering, or related field.
2. 7+ years of experience in designing and building large-scale solutions in an enterprise setting.
3. 3+ years designing and building solutions in the cloud.
4. Expertise in building and managing Cloud databases such as AWS RDS, DynamoDB, DocumentDB or analogous architectures.
5. Expertise in building Cloud Database Management Systems in Databricks Lakehouse or analogous architectures.
6. Deep SQL expertise, data modeling, and experience with data governance in relational databases.
7. Experience with the practical application of data warehousing concepts, methodologies, and frameworks using traditional (Vertica, Teradata, etc.) and current (SparkSQL, Hadoop, Kafka) distributed technologies.
8. Refined skills using one or more scripting languages (e.g., Python, bash, etc.).
9. Embrace data platform thinking, design and develop data pipelines keeping security, scale, uptime and reliability in mind.
10. Expertise in relational and dimensional data modeling.
11. UNIX admin and general server administration experience.
12. Experience with leveraging CI/CD pipelines.
13. Presto, Hive, SparkSQL, Cassandra, or Solr other Big Data query and transformation experience is a plus.
14. Experience using Spark, Kafka, Hadoop, or similar distributed data technologies is a plus.
15. Experience with Agile methodologies and able to work in an Agile manner is preferred.
16. Experience using ETL/ELT tools and technologies such as Talend, Informatica is a plus.
17. Expertise in Cloud Data Warehouses in Redshift, BigQuery or analogous architectures is a plus.