Senior Databricks Data Engineer – Remote & AWS Spark Expert

Eliassen Group; Columbus (OH); Remote; USD 150,000 - 154,000; Full time

Benefits offered by this job

Medical plans

Dental coverage

Vision coverage

Pre-tax accounts

Life and disability insurance

401(k) with match

Sick time where required by law

Job summary

Eliassen Group is seeking a Databricks Data Engineer for a remote, permanent role to design and maintain scalable data pipelines, data lakes, and analytics-ready infrastructure. You will collaborate with ML engineers, data scientists, and analysts to deliver data products and ensure data quality and security.

The ideal candidate has 5+ years in data engineering, strong Python/SQL/Datatbricks/Spark skills, and experience with AWS.

Qualifications

• Five or more years of data engineering experience.
• Expert proficiency in Python, SQL, Databricks, and Spark.
• Experience building cloud-based data pipelines in AWS.
• Knowledge of data governance, security, privacy, and retention practices.
• Strong communication, collaboration, and problem-solving skills.
• Experience working in agile environments using Jira.
• Proficiency with Git, Docker, and CI tools such as Jenkins.
• Passion for data and continuous improvement.
• Preferred: Ability to work independently and proactively.
• Preferred: Experience optimizing ETL workflows in Databricks.
• Preferred: Knowledge of data lineage and data quality practices.
• Preferred: Experience supporting data science and machine learning solutions.
• Preferred: Prior experience in energy or utility industry.

Responsibilities

• Design, develop, and maintain data pipelines, infrastructure, and systems using AWS, Python, Databricks, Spark, and PostgreSQL.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.
• Provide technical guidance and mentorship to junior data engineers.
• Develop and implement data engineering standards and best practices.
• Monitor and troubleshoot data pipelines to ensure data quality, integrity, and availability.
• Research industry trends and evaluate new data technologies.
• Build and maintain data lakes to support analytics and business intelligence.
• Manage the software development life cycle and DevOps activities.
• Partner with data scientists and analysts to deliver effective data solutions.
• Develop data validation and data analysis tools.

Skills

Python

SQL

Databricks

Spark

AWS

Data governance

Jira

Git

CI/CD

Communication

Education

Bachelor's degree in CS/IT

Master's degree in STEM

Tools

Databricks

PostgreSQL

Jira

Git

Docker

Jenkins

Job description

Eliassen Group is seeking a Databricks Data Engineer for a remote, permanent role to design and maintain scalable data pipelines, data lakes, and analytics-ready infrastructure. You will collaborate with ML engineers, data scientists, and analysts to deliver data products and ensure data quality and security.

The ideal candidate has 5+ years in data engineering, strong Python/SQL/Datatbricks/Spark skills, and experience with AWS.