ETRM Databricks Data Engineer

Programmers.io; Houston (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Programmers.io in Houston, TX is seeking a senior ETRM Databricks Data Engineer to design, develop, and maintain scalable data solutions supporting BI, analytics, and data-driven decisions. The role requires expertise in Databricks, Apache Spark, and Azure Cloud, with a focus on data governance and quality.

Responsibilities include building and optimizing ETL/ELT pipelines, integrating multiple data sources, and collaborating with data scientists and business stakeholders.

Qualifications

• Hands-on experience with Databricks and Azure.
• Strong proficiency in Python, SQL, and PySpark.
• Experience with cloud platforms such as Azure
• Knowledge of data warehousing, data modeling, and ETL/ELT concepts.
• Experience with version control tools such as Git.
• Strong analytical, problem-solving, and communication skills.
• Trading Domain experience related for Crude, Refined Products, Natural gas is a must.

Responsibilities

• Design, develop, and optimize data pipelines using Databricks and ADF.
• Must possess good understanding of Trading Domain knowledge.
• Build and maintain scalable ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources into enterprise data platforms.
• Develop and manage data lakes, lakehouses, and data warehouses.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor and troubleshoot data processing jobs and optimize performance.
• Implement CI/CD practices and automate data engineering workflows.
• Create and maintain technical documentation, data models, and process flows.
• Support cloud-based data solutions and migration initiatives.

Skills

Databricks

Azure

Python

SQL

PySpark

Git

Trading Domain

Data Modeling

Tools

Azure Data Factory

Delta Lake

Azure Synapse Analytics

Job description

Location: Houston, Texas (5 days on-site)

Hire Type: Full Time

Experience: 10+ Years

ETRM Databricks Data Engineer to design, develop, and maintain scalable data solutions that support business intelligence, analytics, and data-driven decision-making with understanding of Data Domain & Data Assets in ETRM Domain. The ideal candidate will have strong expertise in Databricks, Apache Spark, Azure Cloud platforms, and modern data engineering practices.

Key Responsibilities:

• Design, develop, and optimize data pipelines using Databricks & ADF.
• Must possess good understanding of Trading Domain knowledge.
• Build and maintain scalable ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources into enterprise data platforms.
• Develop and manage data lakes, lakehouses, and data warehouses.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor and troubleshoot data processing jobs and optimize performance.
• Implement CI/CD practices and automate data engineering workflows.
• Create and maintain technical documentation, data models, and process flows.
• Support cloud-based data solutions and migration initiatives.

Required Qualifications:

• Hands-on experience with Databricks and Azure.
• Strong proficiency in Python, SQL, and PySpark.
• Experience with cloud platforms such as Azure
• Knowledge of data warehousing, data modeling, and ETL/ELT concepts.
• Experience with version control tools such as Git.
• Strong analytical, problem-solving, and communication skills.
• Trading Domain experience related for Crude, Refined Products, Natural gas is a must.

Preferred Qualifications:

• Experience with Azure Data Factory, Azure Synapse Analytics, Delta Lake, or similar technologies.
• Knowledge of streaming data processing and real-time analytics.
• Familiarity with DevOps practices and CI/CD pipelines.
• Databricks, Azure, or related cloud certifications.