ETRM Databricks Data Engineer – Trading Domain & Azure

Programmers.io; Houston (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Programmers.io in Houston, TX is seeking a senior ETRM Databricks Data Engineer to design, develop, and maintain scalable data solutions supporting BI, analytics, and data-driven decisions. The role requires expertise in Databricks, Apache Spark, and Azure Cloud, with a focus on data governance and quality.

Responsibilities include building and optimizing ETL/ELT pipelines, integrating multiple data sources, and collaborating with data scientists and business stakeholders.

Qualifications

• Hands-on experience with Databricks and Azure.
• Strong proficiency in Python, SQL, and PySpark.
• Experience with cloud platforms such as Azure
• Knowledge of data warehousing, data modeling, and ETL/ELT concepts.
• Experience with version control tools such as Git.
• Strong analytical, problem-solving, and communication skills.
• Trading Domain experience related for Crude, Refined Products, Natural gas is a must.

Responsibilities

• Design, develop, and optimize data pipelines using Databricks and ADF.
• Must possess good understanding of Trading Domain knowledge.
• Build and maintain scalable ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources into enterprise data platforms.
• Develop and manage data lakes, lakehouses, and data warehouses.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor and troubleshoot data processing jobs and optimize performance.
• Implement CI/CD practices and automate data engineering workflows.
• Create and maintain technical documentation, data models, and process flows.
• Support cloud-based data solutions and migration initiatives.

Skills

Databricks

Azure

Python

SQL

PySpark

Git

Trading Domain

Data Modeling

Tools

Azure Data Factory

Delta Lake

Azure Synapse Analytics

Job description

Programmers.io in Houston, TX is seeking a senior ETRM Databricks Data Engineer to design, develop, and maintain scalable data solutions supporting BI, analytics, and data-driven decisions. The role requires expertise in Databricks, Apache Spark, and Azure Cloud, with a focus on data governance and quality.

Responsibilities include building and optimizing ETL/ELT pipelines, integrating multiple data sources, and collaborating with data scientists and business stakeholders.