Lead Data Engineer

Black Rock Groups; Florida; Presencial; USD 110.000 - 115.000; Jornada completa

Descripción de la vacante

Black Rock Groups seeks a Senior Data Engineer to join our Florida team in Juno Beach, working onsite from day one. You will design, build, and maintain scalable ETL/ELT pipelines for forecasting data, integrating APIs, databases, and cloud storage to support load, wind, and solar forecasting.

The role blends strong data engineering with production support, troubleshooting issues, and ensuring data quality and resiliency across the cloud-native platform.

Formación

• 8–10 years of data engineering experience.
• Strong SQL and Python skills.
• Experience with AWS and data pipelines.
• Production support and incident handling.

Responsabilidades

• Design and maintain ETL/ELT pipelines for forecasting data.
• Integrate data from APIs, databases, and cloud storage.
• Collaborate with data scientists for model-ready data.
• Support cloud-hosted forecasting workloads (AWS).
• Implement data quality checks and monitoring.
• Provide production support across the stack.

Conocimientos

ETL/ELT pipelines

Advanced SQL

Python

AWS

Data integration

Production support

Git

PostgreSQL

Herramientas

PostgreSQL

Oracle

SQL Server

S3

Lambda

ECS

EC2

CloudWatch

Jira

Confluence

Descripción del empleo

Job title : Senior Data Engineer

Location : Juno Beach , FL (Day one onsite)

Experience : 8 to 10 Year

Position Specific Description

The Senior Data Engineer Forecasting Platforms & Operations will support the development, integration, operational reliability, and continued modernization of data pipelines and applications within the AURA energy forecasting ecosystem.

This role combines strong data engineering capabilities with hands-on production application support. The position will focus primarily on building and maintaining scalable ETL/ELT pipelines, data integrations, databases, APIs, and automated data workflows supporting load, wind, and solar forecasting, while also providing secondary operational support for the cloud-based platforms and services responsible for producing and delivering forecasts to business customers.

The ideal candidate has a strong data engineering foundation but is also comfortable troubleshooting production applications, following data and application dependencies across systems, supporting cloud-hosted services, and responding to operational issues when necessary.

The position is intended to strengthen both the data engineering capacity and operational resiliency of the AURA forecasting team by developing broader technical ownership across the environment.

Key Responsibilities

Data Engineering & Integration

• Design, develop, maintain, and optimize ETL/ELT pipelines supporting forecasting, telemetry, weather, market, asset, and operational data.
• Develop and maintain automated data ingestion and transformation processes across APIs, databases, flat files, SFTP, cloud storage, and external data providers.
• Build scalable and reusable Python and SQL-based data processing frameworks.
• Support PostgreSQL and other relational database environments used by forecasting applications and analytical workflows.
• Improve data pipeline monitoring, logging, restart ability, validation, and failure recovery.
• Implement data-quality checks, reconciliation processes, and source-to-target validation.
• Support onboarding and integration of new wind, solar, storage, and other generation assets into forecasting data pipelines.
• Partner with Data Scientists to ensure reliable, timely, and well-structured data is available for model development, training, production forecasting, and performance analysis.
• Support modernization and standardization of legacy data workflows and reduce manual or hard-coded processing wherever practical.

Forecasting Platform & Operational Support

• Provide secondary technical support for production load, wind, and solar forecasting applications and associated data-delivery processes.
• Troubleshoot production issues across application, data, database, cloud infrastructure, integration, and delivery layers.
• Support AWS-hosted forecasting workloads and services such as S3, Lambda, ECS, EC2, CloudWatch, and related cloud resources.
• Assist with application deployments, release validation, configuration changes, operational readiness, and post-deployment monitoring.
• Support scheduled jobs, automated forecast processes, reporting services, and customer-facing forecast delivery workflows.
• Assist with SFTP services, secure file transfers, SSH keys, directory configurations, and customer data-exchange processes.
• Participate in incident response, root-cause analysis, change management, service-ticket resolution, and production support.
• Maintain sufficient cross-functional knowledge to provide backup coverage for key forecasting platform responsibilities when the primary platform resource is unavailable.

Reliability, Automation & Documentation

• Identify recurring manual processes and develop automation to improve reliability and reduce operational effort.
• Establish effective monitoring and alerting for pipelines, scheduled processes, data feeds, and production forecasting applications.
• Document system dependencies, troubleshooting procedures, support processes, recovery steps, and operational handoffs.
• Participate in cross-training and knowledge-sharing activities to ensure critical systems have appropriate primary and backup technical coverage.
• Support continuous improvement of the team's operational support model and technical resiliency.

Required Technical Experience

• Strong experience designing, developing, and supporting enterprise ETL/ELT or data-integration pipelines.
• Advanced SQL skills and experience with relational databases such as PostgreSQL, Oracle, SQL Server, or similar platforms.
• Proficiency with Python for data processing, automation, integration, or application-support activities.
• Experience integrating data from APIs, databases, flat files, cloud storage, SFTP, or other enterprise data sources.
• Experience supporting production data pipelines, scheduled jobs, and troubleshooting failed processing workflows.
• Experience working with AWS or comparable cloud environments.
• Strong understanding of logging, monitoring, troubleshooting, root-cause analysis, and production-support practices.
• Experience with Git-based source control and modern software-development practices.

Preferred Qualifications

• Experience with AWS services including S3, Lambda, ECS, EC2, CloudWatch, Step Functions, or related services.
• Experience with PostgreSQL/Aurora, Redshift, Databricks, Spark/PySpark, or similar data platforms.
• Experience supporting CI/CD pipelines, application deployments, or cloud-hosted production applications.
• Experience with SFTP administration, secure data exchange, SSH key management, or automated file-transfer processes.
• Experience building data-quality, monitoring, reconciliation, and automated alerting frameworks.
• Familiarity with energy markets, utility operations, renewable generation, SCADA/telemetry data, weather data, or load/wind/solar forecasting.
• Experience supporting machine-learning or forecasting data pipelines is strongly preferred.
• Experience working in Agile environments using tools such as Jira and Confluence.

Candidate Profile

The successful candidate should be a hands-on engineer who naturally follows problems across system boundaries rather than limiting troubleshooting to a single application or technology.

The strongest candidate will be able to move comfortably between:

source data pipeline database forecasting application cloud service delivery customer

and determine where a failure has occurred, how it should be corrected, and how recurrence can be prevented.

This role requires someone who can operate independently, document clearly, learn unfamiliar systems quickly, and balance planned engineering work with occasional production-support priorities.

The pay range for this role is $110k - $115k per annum including any bonuses or variable pay. Tech Mahindra also offers benefits like medical, vision, dental, life, disability insurance and paid time off (including holidays, parental leave, and sick leave, as required by law). Ask our recruiters for more details on our Benefits package. The exact offer terms will depend on the skill level, educational qualifications, experience, and location of the candidate.

Tech Mahindra is an Equal Employment Opportunity employer. We promote and support a diverse workforce at all levels of the company. All qualified applicants will receive consideration for employment without regard to race, religion, color, sex, age, national origin or disability. All applicants will be evaluated solely on the basis of their ability, competence, and performance of the essential functions of their positions with or without reasonable accommodations. Reasonable accommodations also are available in the hiring process for applicants with disabilities. Candidates can request a reasonable accommodation by contacting the company ADA Coordinator at [e-mail].