Senior Python & Azure Data Engineer - Hybrid

Brilliant Infotech Inc.; Fort Worth (TX); Ibrido; USD 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Brilliant Infotech Inc. invites applications for a Senior Python/Azure Data Engineer in Fort Worth, TX.

The role requires 7–10 years in Python development, data engineering, ETL/ELT, Databricks, Spark, and Azure Cloud, with strong application development skills for scalable, secure data solutions. Responsibilities include end-to-end data pipeline design, CI/CD implementation, and collaboration with data scientists and product teams.

Competenze

• 7-10 years of experience in software development, data engineering, or a related field.
• 7+ years of hands-on Python, SQL, ETL, and data processing.
• Strong experience with Databricks and Apache Spark.
• Approx. 5 years of Spark/Databricks development experience.
• Strong knowledge of Azure Cloud services and Azure ML.
• Experience designing scalable data pipelines and components.
• Advanced proficiency in Python, Scala, and/or SQL.
• Strong SDLC practices.
• Experience CI/CD via GitHub Actions, Azure DevOps, Jenkins, or similar.
• Experience deploying code across environments (dev, QA, prod).
• Approximately 5 years of production troubleshooting (logging, monitoring, alerting).
• Experience with data governance, access control, security, compliance, and data lineage.

Mansioni

• Participate in design and implementation of scalable data and app solutions using Python, Azure, Databricks, Spark, and Azure ML.
• Develop complex ETL/ELT pipelines from multiple sources.
• Use Python, Scala, SQL, Spark, and Databricks for data processing and app development.
• Optimize jobs for performance and cost via partitioning, caching, and tuning.
• Collaborate with data scientists, analysts, product teams, and stakeholders.
• Implement data security, access controls, encryption, compliance, and Unity Catalog governance.
• Establish data lineage and ensure security/governance compliance.
• Design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, Jenkins, etc.
• Automate deployment of code, data pipelines, infra, and assets across environments.
• Develop monitoring, logging, and alerting for production pipelines and apps.
• Troubleshoot performance and production issues.
• Follow SDLC best practices including reviews, testing, docs, releases, and support.
• Support Agile planning and cross-pod coordination.

Conoscenze

Python development

SQL

ETL/ELT

Azure Cloud

Databricks

Apache Spark

CI/CD

Logging/Monitoring

Security/Compliance

Agile methodologies

Strumenti

Databricks

Apache Spark

GitHub Actions

Azure DevOps

Jenkins

Descrizione del lavoro

Brilliant Infotech Inc. invites applications for a Senior Python/Azure Data Engineer in Fort Worth, TX.

The role requires 7–10 years in Python development, data engineering, ETL/ELT, Databricks, Spark, and Azure Cloud, with strong application development skills for scalable, secure data solutions. Responsibilities include end-to-end data pipeline design, CI/CD implementation, and collaboration with data scientists and product teams.