Data Engineer: Scalable Pipelines & Lakehouse Expert

Brilliant Infotech Inc.; Edison (NJ); On-site; USD 100,000 - 140,000; Full time

Job summary

Brilliant Infotech Inc. in Edison, NJ seeks a Data Engineer to design, develop, and maintain scalable data pipelines and data solutions. The role focuses on data integration, ETL/ELT, cloud platforms, and data analytics.

You will collaborate with Data Scientists, Analysts, and Engineers to understand data requirements, optimize performance, and ensure governance and security across data assets. Onsite presence is required in Edison.

Qualifications

• Strong proficiency in SQL and relational databases.
• Hands-on experience with Python, PySpark, or similar programming languages.
• Experience with ETL/ELT tools and data pipeline development.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Knowledge of data warehousing concepts and technologies.
• Experience with technologies such as Snowflake, Databricks, Azure Data Factory, AWS Glue, or Redshift.
• Experience working with Apache Spark/PySpark.
• Strong understanding of data modeling, data integration, and database concepts.
• Experience with Git and CI/CD practices.
• Good understanding of data security, governance, and data quality.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop and optimize data integration solutions for structured and unstructured data.
• Work with cloud data platforms such as AWS, Azure, or Google Cloud Platform (Google Cloud Platform).
• Build and maintain data warehouses, data lakes, and lakehouse solutions.
• Develop complex SQL queries, stored procedures, and data transformation processes.
• Work with big data technologies such as Spark, Databricks, or Hadoop.
• Implement data quality, validation, governance, and security processes.
• Integrate data from APIs, databases, applications, and other enterprise data sources.
• Optimize data pipelines and queries for performance, scalability, and reliability.
• Collaborate with Data Scientists, Data Analysts, Software Engineers, and business teams to understand data requirements.
• Monitor data pipelines and troubleshoot data processing and integration issues.
• Maintain technical documentation for data architecture, pipelines, workflows, and processes.

Skills

SQL

Python/PySpark

ETL/ELT

Cloud platforms

Data warehousing

Data modeling

Git/CI/CD

Data security/governance

Education

Bachelor's degree (CS/IT/Eng)

Tools

Snowflake

Databricks

Azure Data Factory

AWS Glue

Redshift

dbt

Airflow

Job description

Brilliant Infotech Inc. in Edison, NJ seeks a Data Engineer to design, develop, and maintain scalable data pipelines and data solutions. The role focuses on data integration, ETL/ELT, cloud platforms, and data analytics.

You will collaborate with Data Scientists, Analysts, and Engineers to understand data requirements, optimize performance, and ensure governance and security across data assets. Onsite presence is required in Edison.