Data Engineer

Blue Data Consulting And It Services; Surat; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Blue Data Consulting And It Services is seeking an experienced Data Engineer to design, build, and maintain scalable data platforms. You will own end-to-end data pipelines, enabling analytics, BI, and AI/ML initiatives across the organization.

The ideal candidate has strong Python and SQL skills, deep modern data engineering practices, and the ability to architect data solutions on cloud platforms, with GCP preferred.

Qualifications

• 3+ years of hands-on data engineering experience.
• Proficiency in Python for data processing and pipeline development.
• Advanced SQL skills and experience with ETL/ELT pipelines.
• Experience with relational and NoSQL databases.
• Hands-on cloud data services, preferably GCP.
• Strong data modeling, warehousing concepts, and architecture knowledge.
• Experience with BigQuery and data lakehouses is a plus.
• Excellent communication skills.

Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines.
• Ingest and integrate diverse data sources including APIs and streams.
• Implement CDC, incremental loading, and idempotent pipelines.
• Collaborate with analytics and BI teams to enable AI/ML initiatives.

Skills

Python

SQL

ETL/ELT pipelines

Data modeling

Cloud architectures

Data quality

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

BigQuery

Dataflow

Dataproc

Pub/Sub

Cloud Composer

dbt

Airflow

Terraform

Job description

Overview:

We are looking for an experienced Data Engineer to design, build, and maintain scalable, reliable, and high-performance data platforms. You will own data pipelines end to end, from ingestion and transformation to modeling, quality, and delivery, enabling analytics, business intelligence, and AI/ML initiatives across the organization.

The ideal candidate has strong hands‑on expertise in Python and SQL, deep knowledge of modern data engineering practices, and the ability to independently architect and deliver data solutions on cloud platforms, with Google Cloud Platform (GCP) preferred.

Key Responsibilities:

Data Pipeline Development

• Design, build, and maintain scalable batch and real-time data pipelines using ETL/ELT patterns.
• Ingest and integrate structured, semi-structured, and unstructured data from diverse sources, including APIs, databases, files, and event streams.
• Implement incremental loading, change data capture (CDC), and idempotent pipeline design.
• Implement CDC pipelines using Datastream to replicate data from operational databases into BigQuery.
• Build batch and streaming pipelines with Apache Beam on Dataflow, using Pub/Sub for event ingestion.

Data Modeling and Architecture

• Design and optimize data models (dimensional modeling, star/snowflake schemas, Data Vault where relevant) for analytics and reporting.
• Build and manage data warehouses and lakehouses following the Medallion architecture (Bronze, Silver, Gold layers), ideally on BigQuery and Cloud Storage.
• Design BigQuery datasets using partitioning, clustering, materialized views, and authorized views for performance and secure sharing.
• Work with modern table formats such as Delta Lake, Apache Iceberg (including BigLake), or Apache Hudi.
• Contribute to data platform architecture, technology selection, and design decisions.

Orchestration and Transformation

• Orchestrate workflows using tools such as Cloud Composer (Apache Airflow), Dagster, Prefect, or other cloud-native schedulers.
• Use Cloud Workflows, Cloud Scheduler, and Cloud Functions or Cloud Run for lightweight, event-driven orchestration.
• Develop modular, tested, and version-controlled transformations using dbt, Dataform, or similar frameworks.

Data Quality, Governance, and Reliability

• Implement data validation, testing, and quality checks using tools such as Great Expectations, dbt tests, Soda, or Dataplex data quality.
• Monitor, troubleshoot, and optimize pipelines for performance, cost, and reliability. Ensure data consistency, lineage, availability, and adherence to security, privacy, and governance standards.
• Implement GCP security and governance controls, including IAM, row-level and column-level security in BigQuery, policy tags, and VPC Service Controls.
• Implement data observability, alerting, and SLA tracking for critical pipelines using Cloud Logging and Cloud Monitoring.

Cloud and DataOps

• Build and manage data solutions on cloud platforms, preferably GCP, using services such as BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, Cloud Composer, and Cloud Run. Equivalent experience on AWS (Glue, Redshift, S3) or Azure (Data Factory, Synapse, Microsoft Fabric) is also considered.
• Apply DataOps practices including version control, CI/CD, containerization, and Infrastructure as Code.
• Manage GCP data infrastructure with Terraform, Cloud Build, and Artifact Registry.
• Manage secrets and credentials securely using Secret Manager.
• Optimize storage and compute usage to control cloud costs, including BigQuery slot, partitioning, and clustering strategies.

Collaboration and Ownership

• Partner with analytics, BI, AI/ML, and software engineering teams to deliver data that meets business needs.
• Support AI/ML initiatives by building feature pipelines and preparing data for model training, RAG, and vector-based applications (e.g., Vertex AI, BigQuery ML). Translate business and technical requirements into well-designed data solutions independently.
• Contribute to testing, deployment, code reviews, and technical documentation. Identify and drive opportunities for automation and process improvement.

Required Skills:

• 3+ years of hands‑on experience in data engineering.
• Strong proficiency in Python for data processing and pipeline development.
• Advanced SQL skills, including complex joins, window functions, CTEs, query optimization, and performance tuning.
• Proven experience building and maintaining ETL/ELT pipelines in production. Strong understanding of data modeling, data warehousing concepts, and database design.
• Experience with relational databases (PostgreSQL, MySQL, SQL Server) and familiarity with NoSQL databases (MongoDB, Cassandra, DynamoDB, Firestore, Bigtable).
• Experience processing large and complex datasets efficiently.
• Hands‑on experience with at least one cloud platform and its data services; GCP experience is preferred.
• Strong hands‑on BigQuery experience, including query optimization, cost management, and schema design (preferred).
• Strong troubleshooting, analytical, and problem‑solving skills.
• Good communication skills, with the ability to explain technical decisions to both technical and non‑technical stakeholders.

Good to Have:

• Production experience on GCP (BigQuery, Dataflow, Pub/Sub, Cloud Composer), especially for candidates at the upper end of the experience range.
• Experience with Apache Beam SDK (Python or Java) for Dataflow pipelines.
• Experience with Apache Spark / PySpark for distributed data processing, including on Dataproc.
• Experience with Databricks, Snowflake, BigQuery, or Microsoft Fabric.
• Hands‑on experience with dbt or Dataform for data transformation.
• Experience with Airflow, Dagster, or similar orchestration tools.
• Experience with streaming technologies such as Google Pub/Sub, Apache Kafka, Spark Structured Streaming, Apache Flink, AWS Kinesis, or Azure Event Hubs.
• Familiarity with GCP databases such as Cloud SQL, AlloyDB, Spanner, or Bigtable.
• Experience with Cloud Data Fusion or Analytics Hub.
• Experience with Docker, Kubernetes (GKE preferred), and CI/CD pipelines (Cloud Build, GitHub Actions, GitLab CI, Azure DevOps).
• Knowledge of Infrastructure as Code tools such as Terraform.
• Familiarity with data catalog and governance tools (Dataplex, Unity Catalog, AWS Glue Data Catalog, Microsoft Purview, DataHub).
• Exposure to AI/ML data pipelines, feature stores, or vector databases, including Vertex AI Pipelines, Vertex AI Feature Store, or Vertex AI Vector Search.
• Experience with BI tools such as Looker (including LookML), Power BI, or Tableau.
• Relevant certifications, with Google Professional Data Engineer preferred (AWS Data Engineer, Azure Data Engineer, Databricks, or Snowflake also valued).

Educational Qualifications:

• Bachelors degree in computer science, Information Technology, Data Science, Engineering, or a related field (or equivalent practical experience).

What We're Looking For:

• In-depth technical knowledge, strong hands‑on experience, and the ability to independently design and deliver data engineering solutions end to end.
• Candidates who can clearly explain the data solutions they have built, including the architecture, technologies used, design trade-offs, challenges faced, and how they addressed performance, reliability, data quality, and scalability.
• For experienced candidates, a proven track record of delivering production data platforms on GCP will be given preference.
• A strong sense of ownership, attention to detail, and a commitment to building trustworthy data systems.