GCP Data Engineering Lead

Job Summary

We are seeking a highly experienced GCP Data Engineering Lead to provide senior technical leadership and delivery management for enterprise-scale cloud data engineering initiatives. The ideal candidate will combine deep hands-on expertise in Google Cloud Platform data pipeline architecture with strong Agile delivery and quality assurance leadership.

This role will be responsible for leading end-to-end pipeline architecture, QA strategy, sprint execution, technical governance, and offshore team coordination across high-volume data ingestion and analytics environments.

Key Responsibilities

Technical Leadership & Architecture

• Lead pipeline architecture design for Netscout and Sandvine ingestion pipelines.
• Design scalable data ingestion and transformation workflows using:
  • GCS landing zones
  • Google Cloud Dataflow
  • Google BigQuery
  • Bronze/Silver/Gold Medallion architecture layers
• Define Medallion Architecture data flow standards and transition rules.
• Lead data reconciliation and parity validation strategies across enterprise data platforms.
• Provide hands-on support for complex transformation logic and pipeline optimization.

QA & Data Quality Management

• Define and own end-to-end QA strategy for data pipelines including:
  • Test coverage
  • Quality gates
  • Defect management
  • Governance validation
• Architect reusable test templates for batch and streaming pipelines.
• Define data quality specifications for each Medallion layer.
• Review and assure quality of all pipeline and QA deliverables.

Delivery & Team Management

• Lead Agile sprint planning, sprint reviews, and retrospectives.
• Maintain RAID logs (Risks, Assumptions, Issues, Dependencies) and drive mitigation activities.
• Produce weekly delivery status reports and maintain project trackers.
• Coordinate with client engineering teams on environments, access, and technical decisions.
• Manage offshore Data Engineers and Observability Engineers including task allocation and quality oversight.
• Drive mentoring, knowledge transfer, and delivery excellence initiatives.

Required Qualifications

• Advanced experience with Google Cloud Platform data engineering services including:
  • Google BigQuery
  • Google Cloud Dataflow / Apache Beam
  • Pub/Sub
  • GCS
• Strong hands-on development expertise in Python and SQL.
• Experience designing QA frameworks for cloud data pipelines using tools such as Great Expectations or dbt tests.
• Strong understanding of Medallion Architecture and enterprise data processing patterns.
• Experience leading Agile delivery processes and technical project governance.
• Strong stakeholder communication and team leadership skills.

Preferred Qualifications

• Advanced Google Cloud Platform certifications.
• Telecom, telemetry, or high-volume streaming data experience.
• Experience with observability and data governance frameworks.
• Prior experience managing offshore delivery teams.
• Strong mentoring and knowledge transfer capabilities.

Key Skills & Competencies

• GCP Data Engineering
• BigQuery & Dataflow
• Apache Beam & Pub/Sub
• Medallion Architecture
• Data Pipeline QA & Governance
• Python & SQL Development
• Agile Delivery Management
• Data Quality & Reconciliation
• Team Leadership & Stakeholder Management