Lead Data Engineer

J M Group Inc - 38 emplois

Toronto, ON

Posté aujourd'hui

Détails de l'emploi :

Temps plein

Expérimenté

We are hiring for a Lead Data Engineer. The role focuses on production support and management of data engineering pipelines, including data ingestion, processing, monitoring, and incident management.
A strong fit will have 8+ years of production data engineering experience with GCP, CDC pipelines, Java, and data processing technologies.

What you bring

8+ years of experience supporting and managing data engineering pipelines in production environments.
Strong experience with Change Data Capture using Debezium and file-based ingestion using CSV.
Hands-on experience with GCP Pub/Sub, Dataflow (Apache Beam), and Datastream.
Experience working with Cloud SQL (PostgreSQL) environments.
Experience managing staging and curated data layers for downstream consumption.
Strong understanding of production monitoring, alerting, and incident management processes.
Experience with API-based integrations within data pipelines.
Strong Java programming experience for data processing and support activities.

What you'll do

Provide end-to-end production support for data ingestion and processing pipelines.
Monitor and manage CDC pipelines and batch ingestion processes for multiple source systems.
Handle incidents, perform root cause analysis, and implement preventive fixes.
Work with GCP Pub/Sub topics, subscriptions, DLQs, and alerting mechanisms.
Support staging and transformed data processing layers.
Perform production deployments and coordinate testing across lower environments.
Troubleshoot and resolve data, performance, and pipeline failures within SLAs.
Serve as technical lead for AMS support and coordinate day-to-day operations.
Own incident communication, escalation, and stakeholder updates.
Maintain operational runbooks and provide guidance to L2/L3 support teams.
Identify recurring issues and drive continuous improvements for system stability and reliability.

#Information Technology carrières

Postulez dès maintenant

Enregistrer

Partager un emploi :

Foire aux questions

What is the job title and primary focus of this role?

The role is Lead Data Engineer, focusing on production support and management of data engineering pipelines, including ingestion, processing, monitoring, and incident management.

What experience and technical skills are required for this position?

The role requires 8+ years in production data engineering, with strong experience in Change Data Capture using Debezium, CSV file ingestion, GCP Pub/Sub, Dataflow, Datastream, Cloud SQL (PostgreSQL), Java, and managing staging and curated data layers.

What responsibilities will I have on a day-to-day basis?

You will provide end-to-end production support for ingestion and processing pipelines, monitor CDC and batch ingestion, handle incidents, manage deployments, troubleshoot failures, and coordinate testing across environments.

Where is this job located and what is the work setup?

This position is based in Toronto, ON and is described as full-time.

What systems and tools will I work with for monitoring and incident management?

You will work with GCP Pub/Sub topics, subscriptions, DLQs, alerting mechanisms, production monitoring, incident management processes, and runbooks for L2/L3 support.

What leadership or coordination responsibilities does this role include?

This role serves as the technical lead for AMS support, coordinates day-to-day operations, owns incident communication and escalation, and guides L2/L3 support teams.