JOB DESCRIPTION

Insight Global is hiring a Sr. Data Engineer in Calgary, AB. This role is hybrid - in-office with 20% flexibility to work from home. This resource will play a key role in architecting and optimizing a modern, scalable, and productized data ecosystem that enables internal teams and external partners to derive maximum value from our data assets. Responsible for designing high-performance data pipelines, storage solutions, and API-driven data access mechanisms that support real-time decision-making and innovation. The ideal candidate has 5+ years of experience in data engineering, with modern data storage and processing, proficiency in ETL frameworks and event-driven architectures. They will also have strong API development skills and proficiency in Go, Java and/or Python.

Duties & Responsibilities:

• Design and build a scalable, cloud-native data platform aligned with microservices
• Develop real-time and batch data pipelines to power data-driven products
• Implement SQL, NoSQL, and hybrid storage strategies

Data as a Product

• Enable self-serve data access with secure, well-documented data APIs.
• Collaborate with Product & Business teams to define and optimize data products.
• Ensure data quality, lineage, and governance in all data pipelines and products.

Microservices Integration & Performance

• Build event-driven architectures using Kafka, Azure Event Hub, or Service Bus.
• Develop scalable ETL/ELT processes for ingestion, transformation, and distribution.
• Optimize query performance, indexing, and caching for data-intensive apps.

Data Governance, Security, and Compliance

• Enforce data privacy, security, and access controls aligned with compliance standards.
• Implement observability and monitoring for data infrastructure and pipelines.
• Work with DevSecOps teams to integrate security into CI/CD workflows.

REQUIRED SKILLS AND EXPERIENCE

• 5+ years of experience in data engineering, with exposure to Data Mesh and Data as a Product preferred.
• Expertise in modern data storage and processing, including SQL, NoSQL (Cosmos DB, PostgreSQL), Data Lakes (Azure Data Lake, Delta Lake, Apache Iceberg).
• Proficiency in ETL frameworks (e.g. Apache Kafka, Airflow, Flink, Spark, Azure Data Factory, Databricks).
• Experience with Event-driven architectures using Queues, Pub/Sub services (e.g. Azure Service Bus, Azure Event Grid, Amazon Event Bridge) and Containerized Environments (Container Apps, AWS ECS).
• Experience with Apache and/or Azure data platforms or similar, e.g. Fabric, Databricks, Snowflake, and Apache Hudi.
• Strong API development skills using GraphQL, REST, and/or gRPC for enabling data as a product.
• Proficiency in Go, Java, and/or Python.
• Deep understanding of data governance, security, lineage, and compliance using Microsoft Purview, OpenLineage, Apache Ranger, or Azure Key Vault.
• Experience with Infrastructure as Code (IaC) using Bicep, Terraform, or CloudFormation for managing cloud-based data solutions.
• Strong problem-solving and collaboration skills, working across data, engineering, and business teams.

NICE TO HAVE SKILLS AND EXPERIENCE

• Knowledge of ML, AI, and LLMs, including data engineering for model training and inference with Azure Machine Learning, Databricks ML, and MLflow.
• Hands-on experience in Notebooks (e.g. Jupyter, Databricks, Azure Synapse) for data workflows.
• Experience in real-time data streaming architectures.
• Exposure to data monetization strategies and analytics frameworks.
• Familiarity with federated data governance and self-serve data platforms.