Job Description:
Responsibilities:
Architectural Design & Planning:

• Define and maintain the overall data architecture for the organization, including data ingestion, processing, storage, and consumption layers on Azure.
• Design scalable and resilient data pipelines using Azure Data Factory (ADF) and Databricks.
• Develop and implement data models and data warehousing solutions leveraging Snowflake.
• Design and implement real-time and near real-time data streaming solutions using relevant technologies on Azure (e.g., Azure Event Hubs, Azure Stream Analytics, Kafka on Azure).
• Establish data governance policies and ensure data quality, security, and compliance.
• Evaluate and recommend new technologies and tools to enhance the data platform capabilities.
• Create and maintain comprehensive architectural diagrams, data flow diagrams, and technical documentation.

Implementation & Oversight:

• Provide technical leadership and guidance to data engineering teams during the implementation of data solutions.
• Develop and enforce coding standards, best practices, and performance optimization techniques.
• Oversee the development and deployment of data pipelines and ETL/ELT processes using ADF and Databricks.
• Manage and optimize the performance and scalability of the Snowflake data warehouse.
• Implement and manage data streaming pipelines for real-time data processing and analytics.
• Integrate and manage batch processing workflows using Control-M.
• Ensure proper monitoring, alerting, and logging are implemented for all data platform components.

Collaboration & Communication:

• Collaborate with data scientists, analysts, and business stakeholders to understand their data requirements and deliver effective solutions.
• Communicate complex technical concepts clearly and effectively to both technical and non-technical audiences.
• Participate in cross-functional teams to deliver end-to-end data solutions.
• Mentor and guide junior data engineers.

Continuous Improvement:

• Stay up-to-date with the latest trends and technologies in data engineering and cloud computing.
• Identify opportunities for process improvements and automation within the data platform.
• Proactively address performance bottlenecks and optimize existing data pipelines.

Technical Skill Sets:

• Azure Cloud Technologies: Deep understanding and hands-on experience with Azure data services, including but not limited to:
• Azure Data Factory (ADF)
• Azure Databricks (Spark, Delta Lake)
• Azure Synapse Analytics (SQL DW, Spark Pools - preferred but not strictly required)
• Azure Blob Storage
• Azure Data Lake Storage (ADLS Gen2)
• Azure Event Hubs
• Azure Stream Analytics
• Azure Functions
• Azure Logic Apps
• Azure DevOps (CI/CD pipelines)
• Data Warehousing: Extensive experience designing and implementing data warehousing solutions, specifically with Snowflake.
• Data Integration (ETL/ELT): Proven ability to design, build, and optimize complex ETL/ELT pipelines using ADF and Databricks.
• Streaming Technologies: Hands-on experience with designing and implementing real-time and near real-time data streaming solutions using technologies like Azure Event Hubs, Azure Stream Analytics, or similar (e.g., Kafka).
• Orchestration: Strong proficiency in using Control-M for scheduling and managing batch data processing workflows.
• Data Modeling: Solid understanding of different data modeling techniques (e.g., Kimball, Inmon, Data Vault).
• SQL: Expert-level proficiency in SQL for data querying, manipulation, and performance tuning.
• Programming Languages: Proficiency in one or more programming languages such as Python, Scala, or Java (Python strongly preferred).
• DevOps & Automation: Experience with CI/CD pipelines, infrastructure-as-code (e.g., ARM templates, Terraform), and automation of data engineering tasks.
• Data Governance & Quality: Understanding of data governance principles, data quality frameworks, and data security best practices.

Technical Certifications (Preferred):

• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert
• Snowflake SnowPro Core Certification
• Relevant certifications in cloud computing, data management, or big data technologies.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field.
• 15 + years of experience in data engineering, with a significant portion focused on cloud-based data platforms.
• 12 + years of experience in a data architect role, defining and implementing data solutions.
• Demonstrated experience leading and mentoring data engineering teams.
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and as part of a collaborative team.
• Strong understanding of data security and compliance requirements.

Nice-to-Have Skills:

• Experience with other cloud platforms (e.g., AWS, GCP).
• Familiarity with data science workflows and tools.
• Experience with NoSQL databases.
• Knowledge of containerization technologies (e.g., Docker, Kubernetes).