This is a contract role with possibility of conversion to full time

Okay with remote but prefer someone who can come in onsite intially

NO OPT/H1/VENDOR CANDIDATES

Position Summary

We are seeking an experienced Senior Data Engineer to spearhead the implementation of Airflow (Astronomer version) on Azure Cloud in a highly scalable and performance-driven environment. This role requires a hands-on expert with proven experience designing, building, and optimizing data workflows, as well as managing cloud-based DevOps systems, specifically with Airflow, Databricks, and Azure.

The ideal candidate will possess deep technical knowledge in data engineering, demonstrate leadership in designing and implementing large-scale data systems, and excel in documentation and process standardization. This is a pivotal role in ensuring the successful migration and optimization of existing workflows while creating a robust foundation for future data engineering projects.

Key Responsibilities

Airflow Implementation & Optimization

• Architect, design, and implement Airflow (Astronomer) ecosystems from the ground up in Azure.
• Lead efforts to establish best practices for Airflow, including branching strategies, deployment pipelines, and repository configuration.
• Migrate existing orchestration workflows from Databricks to Airflow, ensuring seamless integration and enhanced performance.

Data Workflow Development & Optimization

• Build, manage, and optimize scalable data workflows using Databricks , PySpark , and SQL to handle large-scale datasets efficiently.
• Collaborate with stakeholders to design and maintain operational data systems that meet performance, scalability, and availability standards.
• Continuously monitor and fine-tune data processes to improve resource utilization and minimize workflow delays.

DevOps & Cloud Environment Management

• Work directly with environment managers to configure, monitor, and optimize development, testing, and production environments in Azure.
• Design cost-efficient, high-performing Azure cloud infrastructure tailored to project requirements.
• Ensure alignment with CI/CD best practices and version control strategies for a seamless development lifecycle.

Documentation & Best Practices

• Develop comprehensive design documents for review and approval by key stakeholders.
• Standardize coding, testing, deployment, and documentation practices to ensure high-quality deliverables.
• Mentor team members on implementing and maintaining robust data engineering standards.

Minimum Qualifications

Experience

• 5+ years of experience in data engineering with a focus on large-scale environments and complex workflows.
• Proven expertise in implementing Airflow and migrating workflows from Databricks.

Technical Proficiencies

• Databricks : Extensive hands-on experience in building and managing pipelines.
• Airflow : Demonstrated ability to set up and optimize Airflow environments, job scheduling, and automation.
• Azure : Strong experience configuring and managing cloud environments on Azure.
• Snowflake : Expertise in data warehousing with Snowflake.
• PySpark & Python : Advanced proficiency in building and optimizing distributed data operations.
• SQL : Mastery in querying, managing, and transforming data.
• DevOps : Experience with environment configuration, cost optimization, and cloud-based deployment pipelines.

Education

• Bachelor’s degree in Computer Science, Information Systems, or related field, or equivalent experience.

Preferred Skills

• Experience with AWS or GCP alongside Azure.
• Hands-on experience with Data Lakes/Delta Lake for large-scale data storage.
• Familiarity with serverless technologies and event-driven data workflows.
• Proficiency in CI/CD pipelines and tools specific to data engineering.
• Exposure to additional tools in the big data ecosystem.