We are seeking a highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement, and optimize large-scale data pipelines, ensuring scalability, reliability, and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.
Key Responsibilities:
- Data Pipeline Development:
- Build and maintain scalable ETL/ELT pipelines using Databricks.
- Leverage PySpark/Spark and SQL to transform and process large datasets.
- Integrate data from multiple sources including Azure Blob Storage, ADLS, and other relational/non-relational systems.
- Collaboration & Analysis:
- Work closely with multiple teams to prepare data for dashboard and BI tools.
- Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.
- Performance & Optimization:
- Optimize Databricks workloads for cost efficiency and performance.
- Monitor and troubleshoot data pipelines to ensure reliability and accuracy.
- Governance & Security:
- Implement and manage data security, access controls, and governance standards using Unity Catalog.
- Ensure compliance with organizational and regulatory data policies.
- Deployment:
- Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks, and configurations across environments.
- Manage version control for Databricks artifacts and collaborate with the team to maintain development best practices.
Technical Skills:
- Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime, etc.)
- Proficiency in Azure Cloud Services.
- Solid understanding of Spark and PySpark for big data processing.
- Experience in relational databases.
- Knowledge of Databricks Asset Bundles and GitLab.
Preferred Experience:
- Familiarity with Databricks Runtimes and advanced configurations.
- Knowledge of streaming frameworks like Spark Streaming.
- Experience in developing real-time data solutions.
Certifications:
- Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification. (Optional)
#J-18808-Ljbffr