Job Summary We are seeking a skilled ML Engineer with expertise in Unity Catalog and Databricks Feature Store to establish and maintain a robust foundation for data and machine learning workflows.
The successful candidate will organize data, manage access, and enable efficient operationalization of machine learning models in production.
Key Responsibilities - Set up and manage Unity Catalog in Databricks to organize and secure data access across teams.
- Design and operationalize Feature Stores to support machine learning models in production.
- Build efficient data pipelines to process and serve features for ML workflows.
- Collaborate with teams using Databricks, Azure Cosmos DB, and other Azure tools to integrate data solutions.
- Monitor and optimize the performance of pipelines and feature stores.
Required Qualifications - Strong experience with Unity Catalog in Databricks for managing data assets and access control.
- Hands-on experience with Databricks Feature Store or similar solutions.
- Expertise in building and maintaining scalable ETL pipelines in Databricks.
- Familiarity with Azure tools like Azure Cosmos DB and Azure Container Registry (ACR).
- Proficiency in Python and Spark for data engineering tasks.
- Understanding of machine learning workflows and the integration of feature stores.
Preferred Qualifications - Experience with monitoring tools such as Splunk or Datadog to ensure system reliability.
- Familiarity with Azure Kubernetes Service (AKS) for deploying and managing containers.
- Strong problem-solving skills and a collaborative mindset.
Education: Bachelors Degree