Role: Pyspark Tech Lead

Location: Chicago, Illinois(Onsite)

Responsibilities:

• Lead the design and implementation of data solutions using Cloudera HUE and Cloudera Data Platform- Oversee the integration of Apache Iceberg and Spark into existing data workflows
• Provide technical guidance and mentorship to junior team members on PySpark best practices- Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
• Ensure data quality and integrity through rigorous testing and validation processes- Develop and maintain documentation for data architectures and workflows
• Optimize data processing pipelines for performance and efficiency
• Conduct code reviews to ensure adherence to coding standards and best practices
• Troubleshoot and resolve complex technical issues related to data processing
• Stay updated with the latest advancements in data technologies and incorporate them into projects
• Drive innovation by exploring new tools and techniques in the research and development domain
• Communicate effectively with stakeholders to gather requirements and provide project updates
• Contribute to the overall success of the company by delivering high-quality data solutions that drive business value

Qualification:

• Possess strong expertise in Cloudera HUE and Cloudera Data Platform
• Demonstrate proficiency in Apache Iceberg and Spark
• Have extensive experience with PySpark for data processing
• Show a solid understanding of data architecture and data engineering principles
• Exhibit excellent problem-solving skills and attention to detail
• Have a background in research and development is a plus
• Display strong communication and collaboration skills
• Be able to work effectively in a hybrid work model
• Have a proactive approach to learning and staying current with industry trends
• Be capable of mentoring and guiding junior team members
• Demonstrate the ability to deliver high-quality solutions within deadlines
• Show a commitment to continuous improvement and innovation
• Possess a strong sense of ownership and accountability for project outcomes

Regards

Praveen Kumar

Talent Acquisition Group – Strategic Recruitment Manager