Job Summary:

As a Senior Data Engineer, you will be responsible for designing, building, and managing complex data architectures, data pipelines, and data warehouses. You will play a critical role in transforming raw data into actionable insights for analytics and business intelligence. You will work with cross-functional teams to design data models, integrate diverse data sources, and optimize data processing workflows.

Key Responsibilities:

• Design and Development:
• Build, implement, and optimize data pipelines for high-performance, scalable data processing.
• Develop data models, schemas, and structures that support both operational and analytical needs.
• Design and implement ETL (Extract, Transform, Load) processes to integrate and clean data from various sources.
• Create efficient data storage solutions using cloud platforms (AWS, Azure, GCP) or on-premise technologies (e.g., Hadoop, Spark).
• Data Infrastructure Management:
• Ensure that data architectures and systems are highly available, secure, and optimized for performance.
• Oversee the setup, configuration, and management of databases, data lakes, and data warehouses (e.g., Snowflake, Redshift, BigQuery, or SQL Server).
• Maintain and manage cloud-based data environments and ensure their scalability and security.
• Collaboration with Stakeholders:
• Work closely with data scientists, business analysts, and stakeholders to understand data requirements and provide efficient data solutions.
• Collaborate with DevOps and software engineers to ensure smooth data integration with other applications and platforms.
• Data Quality and Governance:
• Monitor and ensure data quality, integrity, and consistency across all systems.
• Implement data governance best practices, ensuring compliance with data privacy laws and organizational policies.
• Develop testing frameworks to validate data accuracy and integrity throughout the pipeline.
• Optimization and Performance:
• Tune and optimize SQL queries, data models, and data systems for performance.
• Troubleshoot performance issues and recommend scalable solutions.
• Implement automation and orchestration tools for data workflows to improve efficiency.
• Leadership and Mentoring:
• Provide leadership and guidance to junior data engineers, offering mentorship on best practices and complex data engineering challenges.
• Lead the adoption of new technologies and improvements to the data infrastructure.
• Collaborate with engineering and product teams to ensure data solutions align with business objectives.

Qualifications:

• Education:
• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, Mathematics, or a related field (or equivalent experience).
• Experience:
• 5+ years of experience as a Data Engineer or in a related role.
• Proven experience designing and building large-scale data processing systems and data pipelines.
• Strong experience with SQL, data modeling, and performance optimization.
• Expertise with big data technologies such as Hadoop, Spark, or Kafka.
• Familiarity with cloud platforms (AWS, Azure, GCP) and related tools for data storage, transformation, and orchestration.
• Technical Skills:
• Proficiency in programming languages such as Python, Java, or Scala for data processing.
• Experience with data storage technologies (e.g., relational databases, NoSQL, columnar databases).
• Expertise in data warehousing solutions like Snowflake, Redshift, or BigQuery.
• Familiarity with orchestration tools such as Apache Airflow, Dagster, or Prefect.
• Experience with version control systems like Git.
• Soft Skills:
• Strong problem-solving skills and the ability to work independently on complex technical challenges.
• Excellent communication and collaboration skills to work across teams.
• Ability to explain complex technical concepts to non-technical stakeholders.

Preferred Skills:

• Experience with machine learning model deployment and working with data science teams.
• Knowledge of data security practices and privacy regulations.
• Familiarity with containerization technologies such as Docker and Kubernetes.