We are seeking a Data Engineer to join our team based in Cape Town, this is a contracting role and onsite. The Data Engineer will be responsible for designing, implementing, and maintaining the data infrastructure necessary for our Insights team to analyze data and derive valuable insights. The Data Engineer will work with Engineering Leads, Data Architects and other stakeholders to ensure that our data is properly sourced, stored, and accessible for analysis. They will also contribute to our mission by extracting maximum predictive value from data assets. Responsibilities Data Pipeline Development and Management
Design, develop, test and maintain data pipelines to extract, transform, and load (ETL) data from various sources into a data warehouse or data lake
Implement data quality and validation checks to ensure that data is accurate, complete and consistent.
Optimize and scale data processing and storage to handle large volumes of data and high concurrency.
Troubleshoot and resolve data pipeline issues and perform root cause analysis
Develop and maintain documentation for data pipelines and processes.
Building and maintaining data pipelines using lakehouse architecture (medallion)
Extracting data from multiple sources, transforming it, and loading it into the target database
Monitoring and troubleshooting ETL jobs to ensure they complete successfully and on time.
Design and implement data models and database schemas for storing and querying data.
Optimize database schema and indexing for efficient data retrieval
Modernise legacy data environments, migrating from on-premises solutions to cloud-native platforms such as Databricks
Ensure database security, availability, and scalability
Perform database tuning and optimization to improve query performance
Develop and maintain data connectors and integrations with various data sources (e.g., databases, APIs, flat files)
Implement data streaming and real-time data processing technologies.
Designing and implementing complex reports that meet business requirements
Developing dashboards and visualizations using tools like Pyramid or other data visualization tools
Develop and maintain BI reports and dashboards
Analysing data to identify trends, patterns, and insights
Providing insights to business stakeholders based on the analysis of data
Ensure compliance with data privacy regulations according the Data privacy policies.
Develop and maintain data governance policies, procedures, and standards
Ensure that data access and authorization policies are in place and adhered to
Monitor and audit data access to ensure data security and prevent data breaches
Implement disaster recovery and business continuity plans for data infrastructure
Work collaboratively with Engineering Leads, Architects, Analysts, Scrum Masters and other stakeholders to understand data requirements and provide data solutions.
Communicate data pipeline issues, status, and progress to stakeholders and project teams.
Participate in cross-functional meetings and initiatives to ensure alignment with data strategies and goals
Must have at least 3 years of relevant IT experience or related qualification.
3+ years experience with Data pipeline design and development experience.
Proficiency in Python and/or SQL.
RDBMS experience in technology such as SQL Server, MySQL and PostgreSQL.
Ability to use common scripting languages such as Powershell, Bash, or Python for automation.
Experience using unit and integration tests.
Experience working with production BI environments and tools such as PowerBI, Tableau or Pyramid Analytics.
Strong analytical/problem solving skills.
Databricks Certified Data Engineer Associate certification.
Experience in Azure data technologies; a relevant Azure certification would be advantageous
Hands-on experience with Azure DevOps, Git, CI/CD practices, Docker and containerization
Exposure to .net entity framework.
Exposure to Databricks, Delta Lake and PySpark.
Experience with data streaming or distributed processing technologies such as Kafka or Hadoop.
Experience delivering data projects in Retail, Finance, Customer or HR domains would be advantageous
Experience mentoring junior engineers would be advantageous
Manages Complexity - interprets and simplifies complex and contradictory information when resolving organisational problems
Strategic Mindset - thinks and plans strategically, focusing on the long-term goals and objectives of the organisation
Tech Savvy - leverages new technology to enhance productivity, improve problem solving, and support business growth
Cultivates Innovation - creates an environment that fosters and nurtures a culture of creativity which drives success
Drives Engagement - inspires, motivates and empowers individuals to go above and beyond for the benefit of the team and the organisation
Desired Skills:
Data
Engineer
ETL