- We are seeking an experienced Senior Data Engineer with strong Azure cloud, data engineering, and AI/ML platform development expertise to independently design, develop, and deliver a scalable enterprise-grade data platform.
- The project involves building and improving data processing solutions, developing reliable data pipelines, integrating cloud-based services, and preparing data infrastructure to support AI, machine learning, and advanced analytics applications.
- This is an independent contract engagement. The selected developer will take end-to-end responsibility for technical execution, from architecture and implementation through testing, documentation, and final delivery.
- We are looking for someone who can work autonomously, make sound technical decisions, troubleshoot complex problems, and deliver production-ready solutions.
Data Platform Architecture and Development
- Design and implement scalable Azure-based data engineering architectures.
- Develop enterprise-grade data platforms supporting structured and unstructured datasets.
- Build batch and real-time data processing solutions.
- Design efficient data storage, processing, and integration architectures.
- Identify opportunities to improve platform performance, reliability, scalability, and cost efficiency.
Data Engineering and ETL Development
- Build, maintain, and optimize end-to-end ETL/ELT pipelines.
- Develop data ingestion and transformation workflows.
- Implement data integration between internal systems, external APIs, and cloud services.
- Develop advanced SQL queries, stored procedures, database functions, and data models.
- Implement data validation, error handling, and data quality controls.
Azure Cloud and Distributed Processing
- Develop solutions using Azure Data Factory, Azure Functions, Event Hubs, and related cloud-native services.
- Implement event-driven architectures using Azure Event Grid and Service Bus.
- Work with Azure Databricks and distributed processing technologies.
- Optimize data workflows for high-volume and real-time processing.
- Support cloud migration, integration, and infrastructure improvements.
AI and Machine Learning Data Infrastructure
- Develop data processing frameworks supporting AI/ML applications.
- Prepare and transform datasets for machine learning and advanced analytics.
- Implement reliable data workflows throughout the AI/ML data lifecycle.
- Develop data algorithms and processing logic for analytical applications.
- Ensure data pipelines are scalable, consistent, and suitable for downstream AI/ML workloads.
Software Development and Integration
- Develop backend services and RESTful APIs.
- Integrate databases, cloud applications, and third-party systems.
- Implement secure and maintainable application components.
- Follow software engineering best practices, including version control, code reviews, and automated testing.
Deployment, Testing, and Documentation
- Configure and maintain CI/CD pipelines using Azure DevOps and GitHub.
- Perform unit, integration, and end-to-end testing.
- Identify and resolve technical issues independently.
- Produce technical documentation, architecture diagrams, and implementation guides.
- Provide a complete handover of the developed solution.
- Cloud and Infrastructure
- Microsoft Azure
- Azure Data Factory
- Azure Databricks
- Azure Functions
- Azure Event Hubs
- Azure Event Grid
- Azure Service Bus
- Azure Cosmos DB
- Azure SQL Data Warehouse / Azure SQL
- Programming and Development
- Python
- SQL / T-SQL
- C# / .NET
- Java
- Node.js
- Unix Shell Scripting
- Data Engineering and Processing
- ETL / ELT
- Apache Spark / PySpark
- Spark SQL
- Hive SQL
- Data Modeling
- RDBMS
- Structured and Unstructured Data Processing
- Batch and Stream Processing
- Event-Driven Architecture
- DevOps and Integration
- Azure DevOps
- Git / GitHub
- CI/CD Pipelines
- RESTful APIs
- JSON / JSON-LD / XML
- Version Control and Automated Deployment