We accelerated AllCargo's Data-Driven Logistics with scalable AWS Data Lake, enabling near Real-Time Analytics, 1-Minute Data Capture, and robust Error Monitoring

CHALLENGE

The company aims to implement a more scalable architecture that can accommodate growing data volumes and meet the dynamic reporting needs of the team and overcome the following gaps:

• The current architecture of the company lacks scalability and robustness to effectively handle the increasing volume of data and evolving reporting requirements.
• There is a gap in data governance and master data management practices, leading to potential data inconsistencies.
• The absence of a centralized source of truth results in conflicting Key Performance Indicators (KPIs) across multiple reports.
• Real-time dashboards are not in place to display updated data, hindering the team's ability to make informed decisions promptly.

Why were we brought in?

Ganit has made a significant dent in various industries using data science and analysis. Ganit partners with clients to translate their data into a tangible, insightful plan of action that delivers on a measurable impact to the clients’ topline & bottom-line growth.

Solution

Ganit built a centralized Data Lake and Warehouse on AWS that could handle the scale of growing data volume for AllCargo. Below is the solution architecture:

• AWS Database Migration Service (DMS) was utilized to migrate data one time from the on-prem MS SQL Server to raw zone on Amazon S3 (full load).
• AWS DMS was used to capture transaction logs to implement CDC
• The data from raw zone was transformed using EMR transformation jobs and moved to S3 curated zone
• EMR jobs were used to transfer data from curated S3 zone to Amazon Redshift
• Data from Redshift is utilized by business for dashboards and performing adhoc queries for reporting

Features of the tool

• Scalability and Robustness
• Data Governance and Consistency
• Real-time Decision Making
• Real-time Decision Making