Client purpose
Tradisa is a comprehensive multimodal logistics service provider with operations throughout Europe for the automotive sector and other industrial sectors.
Their goal was to fill their business with better data instead of just more data. One of their main issues was the data quality. They needed to generate a single source of truth (data warehouse) and consolidate, centralize, and clean all their information that was extracted from various sources.
- Industry: Logistics
- Solution type: Data warehouse.
- Tech stack: Azure, Databricks
How we navigate it
We proposed building a data warehouse in Azure to meet their goals. We worked with 2-week sprints. Each sprint there were new requirements.
To do the implementation, we took the following steps:
- Extracted data from sources using Azure Data Factory.
- Transformed data with Azure Databricks.
- Landed all data in a raw landing area.
- Staged data in transformation area.
- Built transformations to move data to consumption area.
Azure Devops was used to automate deployments between development, test and production environments.
To create the Data Warehouse, we followed the best practices including:
- Implementing immutable raw data storage.
- Lifecycle management from hot to cool storage.
- Optimized file formats like Parquet.
- Partitioning for performance.
- Independent ELT layers for resilience.
- Easy reload capabilities.
- Surrogate keys for reporting systems.
- Well documented code and processes.
- Flexibility to change loads and add new sources.
- Deletion detection from sources.
Where it took us
Our team executed an agile, iterative implementation approach to build Tradisa's new data warehouse architecture on Azure. We worked in 2-week sprints to extract data from sources, transform it, and load it into the data warehouse. This allowed us to continuously expand functionality and meet Tradisa's requirements. We set up dev, test, and production environments to manage the development lifecycle using Azure DevOps for CI/CD deployments.
Throughout the project, we focused on monitoring and optimizing data flows to ensure high performance and reduce data loading and transformation times. We also implemented quality controls for deployments to production.
The low cost of cloud storage allowed us to keep raw data intact for system resilience if needed. The use of cloud tools reduced maintenance costs and operating overheads while allowing for rapid scaling of resources if needed.
Client purpose
Tradisa is a comprehensive multimodal logistics service provider with operations throughout Europe for the automotive sector and other industrial sectors.
Their goal was to fill their business with better data instead of just more data. One of their main issues was the data quality. They needed to generate a single source of truth (data warehouse) and consolidate, centralize, and clean all their information that was extracted from various sources.
- Industry: Logistics
- Solution type: Data warehouse.
- Tech stack: Azure, Databricks
How we navigate it
We proposed building a data warehouse in Azure to meet their goals. We worked with 2-week sprints. Each sprint there were new requirements.
To do the implementation, we took the following steps:
- Extracted data from sources using Azure Data Factory.
- Transformed data with Azure Databricks.
- Landed all data in a raw landing area.
- Staged data in transformation area.
- Built transformations to move data to consumption area.
Azure Devops was used to automate deployments between development, test and production environments.
To create the Data Warehouse, we followed the best practices including:
- Implementing immutable raw data storage.
- Lifecycle management from hot to cool storage.
- Optimized file formats like Parquet.
- Partitioning for performance.
- Independent ELT layers for resilience.
- Easy reload capabilities.
- Surrogate keys for reporting systems.
- Well documented code and processes.
- Flexibility to change loads and add new sources.
- Deletion detection from sources.
How to change an entire data strategy
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way. And above all, the one who steps aside when he knows that he is not helping you or that he will not be able to give you what you need.
I don't know if it is helpful, but somehow I saw on the web a visual explanation of the problem in the data/technology world (I don't know if with this metaphor) before showing how it is to work with us and finally, another visual explanation of the result.
Like a bridge over troubled waters
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way.
Client purpose
Tradisa is a comprehensive multimodal logistics service provider with operations throughout Europe for the automotive sector and other industrial sectors.
Their goal was to fill their business with better data instead of just more data. One of their main issues was the data quality. They needed to generate a single source of truth (data warehouse) and consolidate, centralize, and clean all their information that was extracted from various sources.
- Industry: Logistics
- Solution type: Data warehouse.
- Tech stack: Azure, Databricks
How we navigate it
We proposed building a data warehouse in Azure to meet their goals. We worked with 2-week sprints. Each sprint there were new requirements.
To do the implementation, we took the following steps:
- Extracted data from sources using Azure Data Factory.
- Transformed data with Azure Databricks.
- Landed all data in a raw landing area.
- Staged data in transformation area.
- Built transformations to move data to consumption area.
Azure Devops was used to automate deployments between development, test and production environments.
To create the Data Warehouse, we followed the best practices including:
- Implementing immutable raw data storage.
- Lifecycle management from hot to cool storage.
- Optimized file formats like Parquet.
- Partitioning for performance.
- Independent ELT layers for resilience.
- Easy reload capabilities.
- Surrogate keys for reporting systems.
- Well documented code and processes.
- Flexibility to change loads and add new sources.
- Deletion detection from sources.
How to change an entire data strategy
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way. And above all, the one who steps aside when he knows that he is not helping you or that he will not be able to give you what you need.
I don't know if it is helpful, but somehow I saw on the web a visual explanation of the problem in the data/technology world (I don't know if with this metaphor) before showing how it is to work with us and finally, another visual explanation of the result.
Like a bridge over troubled waters
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way.
.png){kind=logo}
.png){kind=small}
