A process of safely transitioning a data quality flow for a complex architecture
The challenge
Isdin, a global Spanish pharmaceutical company, needed to centralize data validation for information coming from multiple countries and teams. They were using Soda, but this was resulting in considerable costs. The goal was to find the right tool to centralize quality checks, detect errors fast, and deliver actionable alerts.
- Industry: Pharmaceuticals
- Solution type: Data quality
- Tech stack: Soda, Great Expectations, Google Cloud Platform
What we did
First, we benchmarked the approach and proposed alternatives. Our recommendation was moving their data quality process to Great Expectations, because of its version control capabilities.
Then, our team:
- Validated the migration with a PoC in record time: We kicked off a proof of concept to estimate and port the required checks. During the process, we implemented the checks within the expected window and scheduled a parallel run where both tools operated side-by-side to avoid risks, with strong results.
- Embedded the new tool in the client’s tech stack: We integrated Great Expectations with Isdin’s Google Cloud Platform environment.
- Rerouted alerting for richer contexts: Alerts that were previously viewed in Soda and sent via Slack now land directly in GCP, providing richer metadata and faster automation.
- Rolled into production: As Great Expectations moved into production, pipelines continue in Soda for a limited time to provide extra safety during the transition.
What we achieved
- Centralized and consistent data quality framework: This reduces the risk of errors and builds trust in future analytics.
- Clearer alerts for faster actions: Alerts in Slack carry richer context now, so the team can take deliberate actions based on the type of error.
- Cost optimization: Moving to Great Expectations reduced total cost of the Soda service.
- Low-risk migration: The parallel run ensured consistency and a safe transition.
A process of safely transitioning a data quality flow for a complex architecture
The challenge
Isdin, a global Spanish pharmaceutical company, needed to centralize data validation for information coming from multiple countries and teams. They were using Soda, but this was resulting in considerable costs. The goal was to find the right tool to centralize quality checks, detect errors fast, and deliver actionable alerts.
- Industry: Pharmaceuticals
- Solution type: Data quality
- Tech stack: Soda, Great Expectations, Google Cloud Platform
What we did
First, we benchmarked the approach and proposed alternatives. Our recommendation was moving their data quality process to Great Expectations, because of its version control capabilities.
Then, our team:
- Validated the migration with a PoC in record time: We kicked off a proof of concept to estimate and port the required checks. During the process, we implemented the checks within the expected window and scheduled a parallel run where both tools operated side-by-side to avoid risks, with strong results.
- Embedded the new tool in the client’s tech stack: We integrated Great Expectations with Isdin’s Google Cloud Platform environment.
- Rerouted alerting for richer contexts: Alerts that were previously viewed in Soda and sent via Slack now land directly in GCP, providing richer metadata and faster automation.
- Rolled into production: As Great Expectations moved into production, pipelines continue in Soda for a limited time to provide extra safety during the transition.
How to change an entire data strategy
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way. And above all, the one who steps aside when he knows that he is not helping you or that he will not be able to give you what you need.
I don't know if it is helpful, but somehow I saw on the web a visual explanation of the problem in the data/technology world (I don't know if with this metaphor) before showing how it is to work with us and finally, another visual explanation of the result.
Like a bridge over troubled waters
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way.
A process of safely transitioning a data quality flow for a complex architecture
The challenge
Isdin, a global Spanish pharmaceutical company, needed to centralize data validation for information coming from multiple countries and teams. They were using Soda, but this was resulting in considerable costs. The goal was to find the right tool to centralize quality checks, detect errors fast, and deliver actionable alerts.
- Industry: Pharmaceuticals
- Solution type: Data quality
- Tech stack: Soda, Great Expectations, Google Cloud Platform
What we did
First, we benchmarked the approach and proposed alternatives. Our recommendation was moving their data quality process to Great Expectations, because of its version control capabilities.
Then, our team:
- Validated the migration with a PoC in record time: We kicked off a proof of concept to estimate and port the required checks. During the process, we implemented the checks within the expected window and scheduled a parallel run where both tools operated side-by-side to avoid risks, with strong results.
- Embedded the new tool in the client’s tech stack: We integrated Great Expectations with Isdin’s Google Cloud Platform environment.
- Rerouted alerting for richer contexts: Alerts that were previously viewed in Soda and sent via Slack now land directly in GCP, providing richer metadata and faster automation.
- Rolled into production: As Great Expectations moved into production, pipelines continue in Soda for a limited time to provide extra safety during the transition.
How to change an entire data strategy
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way. And above all, the one who steps aside when he knows that he is not helping you or that he will not be able to give you what you need.
I don't know if it is helpful, but somehow I saw on the web a visual explanation of the problem in the data/technology world (I don't know if with this metaphor) before showing how it is to work with us and finally, another visual explanation of the result.
Like a bridge over troubled waters
I imagine 8wires as that sincere and honest partner that takes you out of all that noise and helps you focus on what is important, no matter how unsexy it may be, to achieve a great long-term goal.
The one who accompanies you through the hard times and helps you through the tough decisions, knowing that there is no easy road. The one who gives you the push or the tools so that you climb and be yourself the one who reaches the summits you propose in a healthy, sustainable and energetic way.
.png){kind=small}