How Scalapay, a Fintech Unicorn, reduced time to diagnose from hours/days to minutes
In the fast-paced fintech industry maintaining the accuracy and integrity of our analytics system is not just important—it's critical. As a data engineer at Scalapay, I've witnessed firsthand the challenges of scaling our data infrastructure to meet the demands of our rapid growth. Implementing Elementary into our analytics layer was a game-changer, transforming not just our data processes but how we trust and utilize data across the company.
The Challenge: Keeping Up with Rapid Growth
Scalapay, with its diverse, global team and rapid expansion, faced challenges in ensuring our data quality was reliable as we leveraged it to make decisions shaping our future. Our small but mighty data team, consisting of analysts and engineers, serves a company of over 200 people.
Our team was in a perpetual state of reactivity. We were caught in a constant loop of reactive measures—fielding 2-3 support tickets weekly, with stakeholders reporting issues like "help my customer retention dashboard looks wrong" or "I’m missing data to perform an analysis on our sales funnel".
Each ticket triggers a change in focus from the core value we deliver to the company as we now must identify and fix these issues on the backfoot. This reactive approach was not sustainable, especially as our company grows as does the volume of data we manage.
Integrating Elementary into our “clean” layer
The integration of Elementary was a strategic decision to address these challenges head-on. For Scalapay, our clean layer is a driver of value not just the analytics layer (silver and gold in the medallion architecture).
The clean layer is the foundation of our data system; business users are free to access it to perform their analyses, and it powers the entire analytics layer. For this reason, we chose to integrate Elementary here.
The results:
- With Elementary's seamless integration with dbt, we gained immediate visibility into our data as it arrived through the ingestion processes. This visibility is crucial for a fintech company like Scalapay, where real-time data accuracy can impact everything from customer transactions to compliance reporting.
- We integrated Elementary into our CI/CD process and execute tests on our data at regular intervals, to continuously monitor the results from the ingestion processes.
- Our key testing principles of quality, volume, schema, freshness, and lineage are all covered via Elementary – thanks to its seamless integration with dbt’s source freshness command.
- We also execute tests before any dbt run steps, to ensure we are not building any tables with poor-quality data.
- Another huge benefit is not having to connect to the database via another tool to query data - it seamlessly integrates via dbt and stores all the data in its own contained schema in Snowflake.
- The integrated alerting through Slack makes it very easy to quickly diagnose where the issue is coming from and understand what is impacted downstream.
Since implementing Elementary we've effectively reduced our support tickets from a few each week to zero.
The Impact: From Reactive Firefighting to Proactive Assurance
Since implementing Elementary we've effectively reduced our support tickets from a few each week to zero, a testament to the effectiveness of our new approach. This change has allowed our small team to focus on more strategic initiatives, improving our efficiency and reducing the time spent on troubleshooting.
This proactive stance means we can now inform stakeholders of potential data quality issues before they affect our reports and dashboards. In a fintech context, this ability is invaluable, ensuring our decisions are based on accurate and reliable data.
Beyond Data Quality: Building a Foundation for Future Innovation
For Scalapay, the journey with data observability and Elementary has laid a robust foundation for future growth and innovation. It's not just about solving today's problems but about anticipating the challenges and opportunities of tomorrow. As we continue to scale, the insights gained from this proactive approach to data quality will be instrumental in driving our data-driven culture.
In a broader sense, our experience with Elementary underscores a vital lesson for the fintech industry: investing in data quality is not an expense but a critical enabler of sustainable growth and innovation.
