Executive summary
As data-heavy organizations face rising costs and limitations with managed SaaS solutions like dbt Cloud, many are seeking more flexible and scalable alternatives.
This case study explores how Zelytics partnered with a high-growth tech company to successfully migrate from dbt Cloud to dbt Core, delivering cost savings and improved flexibility.
Zelytics implemented a tailored development environment using a custom VScode devcontainer, enabling the client's data team to work locally, integrate with their company VPN, and leverage essential tools like lineage tracking and database querying.
The migration was more than a simple transition - Zelytics introduced key enhancements, including a custom CLI wrapper to enforce best practices and a YAML linter to ensure configuration consistency across environments.
The key outcomes were:
Cost Reduction: The migration from dbt Cloud to dbt Core reduced licensing and operational expenses by >$100k*.
Improved Development Efficiency: Streamlined processes allowed for faster, more efficient project workflows, reducing development time by 30%*.
*Curious about how we achieved these improvements? Reach out to us, and we’d be happy to share more details.
This migration unlocked new opportunities for the client to optimize their data workflows, reduce costs, and improve productivity—without sacrificing future scalability.
Context & Background
With significant market penetration, there is no getting around data build tool (DBT) in the data transformation space. With large adoption across data engineers and analytics engineers, dbt will remain a staple for many companies' data teams for a while.
Managing how data transformations with dbt are handled in the data team are crucial. In high-growth companies, data teams want to move fast and not break things. Typically, many of these companies opt into managed SaaS solutions for DBT with the native DBT cloud being the most prominent choice.
However, there can be significant costs associated bringing these SaaS solutions on and the value they provide has continued to diminish as data teams and their requirements turn more heterogeneous.
Zelytics has performed countless “reverse migrations” migrating off from DBT cloud to DBT core. This case study dives deeper into a case study of a high growth technology company.
Pain Points & Challenges
Client was spending a significant portion of their budget on DBT cloud they wanted to get rid off
Client had multiple requirements for a DBT development environment that could not be met with DBT cloud
Client did not have the bandwidth or expertise in-house to develop a custom solution
Scope
Any new solution should at least match the capabilities of DBT cloud, most importantly, there should be a good lineage view and capability to query the database.
Solution needs to be compatible with either local machines or remote machines such as github codespaces
Setup and maintenance of the environment should be intuitive.
It should be easy to extend the capabilities of the development environment and steer dbt best practices.
Solutions
Objective: Set up a VScode dev container environment
Zelytics first started out by inquiring about the local machines used by the team and the security policies around them. Through several conversations across the organization, it emerged that a solution working locally was preferred.
For this, Zelytics has a ready-made, easy-to-use VScode devcontainer environment that allows to set up dbt with a connection to the data warehouse, support the company VPN and comes with several valuable tools out of the box such as lineage and a database connection for custom queries.
Over the course of the next few weeks, Zelytics was able to provide the development environment, documentation for the use and scheduled onboarding sessions with each team member to get them set up.
Objective: Customize the dbt command to enforce organizational patterns
After some working with the new environment, the client was able to execute the workflows they were running on DBT cloud within their local environment with ease. Excited for more, the client enquired if it was possible to encode some best practices to prevent overspending on building models in development.
Zelytics took this requirement and translated this into a custom, easily-extendable CLI wrapper for the dbt CLI. We were able to automatically configure dbt builds to use defer commands and block specific undesired practices such as building all models upstream for expensive model chains.
With some back and forths and experimentation, we were able to get to a pattern that significantly increased the efficiency of the development flow of the analytics engineers without additional overhead.
Objective: An enhanced DBT YML experience
Hungry for more, our client enquired about the possibilities for customizing the DBT YML file experience. Specifically, they were struggling with keeping their YAML files organized, on-pattern and in sync with their production environment. They had tried out some resources such as dbt-checkpoint, but were not satisfied with the extendability of that.
Zelytics took these requirements and extended the DBT development with several facets.
First, Zelytics introduced a custom YAML linter within the development environment that would automatically highlight YAMLS in real-time if they were missing configurations.
Second, Zelytics introduced a pytest framework which parsed the internal DBT manifests and allowed for setting up numerous custom testing solutions for a highly customized governance setup integrating straight into the organizations CI/CD process.
Closing remarks
Are you facing similar compliance challenges in your organization? Healthcare, finance, you name it. At Zelytics, we have dedicated consultants to set up comprehensive data governance solutions for your company.
Zelytics offers a complimentary consultation to help you gain clarity around your main challenges and develop a data-driven strategy to overcome them.
Let’s talk and get to know each other and see what we can do for your business.
Comentarios