Executive summary
At the height of the COVID-19 pandemic, diagnostic infrastructure needed to scale rapidly to track and classify SARS-CoV2 viral lineages.
Zelytics partnered with a leading diagnostic firm to build a fully automated system capable of delivering SARS-CoV2 lineage classification in under 30 minutes.
This system processed lineage classification data from millions of COVID-19 tests and tens of thousands of viral genomes generated by Illumina Next Generation Sequencing.
In just one month, Zelytics designed and deployed a bioinformatics pipeline using Nextflow, integrated with an event-driven AWS cloud architecture to handle terabytes of data and ensure rapid scalability.
The system was built to be HIPAA/HITECH compliant and fully auditable, offering robust security and compliance for the healthcare industry.
Key outcomes included:
Rapid Deployment: A fully operational pipeline for SARS-CoV2 lineage classification was built in under 30 days*, with the core pipeline developed in just one week.
Ultra-Fast Classification: The system achieved lineage classification in under 30 minutes, processing data from millions of COVID-19 tests and over 50,000 viral genomes.
Scalable AWS Architecture: Leveraging AWS Lambda, ECS, SNS, and Batch, the infrastructure supported terabytes of genomic data with an event-driven model that scaled effortlessly.
Cost Efficiency: By automating the entire workflow, the client significantly reduced operational costs associated with manual data processing and reporting.
Governance & Audibility: The entire infrastructure was managed through infrastructure-as-code, centralized in one repository, simplifying compliance and future expandability.
*Curious about how these results were measured? Reach out to learn more about the technologies and processes we used.
This system allowed the diagnostic firm to classify SARS-CoV2 lineages with unparalleled speed and precision, positioning them to scale their response during the global pandemic.
Context & Background
During the global COVID19/SARS-CoV2 pandemic it was of paramount importance to rapidly scale diagnostics and public health monitoring capabilities.
As part of these efforts, collaborations with large players in the diagnostic testing space together with the research community and government had a mission critical role to set up this infrastructure at breakneck speed.
In this case study, we describe how Zelytics has set up infrastructure for ultra-rapid turnaround (<30 min) SARS-CoV2 lineage classification feeding from millions of COVID19 tests and tens of thousands genomes generated by Illumina Next Generation Sequencing.
Scope
Client had zero code and needed a system set up in less than a month.
Workflow had to support very high scalability (terabytes of data) for a rapid turnaround time (data -> interpretation < 30 minutes).
Workflow had to be fully automated and event-driven from end-to-end
Infrastructure should be compatible with bioinformatics tooling and should be highly extendable and user friendly for bioinformaticians
Infrastructure must be fully HIPAA/HITECH compliant and must be fully auditable
Solutions
Objective: Set up bioinformatics pipeline for COVID19 Lineaging with Nextflow
We quickly identified Nextflow as the most promising workflow manager to develop this workflow.
Next, we set up a simple bioinformatics pipeline which would handle all the classification steps.
This was completed within a week.
Objective: Set up AWS infrastructure to handle automation, scalability and event-driven architecture
After review with the bioinformatics team at the client, Zelytics moved rapidly to develop the cloud architecture to handle the requirements.
We had set up a very simple architecture leveraging AWS Lambda ,AWS SNS AWS batch, , AWS ECS and more for event-driven triggering of workflows scaling the compute jobs and automated email reporting for internal stakeholders and sharing of results with the GISAID database for external use.
This was completed in a week.
Objective: Bring all infrastructure into a single repository and manage infrastructure as code
With the full workflow in place and in production, Zelytics was able to focus on simplifying governance.
Zelytics proceeded to bring all the code into a single, easy-to-manage infrastructure as code repository fully managed through open source tooling.
This was completed in two weeks.
Closing remarks
Are you facing similar compliance challenges in your organization? Healthcare, finance, you name it. At Zelytics, we have dedicated consultants to set up comprehensive data governance solutions for your company.
Zelytics offers a complimentary consultation to help you gain clarity around your main challenges and develop a data-driven strategy to overcome them.
Let’s talk and get to know each other and see what we can do for your business.
Comments