Learn More

• FlowJo.com
• The Daily Dongle
• FlowDx Blog
• MyCyte.org

NIH Grant
#5R44RR024094-03

Table of Contents
Introduction
Project Plan
Methods
 Use Cases
 Algorithms
 Metrics
 Infrastructure
Results
Discussion
References

1.1 Gather stakeholders (collaborators, team members, support) and describe roles (sets of related skills, competences, and responsibilities).
Work Products: Collaborator List and Letters of Support for Phase I Grant
1.2 Set scope and goals – Approved Phase I Grant Application
1.3 Funding – SBIR Grant #1 R43 RR024094-01A1 through NIH: NIH Approval Letter
The Inception Phase ends when the Lifecyle Objective Milestones are met.

2.1 Project Plan using RUP for project management, planning, and tracking.
2.2 Workflow Document - Describes the workflow of the repository, DB, and utilities.
Work Product: Diagram and documentation of the repository, utilities, and required XML in the workspaces.
2.3 Collaborators - List of Collaborators
2.4 Prelimimary Quality Assurance Plan
2.5 Use Case Studies: Describe the Use Case Using MIFlowCyt Standards
2.6 Algorithm Analysis
2.7 Metrics - Initial Evaluation of comparison metrics
2.8 Apply for Phase II funding. Phase II Grant Application Documents
The Elaboration Phase ends when the Lifecyle Architecture Milestones are met.

Overall this phase will encompass creating the FlowDx file repository, loading data into the FlowDx database, running both manual gating and automated classification analysis with the algorithms on all the use case data, and evaluating populations with all the comparison metrics. This will occur through several iterations, with the result that all the use case data is evaluated with all algorithms and all comparison metrics to be able to fill out the three-dimensional matrix of use case data x algorithm x comparison metric. Algorithms and metrics will have to be refined and possibly combined to fine-tune the automated analysis of each use case. This is covered in more detail in the Construction Phase Iteration Plan

How do we define the endpoint of success for the automated analysis of each use case? We still need to define what determines success (validation, quantitative way, metric value, collaborator approval).

3.1 Infrastructure Tool Refinement - This has turned out to be a very significant portion of the FlowDx project due to the large amounts of data and time required for doing manual analysis. Build the working version of the FlowDx file repository and database with all the utilities that will interface with FlowJo and the users.
Work Products: Working Version of the DB, repository & utilities with Specification, Code, Schema, Location, Contents and Status, GUI to view result reports, and SOPs for use and training documents.

3.2 Evaluating Human Analysis - Evaluate whether there is low enough variability within the group of experts comprising the consensus to have a valid consensus as agreed upon by the PI, collaborators, and the project team.
Work Product: Collaborator, PI, and project team approval of the consensus gates defined by the group of experts manually gating the use case data.

3.3 Completing Gate-a-thon - Have experts manually gate the whole set of use case data for each use case. This will provide the gold standard to compare the automated classifications against. This step will be done only after the contributing scientists have evaluated the initial set of manually gated results, consensus gates, and automated classifiers and have approved of the level of variability and the SOP for gating. This will be using the FlowDx repository and database and the utilities required to automatically save FlowJo workspaces using special FlowDx XML to the FlowDx repository and create records in the FlowDx Database. Potentially have the contributing labs have some technicians do the gating to contribute to the consensus.
Work Product: To have consensus populations for all samples for all use cases loaded into the repository from five experts, one of which should be from the contributing collaborator's lab.

3.4 Scaling up algorithmic analysis - Take the automated methods and create a way to crunch through the use case data without manual intervention. For ANN and SVM this will be done by writing R scripts to do the neural nets and submit the results directly to the FlowDx Database. For Magnetic Gating and Probability Bin Cluster Analysis this will be accomplished by setting up GatingML definitions and applying them to use case data through the FlowJo command line to analyze use case data and submit the results directly to the FlowDx database.
Work Products: Automated algorithms that will require no user interface except to select the set of FCS data files, AssayID, and algorithm to process the files.

3.5 Automate the application of all five comparison metrics (V-Measure, Mallows Distance, ROC, Misclassification Rate, PCA) to evaluate the performance of automated classification algorithms - Apply all metrics to each population of interest (popmask) in our use cases. Develop the workflow for applying and comparing metrics in a high-throughput environment. Develop the SOP for an experimenter to employ our experimental tool to select or craft a metric. Develop a training set generator to sort or filter events based on the agreement of multiple metrics.
Work Products: Refined Code for DB tools to apply all five comparison metrics to the popmask and consensus files in the FlowDx database, along with documentation and training documents.

3.6 Evaluating Algorithmic Analysis utilizing all five comparison metrics and biological relevance -
Work Product: Report of the algorithms results using all five comparison metrics and showing all the values for all samples and all target populations. Report showing the biological evaluation. Example is the Linear Spline analysis of the GvHD dataset or the timecourse analysis for the HIV data set. The biological evaluation report will vary with the use case.

3.7 Improving Algorithmic Analysis - Iteratively improve the algorithms to find the target populations with more-reproducible results. Implement flags for difficult samples that have strange distributions, such as from debris, wrong compensation, and other assay QC failures.
Work Product: Algorithms that meet the collaborator's approval for allowing automated analysis in lieu of manual analysis.

3.8 Statistical Analysis - Determine how to report the comparison metrics for each algorithm for each use case. Example: for the GvHD dataset, there are 4220 populations to combine into one result. How should we report the overall value and variance? How do we highlight outlier values?
Work Product: Collaborator, PI, and project team approval of the method of reporting results.

3.9 Filling up the matrix of results showing: use cases vs. classification methods vs. comparison metrics.
Work Product: Completed matrix of results showing which algorithms perform better than others for each use case and each comparison metric.

3.10 Collaboration Approval - Tree Star will send reports to collaborators at the end of each relevant iteration of the project. These reports may be tailored to each collaborator, but collaborators will have access to complete reports. Tree Star's goal is to have collaborators approve of our decisions before we progress to the next iteration of the project.
Work Product: Collaborator Approval and Feedback before entering the next iteration of the project.

3.11 Workflow Document - To more fully define the experimental protocol, whereby a researcher can compare two or more classifications of identical datasets to study the differences, biases, and effectiveness of human and algorithmic classifiers. Diagram and text describing how a new client or scientist would get FlowDx customized for their assay -- from approaching Tree Star to a working automated analysis system in their hands. FlowDx Collaborative Process Diagram is the prototype workflow that needs to be updated to reflect the complexity on the research side, such as purchasing department, QA, technicians, lab manager, legal department, and all other personnel. There should be two versions of the image, one for use case collaborators and one for FlowDx clients.
Work Product: Diagram and documentation to describe the process of starting with a client with a clinical assay and finishing with FlowDx doing automated analysis for this client. This will also include the Questionnaire based on MiFlowCyt for the potential clients to describe their assay methods, analysis, QC, and requirements for the FlowDx automated service.
The Construction Phase ends when the 2.1.5.3 Initial Operational Capability Milestones are met

4.1 Commercialization Plan – Revised and approved by Premarket Approval Committee
4.2 Quality Assurance Test Plan - Revised and including contemporaneous test results & approved by Premarket Approval Committee
4.3 FDA Premarket Approval –

1. Full reports of all information, published or known to the applicant, or which should reasonably be known to the applicant, concerning investigations that have been made. The purpose is to show whether or not such device is safe and effective.
2. Full statement of the components, ingredients, and properties, and of the principle or principles of operation, of such device
3. Full description of the methods used in, and the facilities and controls used for, the manufacture, processing, and (when relevant) packing and installation of such device
4. An identifying reference to any performance standard under section 514 which would be applicable to any aspect of such device if it were a class II device, and either adequate information to show that such aspect of such device fully meets such performance standard or adequate information to justify any deviation from such standard
5. Samples of such device and of components thereof as the Secretary may reasonably require, except that where the submission of such samples is impracticable or unduly burdensome, the requirement of this subparagraph may be met by the submission of complete information concerning the location of one or more such devices readily available for examination and testing
6. Specimens of the labeling proposed to be used for such device
   Certification required under section 402(j)(5)(B) of the Public Health Service Act [42 USC § 282(j)(5)(B)] (which shall not be considered an element of such application)
7. Other information relevant to the subject matter of the application as the Secretary, with the concurrence of the appropriate panel under section 513, may require.
   Work Product: Premarket Approval

4.4 Enterprise Architecture - Enterprise architects use various business methods and tools to understand and document the structure of an enterprise. In doing so, they produce documents and models, together called artifacts. These artifacts describe the logical organization of:

1. Business strategies
2. Metrics
3. Business capabilities
4. Business processes
5. Information resources
6. Business systems
7. Networking infrastructure

Work Product: A complete collection of these artifacts, sufficient to describe the enterprise in useful ways.

4.5 Security - 21CFR Part 11 Compliance and Prelim Security Analysis - PGP analysis
Work Product: FlowDx will use the signature and cryptographic services of data security leader PGP Corporation to implement 21 CFR Part 11 security requirements. PGP's so-called "weak link" license allows FlowDx to make JAVA calls to security services, including electronic signature and dynamic data security using public/private key encryption. FlowDx can be customized to provide security overall, or individual security elements missing from the user's workflow can be activated independently.

The Transition Phase ends with the 2.1.5.4 Product Release Milestone, transitioning the system from development to production, making it available to, and understood by, the end user.