The ability to comprehend and articulate the big picture necessitates constructing and representing the context, content and relationships from multimedia information sources. To address this challenge, we have created a visual analytic tool suite that incorporates the latest state-of-the-art multimedia characterization, retrieval, filtering, and classification technology to provide end users with a holistic approach to discovery and detection of relationships across the multimedia landscape. This suite will provide a broader picture of relationships, thereby minimizing the user’s workload and ultimately the effort of identifying the critical intelligence for decision makers. Our approach addresses the critical need for a next-generation content analysis system for modern and emerging media.


Canopy is a visual analytic software suite of novel interactive visualizations that support the analysis, discovery, and global overview for mixed media data. A companion tool, Savvy, is an analytic system designed to support investigation of large video collections. Both Canopy and Savvy have been developed to support pluggable interfaces and are currently utilizing the capabilities of Lighthouse. Lighthouse is a multimedia content analysis and retrieval library developed for processing images, videos and audio. These tool suites provide users a visual perspective that enable the ability to detect and discover relationships within mixed media data.

Canopy and Savvy have been developed as enterprise systems supporting the need for scalable and federated data management and indexing systems, while enabling third parties to plug in services that will enhance the analytic platform. Canopy examines the user’s data and provides visual perspectives through the analysis processes of document extraction, characterization, and content and feature associations. We normalize and decompose multi-modal content into base types and use type-specific analysis while maintaining context to the original document. A series of state-of-the-art algorithms for text, images, video, and audio are applied to identify the critical features of the content. From these features, trained classifiers help populate categories for the purpose of filtering and sorting. To truly create a holistic analytic environment, we associate our features, concepts and annotated tags within and across the user’s data providing a unified space for analytics.

Extracting and running analytic algorithms are just the beginning; the Canopy suite was developed with user-centered design practices that support the analytic and decision-making process through innovative user interfaces and interactive visualizations. One of the strengths of the software is its ability to provide a high-level understanding and summarization of the data in a visual way that is both informative and appealing. This understanding is further enhanced by allowing the user to pivot, arrange and relate the data by various attributes and metadata, thereby revealing relationships
that previously were not possible to detect.

These visualization and interaction techniques allow users to gain insight and understanding of large, diverse sets of multimedia without taxing their cognitive state with the information overload that is intrinsic with other systems in use today.


The Canopy suite is positioned to fundamentally change the landscape for how multimedia analysis is performed. The combination of its enterprise platform, its powerful extraction and analytic algorithms and its user interface and interactive visualizations provide a broader picture of the information and its relationships, and thus allow the user to gain the insight needed for more productive and efficient decision making.


Deborah Payne
Pacific Northwest National Laboratory
P.O. Box 999, MSIN J4-32
Richland, WA 99352
(509) 375-2904
debbie.payne at

Shawn Bohn
Pacific Northwest National Laboratory
P.O. Box 999, MSIN J4-32
Richland, WA 99352
(509) 375-2574
shawn.bohn at

Share this article: Download this article as a PDF