DWEL Scope Statement
Purpose of this Document
This document provides guidelines about information coverage (subject matter, grade level, type of resources) and the selection and review criteria for the Digital Water Education Library (DWEL) collection. This collection is aligned with the DLESE Collections Scope and Policy Statement (http://www.dlese.org/documents/policy/CollectionsScope_final.html) as well as the DLESE Accessioning and Deaccessioning Policy (http://www.dlese.org/documents/policy/collections_accession.html) with regard to required metadata and metadata framework, copyright policy, language and overarching quality filters, however these criteria are extended to offer only exemplary, peer-reviewed resources, as detailed below.
The Digital Water Education Library (DWEL) is a thematic collection focusing on the science, economics and policy issues of water (http://www.csmate.colostate.edu:16080/dwel/). As an educational tool it is designed to facilitate learning about all aspects of water in the Earth system. Primary audiences include K-12 and informal education. This collection is intended for dissemination via the Digital Library for Earth System Education (DLESE, http://www.dlese.org), and the National Science Digital Library (NSDL, http://www.nsdl.org). The DWEL collection supports science education standards and inquiry-based learning and has been created by a select and diverse group of practicing educators to ensure excellence and relevance to a wide range of learning environments.
The scope of the DWEL collection is water in the Earth system, with particular emphasis on interdisciplinary topics that integrate science, policy and economics. The collection seeks resources that bring exemplary water related resources into classrooms or other learning sites, and which demonstrate the application of science to solving real world problems. The collection favors resources that support the teaching, learning and assessment of core water-related concepts and issues, promote inquiry into diverse aspects of the Earth system, and align with science education standards. The collection also favors resources that are well-documented, easy to use, bug-free, motivational for learners, pedagogically effective, scientifically accurate, and which foster mastery of important scientific and mathematical understandings and technology skills.
The types of materials to be collected include:
All resources in the DWEL collection have been reviewed by teachers and science educators. The rationale for establishing a reviewed collection is to help library users quickly find exemplary teaching and learning materials relating to the core concepts of water. Several selection criteria are used to judge resources for inclusion into this collection of exemplary resources:
Note that various supplementary resources (such as images and data sets) may not meet all of the criteria outlined above as stand-alone resources yet provide important support for the learning activities. Furthermore, different types of resources (text, visualizations, classroom and lab activities) each have their own additional set of secondary criteria that help to distinguish them as exemplary. To accommodate this, additional criteria for judging each resource type have been developed. The documentation for these criteria was developed and reviewed by the DWEL teachers and staff (it is no longer available online).
I. Structure of Working Groups
All of the discovery and evaluation work on the DWEL project was conducted by four teams tasked with selecting and cataloging resources. Twenty classroom teachers and informal setting educators were selected to participate. Their qualifications included extensive experience teaching in formal or informal settings, a strong professional interest in water-related education, a high level of computer comfort, leadership skills, and geographic distribution. By selecting a diverse set of teachers, project leaders hoped to achieve a broad representation of educators.
The target audiences served by this project are sufficiently different to warrant their own distinct working groups. Formal education was broken into three groups covering K-4, 5-8, and 9-12, and a fourth group was established to specifically address the informal educational setting. Each group was composed of four to six educators who work with their target audience.
Each of the three grade-level groups had a single person designated as Group Leader. The Informal Setting group shared this leadership position between two people. The full-time manager of the project was in charge of all operational aspects of the project. The part-time project director was responsible for providing guidance to the project and served as the liaison for communication with NSF, other NSDL projects and the DWEL Advisory Board. Other DWEL personnel were responsible for the day-to-day communications, database and website management for the project.
II. Topic and Resource Selection Process
Early on in the DWEL project it was determined that trying to simultaneously address all aspects of the water theme for the collection was not realistic. Consequently, the overall theme of water was broken down into several sub-themes, each of which was worked on exclusively for a 4-6 week period. We called each period a campaign and a complete list of all campaigns can be found in Figure 1. The activities and decision processes for each campaign have been broken down into a number of discrete steps that have been formalized in the DWEL Work Hub (http://22.214.171.124/dwel/workhub.html). The link "Click HERE for the DWEL CALENDAR" at the top of the DWEL Work Hub provides a calendar that outlines the flow of individual work elements for each of the campaigns.
Long before a campaign begins, the project managers prepare a concept map of the terms and features of a campaign’s theme (see Figure 2 for an example). The purpose of the concept map is to have a graphical representation of key ideas that the collection should address. In the initial search, evaluation and cataloging stage of the project the concept map serves as a guiding document to focus the work of the project staff.
DWEL resources are identified and cataloged by practicing teachers from K-12 schools and science educators from informal setting organizations. Four working groups represent the teachers and learners in grades K-4, 5-8, 9-12 and the informal education setting. Group members move through a prescribed series of steps that are accessible through the DWEL Work Hub (http://126.96.36.199/dwel/workhub.html).
At this step, members of each working group examine the concept map and consider the National Science Education Standards that are relevant for that group. With the standards in mind, members of the working groups “vote” for the top 15 topics in a campaign that are most relevant for their target audience. The votes from each working group are compiled separately to generate a list of 10 most relevant topics that would direct the subsequent work of each group. Given that the audiences and standards are very different for each group this step leads to widely varying topic lists. Figure 3 shows the different sets of top 10 topics for the four working groups for the Water in the Atmosphere campaign.
The collection is organized by key water themes that are presented as concept webs which enumerate the supporting concepts and content of that theme. These include Groundwater, Surface Water, Oceans, Water and Life, Water Quality, Water Use, Properties of Water, Water in the Atmosphere, Water in the Solar System, Water in the Cryosphere, and Water Policy and Economics. More themes may evolve as the project continues. Working groups identify 10-12 concepts that are most relevant to their audience, using their personal expertise and national science education standards as a guide. Resources that focus on each of these topics are then identified by the group members and are distributed across these topics in a coordinated fashion.
III. Web Mining
Before working group members set off to locate potentially useful resources for a campaign it is important to ensure they are not duplicating each other’s efforts. At this step each person “stakes a claim” to three topics from their group’s list. When subsequent group members go to this step they can see the updated list of who is doing what, and can then stake their claims so as to avoid duplication of efforts.
IV. Internal Review
A comprehensive review system has been developed for this project which involves a set of standardized questions that provides a first cut in bringing the higher quality resources to the surface. Although the review criteria for this collection (Rapid_Review_Worksheet.pdf) do not follow the exact wording of the seven selection criteria outlined in the DLESE Collections Scope and Policy Statement, they do follow them in spirit. The seven criteria (listed below) have been matched to the DWEL review criteria.
The “rapid review” submission process automatically records and archives the review data by working group and individual group member. Each group member also conducts an independent review of at approximately 5 - 10 resources that have been submitted by fellow group members. Review data are then compiled and sites are given a composite score that helps to determine whether the site is "exemplary" and should be included in the DWEL collection. These resources then move forward and are cataloged using an instance of the DLESE Catalog System hosted by the DLESE Program Center.
DLESE Collections Scope and Policy Statement Selection Criteria (Matching DWEL Criteria)
1. Scientifically accurate (2, 3, 17)
2. Importance or significance (6)
3. Pedagogical effectiveness: has student learning occurred? (7,9,14,15,18,19)
4. Well-documented (3, 23)
5. Ease of use for students and faculty (11, 21, 22, 23, 24, 25)
6. Power to inspire or motivate students (10, 12, 13, 25)
7. Robustness as a digital resource (4, 16, 20)
DWEL Review Criteria (Matching DLESE Selection Criteria)
1. Characterize the resource type and its components (N/A)
2. Does the accuracy of the scientific content provided through the resource appear to be acceptable? (1)
3. Is the resource from a credible source? (1, 4)
4. Is the resource well written and appropriate for your target audience? (7)
5. Was the speed at which the site loaded on your equipment acceptable? (7)
6. Does the resource address one or more National Science Education Standards? (2)
7. Is there adequate coverage of the content presented in the resource? (3)
8. What is the geographic focus of the resource? (NA)
9. Can the material in the resource be transferred or applied to other content areas? (3)
10. Does the resource present information in an inquiry-based (engaging hands-on or minds-on) way? (6)
11. Does the resource have challenging, but doable activities? (5)
12. Does the resource have ties to other subject areas such as social studies, history, etc.? (6)
13. Does the resource deal with issues, problems and/or situations relevant to the students' lives? (6)
14. Can the materials in the resource be adapted to different instructional methods and/or instructional settings? (3)
15. Does the resource present the information in a way that would be compatible to users with different learning styles? (3)
16. Does the resource take advantage of the unique characteristics and potential of the web medium? (7)
17. Is the site free from any obvious bias? (1)
18. Is the resource sensitive to cultural and diversity issues? (3)
19. Is there any advertising on the site? (3)
20a. Is the site currently maintained and updated? (Are there hardly any bad links or out-dated references?) (7)
20b, Does it appear that the site will continue to be maintained and updated? (See whether the site offers the date of site launch, the date of last update and/or info about how often the site is checked and updated.) (7)
21. Can visitors find the primary information they need within 3 or fewer clicks? (5)
22. Is the site easily navigated by its intended audience (5)
23. Does the resource employ any of the following tools? (Check all that apply - a consistent menu bar, site map/index, internal search engine, a means for user support, online message boards/chat rooms) (4,5)
24. Is attention paid to the needs of users with disabilities or special needs (i.e.: text-only, frames, audio and/or large print versions of the resource, good use of white space, good color contrast between elements, graphics labeled with alternative text, etc.)? (5)
25. Do the graphics/art serve to add value to the site? (5,6)
Every resource considered for inclusion into the DWEL collection has an original review by the person who first identified the site. Subsequently, each site also has a minimum of one secondary review that serves to validate (or refute) the original review. Another validation of the resource comes from voluntary reviews from our user community. The DWEL web site (http://www.csmate.colostate.edu:16080/dwel/) has a page where members of the educational community can submit reviews of resources using the same criteria employed by the DWEL working groups.
Because the collection has many types of resources (text, labs, activities, graphics, animations, simulations, etc.) one set of criteria cannot be uniformly applied to every resource type. While all sites in the DWEL collection are related to central water concepts, easily navigated by their intended audience, and are scientifically accurate, not every site can directly address National Science Education Standards (data sets and imagery, for example). DWEL resources all have the capacity to promote sound teaching and/or learning to the standards, but they may not specifically address the standards in the same way as a classroom lab or field activity.
Consequently, the DWEL Exemplary Criteria have two levels of review for each type of resource. Each resource type has several primary criteria that must be met to be considered for inclusion into the collection (DWEL criteria 1 through 5). However, once the working groups determine that the initial conditions are met; additional sets of secondary criteria for each resource type are used to help further identify sites that might be considered the “best of the best.” These criteria range from “the site doesn't take forever to load,” to “the site is interdisciplinary,” “the site promotes critical thinking,” and “the site provides relevant world issues or situations.” As with the primary criteria, different resource types have their own set of secondary criteria.
V. Cataloging Assignments
All review data are automatically entered into a database, with numerical scores for each criterion. At the end of the review period for each campaign, members of the DWEL management team put the data into a spreadsheet which automatically tabulates the total score for each review. The spreadsheet is then sorted by URL, and the total scores for each unique URL are then averaged (see Figure 4 for an example of the final scoring matrix). URLs that achieved an average score above a certain threshold are then deemed worthy of cataloging into the DWEL collection. Typically, resources with an average score of 75% or above were considered worthy of inclusion. On the other hand, the minimum threshold values varied with resource type. Imagery and data sets, by their very nature, obviously would fail to score points on many of the DWEL review criteria (e.g. addressing National Science Education Standards, having doable activities, adaptable for different learning styles).
If there were a number of resources that essentially addressed a single topic for a given campaign, the resource(s) with the highest average scores are the ones which are considered the “best of the best” and recommended for cataloging. In all cases, the managers of the DWEL project made the actual decisions of which resources should be cataloged. The DWEL Work Hub facilitated the asynchronous workflow of the many people who contributed to this project, and provided a mechanism for creating a database of all of their reviews.
VI. Catalog Tool & Project Process & Database Search
Both of these steps are designed to help the workflow. The first simply allows the user easy access to the DWEL cataloging tool. The second is a tool that allows the user to check on the progress of the project by examining the project database through various search parameters (e.g. by campaign, the cataloger, etc).
VII. Cataloging Protocol
The overall cataloging philosophy is to describe resources with sufficient detail to support discovery, comprehension and reuse.
Community Contributions: As a community-based collection project, DWEL encourages and depends upon community contributions of resources for consideration. Community members also participate in the review of those resources. Resources may find their way into to collection through two paths. The primary path is through the efforts of the four working groups of K-4, 5-8 and 9-12 teachers and informal science educators. The second path is through recommendations that come from the larger K-12 and informal setting user audiences. Recommendations are submitted to DWEL Project Manager, Bryan Aivazian, at firstname.lastname@example.org. Determining the suitability of the resources, and the subsequent cataloging of those determined to be exemplary, is solely the responsibility of the working groups.
Granularity: In addition to providing access to resources of exemplary quality, a goal of the DWEL collection is to ensure that the collection is populated with items that specifically address the many particular components of the core concept maps. When a teacher or learner conducts a search for a specific item, the resulting list of records should be both relatively small and highly focused. As such, our approach is to catalog at a fine level of granularity, carefully cataloging the smaller specific items within larger sites that meet the exemplary criteria.
Resource Description: All resources in the DWEL collection will be cataloged with the required DLESE metadata using the accepted DLESE metadata framework, currently DLESE-IMS. We will transform our records to the ADN framework when it is available, and will keep in step with framework changes as they evolve. A complete description of DLESE's required metadata is available at: http://www.dlese.org/Metadata/adn-item/0.6.50/index.htm#required. Resources also will be cataloged with additional metadata on science and geography educational standards where appropriate.
VIII. External Review
Once cataloged, resources are then displayed on the DWEL project site for the educational community to examine and use in a variety in classroom situations. The public also has access to a review page that provides the same review criteria as used by the working groups, plus additional fields for user demographics. The result is an external review process that complements the internal DWEL review. Review data and comments will be retained, and will assist in the cataloging process, such that the appropriate audience is noted, and the description will note the exemplary aspects of the materials.
IV. Filling in the Gaps
In the later stages of the project the concept webs served as one criterion by which the success of the project was measured. Formative assessment of the collection was conducted by comparing the cataloged sites to the categories on the original concept webs. This process is called a gap analysis. While there were many instances in which an exemplary resource was not found for every category identified, efforts were made to seek as wide a balance of topics possible. Attempts to fill in “holes” was conducted by revisiting each campaign, seeking out assistance from the DWEL Advisory Board, and through directed appeals for assistance from the larger DLESE user community and professional working the areas of each campaign topic.
Work is currently underway to fill in Gaps in the DWEL collection by soliciting input from scientific and education organizations and individuals with expertise in specific water concept areas. For example, Ocean.US, members of the National Marine Educators Associtation, and participants in Project Maury have all been sent invitations to act as reviewers of DWEL-Ocean resources, to suggest resources to fill in Gaps in the collection, and to suggest additions of other high-quality ocean resources that our initial collection efforts did not cover.
In terms of the DWEL metatdata, all metadata is copyrighted as 2005 University Corporation for Atmospheric Research (UCAR), but metadata may be used as long it is consistent with the previously established DLESE Intellectual Property Policy, which can be found at: http://www.dlese.org/documents/policy/DLESE_IP_approved.html. This allows for the redistribution of metadata records. Individual metadata records are available for harvest from DLESE.
For the DWEL collection, quality assurance has two issues: one for the resources in collection and one for the metadata records of the collection. By definition, the resources in the DWEL collection are very high-quality because they have gone through a rigorous review process. A formal set of review criteria have been developed for the DWEL project and are applied uniformly to all collection items as described above. The metadata records are generated by DWEL working group members and DWEL staff. Each are reviewed to ensure that minimum metadata requirements are met, for proper spelling and grammar, and overall data integrity. All resource catalogers submit their initial few metadata records for review to the DPC’s Coordinator for Metadata Operations and Outreach for quality review and formative feedback. After the initial period of training all working group members submit their metadata records to the DWEL project manager for quality assurance review. However, all unusual or nonstandard cases are referred back to the DLESE Coordinator for Metadata Operations and Outreach for final a ruling. This process will facilitate the creation of metadata records with a high degree of consistency between individual catalogers and reflect the best practices in resource cataloguing as defined by DLESE.
The collection is expected to continue growing throughout the life of this project. Currently, funding is available through until the end of December 2005. Is not anticipated that additional reviewed resources will be added to this collection after this date. Furthermore, after this date there will be no special maintenance activities for the collection other than what would be expected for any resource in the overall DLESE collection. However, resources may be deaccessioned by DLESE over time if they no longer are accessible or meet accession criteria. Furthermore, the DPC also has permission to modify DWEL records as it sees fit to maintain the integrity of the metadata (e.g. to correct a URL that has changed).
The pathway into the DWEL reviewed collection is very well defined and is designed to identify resources of extremely high quality that meet specific identified needs within the larger framework of water in the Earth system. This highly structured pathway has evolved through the first year of this project. Since this process was successful in setting the workflow and providing the mechanism for making decisions, it did not change for the remainder of the project. Once the project is complete this pathway will exist only in theory since the work will end. However, the pathway to this reviewed collection may be adopted for work in generating other reviewed collections. adopted for work in generating other reviewed collections.DRAFT
Last modified: June 29, 2005
Previous modifications: May 20, 2003, April 21, 2003, January 30, 2003 November 26, 2002
Figure 1. The DWEL Collection Campaigns.
Figure 2. Water in the Atmosphere Concept Web.
Figure 3. Examples of "Top 10" topics for the Water in the Atmosphere Campaign.
Figure 4. Example of the final scoring matrix for DWEL reviews.
Last updated: 12-11-03