Search Service API Documentation

Overview

Definitions and concepts

• Base URL - the base URL used to access the Web service. This is the portion of the request that precedes the request arguments. For example http://www.dlese.org/dds/services/ddsws1-1.
• Request arguments - the argument=value pairs that make up the request and follow the base URL.
• DDSWS response envelope - the XML container used to return data. This container returns different types of data depending on the request made.

HTTP request format

Service requests

http://www.dlese.org/dds/services/ddsws1-1?
        verb=GetRecord&id=DLESE-000-000-000-001

Search

• q - (query) an optional argument that may contain plain text or field/term specifiers. Boolean logic, field/term specifiers and boosting must be specified using the Lucene Query Syntax (LQS). Plain text terms (when no field is indicated) are used to search in the default field, which contains textual metadata extracted from the title, description, keywords, grade ranges, resource types and other areas of the repository metadata. See available search fields for detailed information about the fields that are available for searching.

• ky - (collection) an optional repeatable argument that limits the search to records that reside in the given metadata collection(s).

• gr - (grade range) an optional repeatable argument that limits the search to ADN records that contain the given grade range(s). These grade ranges are a DLESE-specific controlled vocabulary that is part of the ADN framework.

• re - (resource type) an optional repeatable argument that limits the search to ADN records that contain the given resource type(s). These resource types are a DLESE-specific controlled vocabulary that is part of the ADN framework.

• su - (subject) an optional repeatable argument that limits the search to ADN records that contain the given subject(s). These subjects are a DLESE-specific controlled vocabulary that is part of the ADN framework.

• cs - (content standard) an optional repeatable argument that limits the search to ADN records that contain the given content standard(s). These content standards are a DLESE-specific controlled vocabulary that is part of the ADN framework.

• dateField - an optional argument that indicates which index date field to search in. If supplied, one or both of either the fromDate or toDate arguments must be supplied.

• fromDate - an optional argument that indicates a date range to search from. If supplied, the dateField argument must also be supplied.

• toDate - an optional argument that indicates a date range to search to. If supplied, the dateField argument must also be supplied.

• geoPredicate - a conditionally required argument that indicates the relationship to the query region. Values must be one of [ within | overlaps | contains ].

• geoBBNorth - a conditionally required argument that indicates the northern most latitude of search. Values must be a floating point number in the range [-90,90].

• geoBBSouth - a conditionally required argument that indicates the southern most latitude of search. Values must be a floating point number in the range [-90,90].

• geoBBWest - a conditionally required argument that indicates the western most longitude of search. Values must be a floating point number in the range [-180,180].

• geoBBEast - a conditionally required argument that indicates the eastern most longitude of search. Values must be a floating point number in the range [-180,180].

• geoClause - an optional argument that indicates the boolean clause applied to the geospatial portion of the search. Values must be one of [ must | should ], where must indicates the geospatial portion of the search criteria must match a given record in order for it to be included in the results; should indicates it should match but is not required in order to appear in the search results. Default value is must.

• s - (starting offset) - a required argument that specifies the starting offset into the results set upon which metadata records should be returned. May be any integer grater than or equal to 0.

• n - (number returned) - a required argument that specifies the number of metadata records to return, beginning at the offset specified by s. Must be a integer from 1 to maxSearchResultsAllowed, as indicated in the response to the ServiceInfo request. The maximum allowed by this server is 1000.

• xmlFormat - an optional argument that indicates the format the records must be returned in. If specified, searches are limited to only those records that can be disseminated in the given format. If not specified, the records will be returned in their native format using a localized version of XML (e.g. stripped of their namespace and schema declarations). The available formats may be discovered using the ListXmlFormats request. Not supported in the UserSearch request.

• client - an optional argument that may be supplied by the client to indicate where the request originated from. Example values might be ddsExamplesSearchClient or myLibrarySearchClient. When supplied, this information is used by the services administrators to help understand how people are using the service on a client-by-client basis.

• so - (search over) an optional argument that must contain the value allRecords or discoverableRecords. Clients that request to search over allRecords must be authorized by IP, otherwise an error is returned. Defaults to discoverableRecords. Not supported in the UserSearch request.

• sortAscendingBy - an optional argument that instructs the service to sort the search results in ascending lexical order by a given index field.

• sortDescendingBy - an optional argument that instructs the service to sort the search results in descending lexical order by a given index field.

http://www.dlese.org/dds/services/ddsws1-1?
           verb=Search&q=ocean&s=0&n=10

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <Search>
    <resultInfo>
      <totalNumResults>520</totalNumResults>
      <numReturned>10</numReturned>
      <offset>0</offset>
    </resultInfo>
    <results>
      <record>
        <head>
          <id>DLESE-COLLECTION-000-000-000-018</id>
          <collection recordId="DLESE-COLLECTION-000-000-000-012">
              Science Ed Resource Center (SERC)</collection>
          <xmlFormat>dlese_collect</xmlFormat>
          <fileLastModified>2004-03-29T20:44:41Z</fileLastModified>
          <whatsNewDate type="collection">2004-03-29</whatsNewDate>
          <additionalMetadata realm="dlese_collect">
            <formatOfRecords>adn</formatOfRecords>
            <isEnabled>true</isEnabled>
            <numRecords>325</numRecords>
            <numRecordsIndexed>324</numRecordsIndexed>
            <partOfDrc>false</partOfDrc>
          </additionalMetadata>
        </head>
        <metadata>
          <collectionRecord>
            <general>
              <fullTitle>Carleton College Science Education 
               Resource Center (SERC) - Starting Point Entry 
               Level Geoscience Collection
              </fullTitle>
              ...

</DDSWebService>

http://www.dlese.org/dds/services/ddsws1-1?
           verb=Search&q=ocean&gr=02&s=0&n=10

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <Search>
    <resultInfo>
      <totalNumResults>208</totalNumResults>
      <numReturned>10</numReturned>
      <offset>0</offset>
    </resultInfo>
    <results>
      <record>
	<head>
          <id>NASA-Edmall-2315</id>
          <collection recordId="DLESE-COLLECTION-000-000-000-014">
             NASA ED Mall Collection</collection>
          <xmlFormat>adn</xmlFormat>
          <fileLastModified>2004-06-17T18:24:10Z</fileLastModified>
          <whatsNewDate type="itemnew">2003-07-29</whatsNewDate>
          <additionalMetadata realm="adn">
            <accessionStatus>
               accessioneddiscoverable
            </accessionStatus>
            <partOfDrc>false</partOfDrc>
          </additionalMetadata>
        </head>
        <metadata>
          <itemRecord>
            <general>
              <title>Coriolis Force</title>
              ...

</DDSWebService>

http://www.dlese.org/dds/services/ddsws1-1?
verb=Search&s=0&n=10&fromDate=2004-07-08&dateField=wndate
&sortDescendingBy=wndate&xmlFormat=adn-localized

UserSearch

Sample request

The following request performs a search for the term "ocean" and returns 10 search results, starting at position 0:

http://www.dlese.org/dds/services/ddsws1-1?verb=UserSearch&q=ocean&s=0&n=10&client=ddsws-documentation

Summary and usage

The UserSearch request is nearly identical to the Search request except that it operates over educational resources in the ADN metadata format only, and it applies a default searcher that automatically performs word stemming and relevancy rank boosting for items that match higher relevancy search indicators such as when a matching term appears in the title field as opposed to elsewhere. These search algorithms are the same as those that are applied to user's searches in the DLESE library. This request is meant to to be used by clients working with ADN resources and that wish to leverage the automatic word stemming and search rank algorithms that are applied.

The UserSearch response is identical to the Search response and consists of an ordered set of ADN records, sorted by relevancy. The default searcher that is used incorporates several Information Retrieval techniques designed to augment the search rank and total number of results. These augmentations include word stemming, boosting of records that contain search terms in their title or description and a slight boosting of records that are cataloged by two or more collections or are part of the DLESE Reviewed Collection. The default searcher's algorithms are applied automatically to all terms and phrases supplied by the client in the default field portion of the query sent in the request, and all other fields in the query are treated normally (see available search fields for examples and details). Clients may use this request as a starting point and apply additional boosting to what is provided by the default searcher by supplying additional ranking clauses in the query sent in the request. Clients wishing to implement their own search rank algorithms fully from scratch, or to search over records in formats other than ADN, should use the Search request.

Arguments

UserSearch accepts the same arguments as the Search request, with the exception of xmlFormat, sortAscendingBy, sortDescendingBy, dateField, fromDate, toDate, and so.

Errors and exceptions

See error and exception conditions.

Examples

Request

Same as the Search request, however the verb argument must be indicated as 'UserSearch' and the arguments listed above are not accepted.

Response

Identical to that of the Search request. UserSearch only returns ADN records.

GetRecord

• id - a required argument that specifies the identifier for the record.

• xmlFormat - an optional argument that indicates the format the record must be returned in. If specified, responses are limited to only those records that are available in the given format. If not specified, the record will be returned in it's native format using a localized version of XML (e.g. stripped of it's namespace and schema declaration). The available formats may be discovered using the ListXmlFormats request.

• so - (search over) an optional argument that must contain the value allRecords or discoverableRecords. Users who request to search over allRecords must be authorized by IP, otherwise an error is returned. Defaults to discoverableRecords.

http://www.dlese.org/dds/services/ddsws1-1?
        verb=GetRecord&id=DLESE-000-000-000-337

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <GetRecord>
    <record>
      <head>
        <id>DLESE-000-000-000-337</id> 
        <collection recordId="DLESE-COLLECTION-000-000-000-015">
          DLESE Community Collection (DCC)</collection> 
        <xmlFormat>adn</xmlFormat> 
        <fileLastModified>2004-06-24T19:06:08Z</fileLastModified> 
        <whatsNewDate type="itemnew">2003-07-10</whatsNewDate> 
        <additionalMetadata realm="adn">
          <accessionStatus>accessioneddiscoverable</accessionStatus> 
          <partOfDrc>true</partOfDrc> 
          <alsoCatalogedBy collectionLabel="NASA ESE Reviewed 
             Collection" 
             collectionRecordId="DLESE-COLLECTION-000-000-000-023">
                  NASA-ESERevProd333</alsoCatalogedBy> 
        </additionalMetadata>
      </head>
      <metadata>
        <itemRecord>
          <general>
            <title>Earth Science Picture of the Day</title> 
            ...

</DDSWebService>

ListFields

Errors and exceptions

See error and exception conditions.

Examples

See link above

ListTerms

• field - a required repeatable argument that contains the name of a field. The field argument may be repeated as many times as desired within a single request. Note that response times will increase dramatically when more than one field is requested.

Errors and exceptions

See error and exception conditions.

Examples

See link above

ListCollections

ListGradeRanges

ListSubjects

ListResourceTypes

ListContentStandards

Vocabulary list requests

http://www.dlese.org/dds/services/ddsws1-1?verb=ListGradeRanges

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <ListGradeRanges>
    <searchField>gr</searchField>
    <gradeRanges>
      <gradeRange>
        <vocabEntry>DLESE:Primary elementary</vocabEntry>
        <searchKey>07</searchKey>
        <renderingGuidelines>
          <label>Primary (K-2)</label>
          <noDisplay>false</noDisplay>
          <wrap>false</wrap>
          <divider>false</divider>
          <hasSubList>false</hasSubList>
          <isLastInSubList>false</isLastInSubList>
          <groupLevel>0</groupLevel>
        </renderingGuidelines>
      </gradeRange>
      <gradeRange>
        <vocabEntry>DLESE:Intermediate elementary</vocabEntry>
        ...

</DDSWebService

ListXmlFormats

• id - an optional argument that specifies an ID in the repository. If supplied the request will show only those XML formats that are available for that ID. If omitted, the response will indicate all XML formats that are available in the repository.

http://www.dlese.org/dds/services/ddsws1-1?
             verb=ListXmlFormats&id=DLESE-000-000-000-001

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <ListXmlFormats>
    <xmlFormat>adn</xmlFormat>
    <xmlFormat>adn-localized</xmlFormat>
    <xmlFormat>briefmeta</xmlFormat>
    <xmlFormat>nsdl_dc</xmlFormat>
    <xmlFormat>oai_dc</xmlFormat>
  </ListXmlFormats>
</DDSWebService>

UrlCheck

• url - a required repeatable argument that contains a URL. The url argument may be repeated as many times as desired within a single request.

http://www.dlese.org/dds/services/ddsws1-1?
    verb=UrlCheck&url=http://epod.usra.edu/

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <UrlCheck>
    <resultInfo>
      <totalNumResults>1</totalNumResults> 
    </resultInfo>
    <results>
      <matchingRecord>
        <url>http://epod.usra.edu/</url> 
        <head>
          <id>DLESE-000-000-000-337</id> 
          <collection recordId="DLESE-COLLECTION-000-000-000-015">
            DLESE Community Collection (DCC)</collection> 
          <xmlFormat>adn</xmlFormat> 
          <fileLastModified>2004-06-24T19:06:08Z</fileLastModified> 
          <whatsNewDate type="itemnew">2003-07-10</whatsNewDate> 
          <additionalMetadata realm="adn">
            <accessionStatus>accessioneddiscoverable</accessionStatus> 
            <partOfDrc>true</partOfDrc> 
            <alsoCatalogedBy collectionLabel="NASA ESE 
                 Reviewed Collection" 
              collectionRecordId="DLESE-COLLECTION-000-000-000-023">
                 NASA-ESERevProd333</alsoCatalogedBy> 
          </additionalMetadata>
        </head>
      </matchingRecord>
    </results>
  </UrlCheck>
</DDSWebService>

http://www.dlese.org/dds/services/ddsws1-1?
   verb=UrlCheck&url=http://epod.usra.edu/&
   url=http://www.marsquestonline.org/index.html

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <UrlCheck>
    <resultInfo>
      <totalNumResults>2</totalNumResults> 
    </resultInfo>
    <results>
      <matchingRecord>
        <url>http://www.marsquestonline.org/index.html</url> 
        ....
      </matchingRecord>
      <matchingRecord>
        <url>http://epod.usra.edu/</url> 
        ...
      </matchingRecord>
    </results>
  </UrlCheck>
</DDSWebService>

http://www.dlese.org/dds/services/ddsws1-1?
         verb=UrlCheck&url=http://www.dlese.org* 

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <UrlCheck>
    <resultInfo>
      <totalNumResults>2</totalNumResults> 
    </resultInfo>
    <results>
      <matchingRecord>
        <url>http://www.dlese.org/vgee/index.htm</url> 
        ...
      </matchingRecord>
      <matchingRecord>
        <url>
   http://www.dlese.org/documents/policy/CollectionsScope_final.html
        </url> 
        ...
      </matchingRecord>
    </results>
  </UrlCheck>
 </DDSWebService>

http://www.dlese.org/dds/services/ddsws1-1?
        verb=UrlCheck&url=http://epod.usra.edu/zzzz 

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <UrlCheck>
    <resultInfo>
      <totalNumResults>0</totalNumResults> 
    </resultInfo>
  </UrlCheck>
</DDSWebService>

ServiceInfo

http://www.dlese.org/dds/services/ddsws1-1?verb=ServiceInfo

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService>
  <ServiceInfo>
    <serviceName>
      Digital Library for Earth System Education (DLESE) 
      discovery Web service
    </serviceName>
    <baseURL>http://www.dlese.org/dds/services/ddsws1-1</baseURL>
    <serviceVersion>1.1</serviceVersion>
    <adminEmail>support@dlese.org</adminEmail>
    <compression>gzip</compression>
    <maxSearchResultsAllowed>1000</maxSearchResultsAllowed>
    <description> ... description here ... </description>
  </ServiceInfo>
</DDSWebService>

Service responses

Common response elements

<?xml version="1.0" encoding="UTF-8" ?> 
...
<head>
   <id>CEIS-000-000-001</id>
   <collection recordId="DLESE-COLLECTION-000-000-000-003">
          Discover Our Earth</collection>
   <xmlFormat>adn</xmlFormat>
   <fileLastModified>2004-07-02T17:32:29Z</fileLastModified>
   <whatsNewDate type="itemnew">2003-07-19</whatsNewDate>
	<additionalMetadata realm="adn">
         ...
        </additionalMetadata>
</head>
...

<?xml version="1.0" encoding="UTF-8" ?> 
...
<additionalMetadata realm="adn">
   <accessionStatus>accessioneddiscoverable</accessionStatus>
   <partOfDrc>false</partOfDrc>
   <alsoCatalogedBy collectionLabel="DLESE Community Collection (DCC)"
        collectionRecordId="DLESE-COLLECTION-000-000-000-015">
           DLESE-000-000-000-840</alsoCatalogedBy>
   <alsoCatalogedBy collectionLabel="Cutting Edge" 
        collectionRecordId="DLESE-COLLECTION-000-000-000-010">
           SERC-NAGT-000-000-000-322</alsoCatalogedBy>
</additionalMetadata>
...

Error and exception conditions

http://www.dlese.org/dds/services/ddsws1-1?
          verb=GetRecord&id=BAD-ID-123

<?xml version="1.0" encoding="UTF-8" ?> 
<DDSWebService 
    xmlns="http://www.dlese.org/Metadata/ddsws" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://www.dlese.org/Metadata/ddsws 
	http://www.dlese.org/Metadata/ddsws/1-1/ddsws.xsd">
	<error code="idDoesNotExist">ID "BAD-ID-123" does not exist in the repository</errror>
</DDSWebService>

Requesting JSON output

• output=json - An optional argument that, when used, instructs the service to return JSON output instead of XML.

• callback=function - An optional argument that, when used in conjunction with the output=json argument, instructs the service to return the JSON output wrapped in parentheses and a function name of your choosing, as indicated by the argument value.

Removing namespaces from output

• transform=localize - An optional argument that, when used, instructs the service to return the XML or JSON output without namespaces.

Search fields

This section describes the search fields that are available in the DDSWS Search and UserSearch requests. The repository contains fields that are extracted from each of the XML records within, and a given repository may contain records in many different native XML formats. Searches within a given field operate over the set of records that contain that field. For example, a search in the default field operates over all records in the repository, since all records are guaranteed to contain the default field, whereas a search in the title field operates over a potentially smaller sub-set of records that contain the title field. Boolean searches may be performed across and within each of the fields using the Lucene Query Syntax (LQS) supplied in the q argument of the Search or UserSearch request. Example search queries are provided below.

Fields may contain plain text, controlled vocabularies or encoded field values.

Certain fields may be used to sort the search results when used in the sortAscendingBy or sortDescendingBy arguments of the Search request. Sortable fields are indicated below.

How search fields are generated

At index creation time, each record is inserted in the repository in it's native XML format. The indexer extracts standard, XPath and custom search fields from the content of the native XML and additional fields associated with the item may also be extracted from other sources, such as text derived from a crawl of the resource described by the metadata record. The indexer then generates a single entry containing each of the fields and inserts it into the repository. All records are guaranteed to contain certain fields such as the default and stems fields, as well as XPath fields for their native XML format. Details about the standard, XPath and custom fields are provided below.

Searching across and within specific XML formats

The Search request operates over and disseminates records in any available XML format. By default, searches operate over the available fields for all records in the repository regardless of format, and results may contain records of mixed XML formats. For example, a search for default:ocean searches the for the term ocean in the default field across all records in the repository and may return records in oai_dc, adn, dlese_anno and other formats in a single result set depending on what matches are found.

Requesting search results in a specific XML format: Certain XML formats can be disseminated from the service in multiple formats, for example records that reside natively as adn can also be disseminated in the oai_dc format. The Search request accepts an optional xmlFormat argument, which instructs the service to search over and return only those records that can be disseminated in the given format. In this case, the search still operates over the fields associated with the record's native XML format, however the results will be returned in the requested XML format only, and records that reside in a different native format will be transformed and returned in the requested XML format.

Limiting search to specific XML formats: Each record contains the special field xmlFormat, which contains the format key associated with the native format for the record. To search over and return records that reside in specific native XML formats, include this field in the query for the Search request. For example, the query xmlFormat:oai_dc ocean will search for and return all records in the repository that reside in the native oai_dc format and that contain the term ocean in the default field.

The xml format keys that may be used in the xmlFormat argument or xmlFormat search field in the Search request may be discovered using the ListXmlFormats request.

Text versus stemmed text

When searching in a text field, exact terms are matched. For example a search for ocean will return all records that contain the exact term ocean in the given field. Where indicated, certain textual fields have stemming applied to them using the Porter stemmer algorithm (snowball variation). When searching in a field that has been stemmed, all records containing morphologically similar terms in the given field are matched. For example a search for stems:ocean will return all records that contain the terms ocean, oceans or oceanic in the stems field. Note that when searching in a stemmed field, the client should not apply stemming to the terms it supplies for search. Stemming will be applied automatically by the search engine for these fields and no pre-processing is necessary by the client.

Standard Search Fields

Standard search fields are available across all XML formats that support them, which includes oai_dc, nsdl_dc, ncs_collect, adn, dlese_collect, dlese_anno, news_opps, concepts and all other formats that have them configured in a given DDS repository.

• default - The default field represents the text that is most appropriate for searching by humans using natural language, textual queries. For the Search request, this field is searched when no field specifier is indicated in the query (e.g. the query default:ocean and ocean are equivalent). For the UserSearch request, the default field must be explicitly indicated in the query and if no field specifier is indicated, UserSearch performs an expanded search as described above in the UserSearch description.
  
  The default field contains text extracted from different locations in the metadata depending on the XML format. May not be sorted. Available for all formats. Contains the following content:
  • adn: Includes title, description, keyword, resource type, subjects, event names, place names, temporal coverage names, terms found in the primary URL, and creators last name. May be sorted.
  • dlese_collect: Includes full title, short title, description, subjects, keywords, and terms extracted from the primary URL, scope URL and review process URL.
  • dlese_anno: Includes title, description and terms extracted from the URL.
  • news_opps: Includes title, description, keywords, announcements, topics, audience, diversities, location, and sponsors institution.
  • All other formats: Contains the full content from all Elements within the XML. Content from Attributes within the XML is *not* included.

• admindefault - This field is meant to support users who are responsible for maintaining the repository. It contains the full content from all Elements as well as Attributes within the XML. May not be sorted. Available for all formats.

• stems - Contains the same content as the default field, however all terms are stemmed. May not be sorted. Available for all formats.

• title - Contains the titles of resources or items, as text. May be sorted. Available for formats: adn, news_opps, ncs_collect and all formats that specify this field for indexing.

• titlestems - Contains the same content as the title field, however all terms are stemmed. May not be sorted. Available for formats: adn, news_opps, ncs_collect and all formats that specify the title field for indexing.

• url - Contains the URL for the resource encoded as text. Useful search query examples include http*nasa.gov* or http*.edu*. May be sorted. Available for formats: adn, ncs_collect and all formats that specify this field for indexing.

• description - Contains the descriptions of resources or items, as text. May be sorted. Available for formats: adn, dlese_collect, news_opps, ncs_collect and all formats that specify this field for indexing.

• descriptionstems - Contains the same content as the description field, however all terms are stemmed. May not be sorted. Available for formats: adn, dlese_collect, news_opps, ncs_collect and all formats that specify the description field for indexing.

• xmlFormat - Contains the native XML format key for the record, for example oai_dc or adn, which may be discovered via the ListXmlFormats service request. Available for formats: all formats.

• idvalue - Contains the internal unique identifier for the record, for example MY-ID-001, indexed untokenized as a keyword. Available for formats: all formats.

• allrecords - Special field that matches all records in the repository by applying allrecords:true to the query. This is useful for constructing certain types of queries, for example allrecords:true NOT ocean returns all records in the repository that do not contain the term ocean in the default field. Single valid value is true. This has the same effect as the Lucene query *:*. Available for formats: all formats.

• hasBoundingBox - Boolean value that indicates whether the record has a geospatial bounding box footprint available for search. Valid values are either true or false. Available for formats: all formats.

XPath Search Fields

XPath search fields provide separate searchable fields for the contents of every element and attribute found in the native XML of the records. For each element and attribute there are three forms of search fields: text, stemmed text and untokenized keywords. These provide a powerful, flexible way to search for specific text or data within and across the records in the repository.

The XPath fields consist of a prefix followed by an XPath that addresses a specific XML element or attribute in the XML record. Prefixes are one of /text/, /stems/, or /key/, which specify to search over text, stemmed text or untokenized keyword forms of the data, respectively. This is followed by a namespace-free, position-free XPath addressing a specific element or attribute in the XML.

The three types of search fields are processed in the following manner:

text - Text is processed using the Lucene StandardAnalyzer.
stems - Text is processed using the Lucene SnowballAnalyzer for the english language.
key - Text is processed using the Lucene KeywordAnalyzer, which is case-sensitive and includes the entire element or attribute as a single token.

The XPaths used for the search fields are the most simple form of XPath expression, containing no namespaces or position specifiers. For more information about XPath see XPath Language 1.0. The ZVON XPath Tutorial is also useful. Note that this is not an implementation of XQuery but rather a mapping of simple XPaths to searchable Lucene fields.

For example, consider this simple XML instance document:

<book>
  <author birthDate="1955-01-25">
    <firstName>John</firstName>
    <lastName>Doe</lastName>
  </author>
  <identifier>http://books.org/catalog_123</identifier>
</book>

The index will contain the following search fields for this record:

/text//book/author/firstName
/stems//book/author/firstName
/key//book/author/firstName

/text//book/author/lastName
/stems//book/author/lastName
/key//book/author/lastName

/text//book/author/@birthDate
/stems//book/author/@birthDate
/key//book/author/@birthDate

/text//book/identifier
/stems//book/identifier
/key//book/identifier

As another example, consider the following Dublin Core oai_dc record:

<oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"
    xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ 
	http://www.openarchives.org/OAI/2.0/oai_dc.xsd" 
    xmlns:dc="http://purl.org/dc/elements/1.1/" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <dc:title xmlns:dc="http://purl.org/dc/elements/1.1/">Ocean Science Leadership Awards</dc:title>
  <dc:description xmlns:dc="http://purl.org/dc/elements/1.1/">This is a description of the 
  Ocean Science Leadership Awards... </dc:description>
  <dc:subject xmlns:dc="http://purl.org/dc/elements/1.1/">Earth system science</dc:subject>
  <dc:subject xmlns:dc="http://purl.org/dc/elements/1.1/">Education</dc:subject>
  <dc:format xmlns:dc="http://purl.org/dc/elements/1.1/">text/html</dc:format>
  <dc:type xmlns:dc="http://purl.org/dc/elements/1.1/">Text</dc:type>
  <dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/">
     http://www.usc.edu/org/cosee-west/quikscience/OceanLeadershipAwards.html
  </dc:identifier>
</oai_dc:dc>

The following Lucene queries are examples that match specific text and data in this record. As with all fielded Lucene queries, these queries consist of a field name followed by a colon ":" and then followed by the term(s) to search for. Note that XPaths do not contain namespaces or position specifiers:

/stems//dc/title:oceans - Matches the stemmed form of the term ocean found in the title element of the XML record.

/text//dc/subject:education - Matches the term education found in one of the subject elements of the XML record.

/key//dc/format:"text/html" - Matches the untokenized keyword term text/html found in the format element of the XML record.

Determining which XPaths have been indexed

In addition to the XPaths fields, a special field named indexedXpaths contains each XPath that has been indexed for a given record, as a keyword. Using this field it is possible to search for all records that have any value assigned for a given XPath. For example, the following query:

indexedXpaths:"/dc/subject" - Matches all records that have any value in the /dc/subject field.

Conversely, the following query:

allrecords:true !indexedXpaths:"/dc/subject" - Matches all records that have no value in the /dc/subject field.

Custom Search Fields

Custom search fields are available for specific XML formats as indicated below. Additional custom search fields that are not described here may also be available for a given DDS repository configuration.

Text fields - These fields contain plain text or, where indicated, text that has been stemmed using the Porter stemmer algorithm (snowball variation).

• keyword - Contains keywords associated with the resource or item, as text. May be sorted. Available for formats: adn, dlese_collect, news_opps.

• creator - Contains the first, middle and last name of each contributor for the resource. May not be sorted. Available for formats: adn.

• organizationInstName - Contains the name of the contributing institution. May be sorted. Available for formats: adn.

• organizationInstDepartment - Contains the name of the contributing institution's department. May be sorted. Available for formats: adn.

• personInstName - Contains the name of the contributing person's institution. May be sorted. Available for formats: adn.

• personInstDepartment - Contains the name of the contributing person's institutional department. May be sorted. Available for formats: adn.

• emailPrimary - The primary contributor's e-mail. May be sorted. Available for formats: adn.

• emailOrganization - The contributing organization's e-mail. May be sorted. Available for formats: adn.

• emailAlt- The alternate contributor's e-mail. May be sorted. Available for formats: adn.

• placeNames - Place names, for example "colorado," "AZ," "Brazil," as text. May be sorted. Available for formats: adn.

• eventNames - Event names, for example "windstorm," "Destruction of Pompeii," as text. May be sorted. Available for formats: adn.

• temporalCoverageNames - Temporal coverage names, for example "cambrian," "Triassic Period," as text. May be sorted. Available for formats: adn.

• itemAudienceTypicalAgeRange - The typical age range for this resource. Available for formats: adn.

• itemAudienceInstructionalGoal - The instructional goals for this resource. Available for formats: adn.

• newsOppstitle - News & Opportunities title. May be sorted. Available for formats: news_opps.

• newsOppsdescription - News & Opportunities description. May be sorted. Available for formats: news_opps.

• newsOppskeyword - News & Opportunities keywords. May be sorted. Available for formats: news_opps.

• ncsCollectOaiBaseUrl - Contains the NSDL Collection OAI baseURL. Useful search clause examples include http*nasa.gov* or http*.edu*. May be sorted. Available for formats: ncs_collect.
  From xpath: /record/collection/ingest/oai/@baseURL

• itemContent - The full textual content of the resource. May be sorted. Available for formats: adn.

• itemContentTitle - The HTML title element text. May be sorted. Available for formats: adn.

• itemContentHeaders - The HTML header element (H1, H2, etc.) text. May be sorted. Available for formats: adn.

• itemContentType - The HTTP content type header terms that were returned by the Web server that holds the resource, for example "text html", "application pdf". May be sorted. Available for formats: adn.

• gradeRange - The DLESE grade range vocabularies verbatim as text, for example "DLESE:Primary elementary." These values may be discovered using the ListGradeRange request within the vocabEntry element. May be sorted. Available for formats: adn, dlese_collect.

• resourceType - The DLESE resource type vocabularies verbatim as text, for example "DLESE:Learning materials:Classroom activity." These values may be discovered using the ListResourceTypes request within the vocabEntry element. May be sorted. Available for formats: adn, dlese_collect.

• subject - The DLESE subject vocabularies verbatim as text, for example "DLESE:Atmospheric science." These values may be discovered using the ListSubjects request within the vocabEntry element. May be sorted. Available for formats: adn, dlese_collect.

• contentStandard - The DLESE content standard vocabularies verbatim as text, for example "NSES:K-4:Unifying Concepts and Processes Standards:Change, constancy, and measurement." These values may be discovered using the ListContentStandards request within the vocabEntry element. May be sorted. Available for formats: adn.

• itemAudienceBeneficiary - The intended audience beneficiary, for example "GEM:Bilingual students," as text. See "Beneficiaries" under http://www.dlese.org/Metadata/adn-item/0.6.50/docs/xml-info.htm#vocabs. May be sorted. Available for formats: adn.

• itemAudienceToolFor - The intended audience for this tool, for example "GEM:Middle school teachers," as text. See "Tool for educators" under http://www.dlese.org/Metadata/adn-item/0.6.50/docs/xml-info.htm#vocabs. May be sorted. Available for formats: adn.

• itemAudienceTeachingMethod - The intended audience teaching methods, for example "DLESE:Small group learning," as text. See "Teaching methods " under http://www.dlese.org/Metadata/adn-item/0.6.50/docs/xml-info.htm#vocabs. May be sorted. Available for formats: adn.

• itemannotypes - Indicates the type of annotation that this item has, for example "Teaching tip," "Information on challenging teaching and learning situations," as text. These values are shown in the types schema. May be sorted. Available for formats: adn.

• itemannostatus - Indicates the status of an annotation that this item has, for example "Text annotation completed," as text. These values are shown in the status schema. May be sorted. Available for formats: adn.

• itemannoformats - Indicates the format of an annotation that this item has. Values include 'text', 'audio', 'video' and 'graphical'. May be sorted. Available for formats: adn.

• itemannopathways - Indicates the pathway of an annotation that this item has, for example "CRS (Community Review System)," as text. These values are shown in the pathway schema. May be sorted. Available for formats: adn.

• newsOppsannouncement - News & Opportunities announcements. May be sorted. Available for formats: news_opps.

• newsOppsaudience - News & Opportunities audience. May be sorted. Available for formats: news_opps.

• newsOppsdiversity - News & Opportunities diversity. May be sorted. Available for formats: news_opps.

• newsOppslocation - News & Opportunities locations. May be sorted. Available for formats: news_opps.

• newsOppstopic - News & Opportunities topics. May be sorted. Available for formats: news_opps.

• ncsCollectEdLevel - NSDL Collection education level field. May be sorted. Available formats: ncs_collect.
  From xpath: /record/educational/educationLevels/nsdlEdLevel, /record/educational/educationLevels/otherEdLevel

• ncsCollectCollectionPurpose - NSDL Collection collection purpose field. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/collectionPurposes/collectionPurpose

• ncsCollectAudience - NSDL Collection audience field. May be sorted. Available formats: ncs_collect.
  From xpath: /record/educational/audiences/nsdlAudience, /record/educational/audiences/otherAudience

• ncsCollectSubject - NSDL Collection subject field. May be sorted. Available formats: ncs_collect.
  From xpath: /record/general/subject

• ncsCollectCollectionSubject - NSDL Collection collection subject field. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/collectionSubjects/collectionSubject

• ncsCollectPathwayName - NSDL Collection pathway name. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/pathways/name

• gr - The DLESE grade range vocabularies encoded as a two or three character key, for example "05." These values may be discovered using the ListGradeRanges request within the searchKey element. May be sorted. Available for formats: adn.

• re - The DLESE resource type vocabularies encoded as a two or three character key, for example "05." These values may be discovered using the ListResourceTypes request within the searchKey element. May be sorted. Available for formats: adn.

• su - The DLESE subject vocabularies encoded as a two or three character key, for example "05." These values may be discovered using the ListSubjects request within the searchKey element. May be sorted. Available for formats: adn.

• cs - The DLESE content standard vocabularies encoded as a two or three character key, for example "05." These values may be discovered using the ListContentStandards request within the searchKey element. May be sorted. Available for formats: adn.

• ky - Contains the search key for the collection in which the record resides, which may be used to limit search to within one or more collections of records. These values may be discovered using the ListCollections request within the searchKey element. May be sorted. Available for formats: adn.

• collection - Similary to ky, contains the record's collection vocabulary entry appended with a 0, for example "0dcc," "0comet.". These values may be discovered using the ListCollections request within the vocabEntry element. May be sorted. Available for all formats.

• itemhasanno - Indicates whether an item has an annotation. Values are either "true" or "false." May be sorted. Available for formats: adn.

• partofdrc - Indicates whether the item or collection is part of the DLESE Reviewed Collection (DRC). Values are either "true" or "false." May be sorted. Available for formats: adn, dlese_collect.

• multirecord - Indicates whether the resource that the record catalogs is also cataloged by other records in other collections. Values are either "true" or "false." May be sorted. Available for formats: adn.

• wntype - Indicates the reason the item is new to the repository, corresponding to the 'wndate' field. Possible values are: itemnew, itemannocomplete, itemannoinprogress, annocomplete, drcannocomplete, drcannoinprogress, collection. May be sorted. Available for all formats.

• ncsCollectHasOai - Boolean value that indicates whether the NSDL Collection metadata contains OAI information (an OAI baseURL). Possible values are: true, false. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/ingest/oai/@baseURL

• ncsCollectOaiVisibility - The NSDL Collection OAI visibility field falue. Possible values are: public, protected, private. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/OAIvisibility

• ncsCollectIsPathway - Boolean value that indicates the NSDL Collection pathway value. Possible values are: true, false. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/pathway

• itemannoaveragerating - Contains the average of all star ratings assigned to a given resource. Values range from 1.000 to 5.000. Example search syntax itemannoaveragerating:[3.500 TO 5.000] - returns all resources with an average star rating of 3.5 to 5.0. May be sorted. Available for formats:adn.

• itemannoratingvalues - Contains all star ratings assigned to a given resource. Values range from 1 to 5. Example search syntax itemannoratingvalues:[3 TO 5] - returns all resources that have one or more ratings of 3, 4, or 5 stars assigned to them. May be sorted. Available for formats:adn.

• itemannonumratings - Contains the number of star ratings that have been assigned to a given resource. Values are encoded to 5 digits, for example 00000 or 00014. Example search syntax itemannonumratings:[00004 TO 99999] - returns all resources that have from 4 to 99999 star ratings assigned to them. May be sorted. Available for formats:adn.

• annorating - Contains the star rating of a given annotation record. Values are integers from 1 to 5. Example search syntax annorating:[3 TO 5] - returns all annotations that have a start rating of 3 to 5. May be sorted. Available for formats: dlese_anno.

• ncsCollectOaiFrequency - Integer and float values that indicate the NSDL Collection OAI harvest frequency in months. Range queries are not supported. (search by value only). Possible values are: 1, 2, ... n; 0.5. May be sorted. Available formats: ncs_collect.
  From xpath: /record/collection/ingest/oai/@frequency

• wndate - A date field that indicate the date the item was new to the repository, corresponding to the 'wntype' field. May be sorted. Available for all formats.

• accessiondate - The ADN accession date for the record. May be sorted. Available for formats: adn.

• collaccessiondate - The dlese_collect accession date for the collection. May be sorted. Available for formats: dlese_collect.

• modtime - A date field that corresponds to the time the items file was last modified or touched. This does not necessarily indicate that the content in the record changed. May be sorted. Available for all formats.

• newsOppsapplyBydate - News & Opportunities applyBy date. May be sorted. Available for formats: news_opps.

• newsOppsarchivedate - News & Opportunities archive date. May be sorted. Available for formats: news_opps.

• newsOppsduedate - News & Opportunities due date. May be sorted. Available for formats: news_opps.

• newsOppseventStartdate - News & Opportunities eventStart date. May be sorted. Available for formats: news_opps.

• newsOppseventStopdate - News & Opportunities eventStop date. May be sorted. Available for formats: news_opps.

• newsOppspostdate - News & Opportunities post date. May be sorted. Available for formats: news_opps.

• newsOppsrecordCreationdate - News & Opportunities recordCreation date. May be sorted. Available for formats: news_opps.

• newsOppsrecordModifieddate - News & Opportunities recordModified date. May be sorted. Available for formats: news_opps.

Example search queries

ocean

stems:ocean

stems:(currents in the oceans)

itemannoaveragerating:[3.500 TO 5.000]

url:http*noaa.gov*

url:http*noaa.gov* AND ocean

stems:estuary AND su:02

ocean AND (allrecords:true OR titlestems:ocean^2)

su:02 AND (allrecords:true OR titlestems:florida*^20 
           OR description:florida*^20 OR placeNames:florida^20) 

Glossary

whatsNewDate - A date that describes when an item was new to the repository. Generally this corresponds to the item's accession date or the date in which the item first became accessible in the repository.

Configure search fields

The following document provides information for system administrators who are installing and managing a DDS repository system, which includes the Digital Discovery System (DDS) and the NSDL Collection System (NCS).

• Configure Search Fields - This document describes how to configure the search fields for specific XML frameworks in the repository.

---

DLESE HOME | FAQ | GLOSSARY | SITE MAP | CONTACT US