Indexing & Searching

The field type tells the search engine how to process the field. This is a mandatory, case-insensitive, attribute.

The following table summarizes the available types. The listed Java classes are guaranteed to work. The backend might implement more converting facilities.

Although the Search Service is meant to provide an unified abstraction on the tasks of indexing and querying, text fields have to be somewhat an exception. Indeed, search engines have very different capabilities, depending on provided analyzers. They are nonetheless all expected to provide a direct syntax for full text searches, that an end user can use from, e.g., an input box on a web page. Given the very special kind of constraint that indexing a text field represents, it's not guaranteed that exact matches are supported.

See Section 10.4.4, “Text fields behavior” from the documentation of the Compass backend to get a more concrete view on this (with examples).

On a multi-valued field, the = operator is true if the right operand belongs to the set of field values. The IN operator is true if the intersection between the set of field values and the right operand is non empty.

SELECT * FROM Document WHERE dc:contributors = 'sally'

SELECT * FROM Document WHERE dc:contributors IN ('sally', 'harry')

This behavior is in conformance with the JCR specification, which states it the following more general terms:

The following fields are available on all document resources. They don't correspond to document fields and aren't configurable, that's why they're called pseudo-fields.

The names of these fields are synchronized with constants from the class BuiltinDocumentFields. Any use from java code should rely on these.

All Compass specific configuration files are relative to the classpath of the compass plugin. A default configuration is provided for Nuxeo EP 5 WebApp. To customize it, one sadly has to put the configuration at the right place within the backend's JAR.

Here is an ant fragment to perform this in a JBoss context, assuming that the Search Service has already been installed in the application server and that the server's deployment directory is stored in the deploy.dir property.

<copy todir="${deploy.dir}/nuxeo.ear/platform/nuxeo-platform-search-compass-plugin-1.0.0-SNAPSHOT.jar"
      overwrite="true" failonerror="false">
  <fileset dir="src/main/resources">
    <include name="myfile.xml" />
  </fileset>
</copy>

The backend's JAR is included as a directory in nuxeo.ear during the Maven build of nuxeo-platform-ear for this single purpose. This is prone to change in the future.

The Compass backend itself is registered against the Search Service through the searchEngineBackend extension point of org.nuxeo.ecm.core.search.service.SearchServiceImpl. Your component can use the configurationFileName element to specify a path to the master configuration file, like this:

<searchEngineBackend name="compass" default="true"
    class="org.nuxeo.ecm.core.search.backend.compass.CompassBackend">
  <configurationFileName>/mycompass.cfg.xml</configurationFileName>
</searchEngineBackend>

The default path is /compass.cfg.xml.

Compass supports several storage possibilities, called connections in Compass configuration objects. The configuration is done trough a Nuxeo Runtime extension point and possibly within the Compass master configuration file. The extension point always takes precedence over the Compass file, but can be used to fall back to Compass file, that offers currently more possibilites.

The target is org.nuxeo.ecm.core.search.backend.compass.CompassBackend, and the point is connection. Contributions are made of a single XML element; the latest one wins.

To set the connection to a file system Lucene store, put a file element in the contribution, and set the path attribute to the target location. If the path doesn't start with /, it will be interpreted as being relative to Nuxeo Runtime's home directory, e.g, /opt/jboss/server/default/data/NXRuntime in the default Nuxeo EP installation on JBoss.

Other connection types, notably JDBC, are defined by the Compass configuration file, one has to put the default XML element in the contribution, like this:

<extension target="org.nuxeo.ecm.core.search.backend.compass.CompassBackend"
           point="connection">
  <default/>
</extension>

The default connection is a relative file system one, hosted in the nxsearch-compass sub directory of Nuxeo Runtime's home.

The master configuration file holds the definition and configuration of analyzers: a lookup name gets associated to an analyzer type and options. The Compass backend makes Compass use directly the name declared in the Search Service as the lookup name, configuration, therefore one has to ensure here that all of these do exist in the Compass configuration.

Together with the registration itself comes the configuration of analyzers. For instance, an analyzer discarding stop words might be given the full list of stop words.

Compass comes with a two predefined analyzers: default and search. You can reconfigure them as well.

See the relevant part in Compass documentation for details and sample configurations.

Lucene only knows about character strings. Therefore, typed data such as dates and integers must be converted in strings to get into Lucene and back. Compass provides helpers for this and the Compass backend uses them directly.

In the master configuration file, one register available converters in the form of a lookup name and a Java class. Lots of converters are already registered by default, covering most basic types. The compass.cfg.xml file shipping with the Compass backend redefines one (the date converter).

Currently, the Compass backend can't force Compass to use a given converter and/or analyzer on a given field. It must therefore be specified in the mappings file, which is itself loaded from the master configuration file. The default name for this file is nxdocument.cpm.xml.

Here's a sample, inspired from the mappings file provided with the backend.

<?xml version="1.0"?>
<!DOCTYPE compass-core-mapping PUBLIC
    "-//Compass/Compass Core Mapping DTD 1.0//EN"
    "http://www.opensymphony.com/compass/dtd/compass-core-mapping.dtd">

<compass-core-mapping>

  <resource alias="nxdoc"
            sub-index="nxdocs"
            analyzer="default"
            all="false">
    <resource-id name="nxdoc_id"/>

    <resource-property name="dc:created" converter="date"
                       store="yes" index="un_tokenized" />
    <resource-property name="dc:title" analyzer="french" />

  </resource>
</compass-core-mapping>

In Compass' terminology, fields are called properties. The name of the Compass property corresponding to a Nuxeo indexed field coincides with the field's prefixed name.

Some important remarks:

Follow the instructions from Section 10.4.1.1, “Configuration files location”

At indexing time, the text field is analyzed using the analyzer from the Compass configuration file regardless what has been configured through the Search Service extension point.

Equality statements in WHERE clauses are transformed into the closest thing that Lucene can provide on an analyzed field, namely a phrase query.

On the other hand, LIKE clauses are directly fed to Lucene's QueryParser. To search documents whose title starts with "nux", one may write

SELECT * FROM Document WHERE dc:title LIKE 'nux*'

The following two statements are equivalent. The second one is the QueryParser syntax for phrase queries.

... WHERE dc:title='Nuxeo Book'
... WHERE dc:title LIKE '"Nuxeo Book"'

Lucene's QueryParser syntax is really powerful, you can specify how close two words can be, apply fine grained boosts for the relevance ordering, and more. The only restriction you have on LIKE statements for text fields within the Compass backend is the choice of field. In other words, the colon character is escaped.

You would need to query date fields, like creation, modification or expiration date, that are provided by Nuxeo Platform. In these cases, it would be interesting to use BETWEEN clauses, associated with DATE keyword, which allows to convert strings as date values.

... WHERE dc:created BETWEEN DATE '2008-01-01' AND DATE '2008-03-31'

... WHERE dc:modified BETWEEN DATE '2007-05-01' AND DATE '2007-05-31'

You should be aware of the following trap in Lucene queries: purely negative queries don't match anything, even if they are themselves nested in a boolean query that has positive statements. The Compass backend uses the standard way to circumvent this limitation, provided that the negative aspect can be seen from the NXQL structure, i.e., not enclosed in a Lucene QueryParser literal.

... WHERE dc:title LIKE '-book'
... WHERE dc:title LIKE 'nuxeo' AND dc:title LIKE '-book'

... WHERE dc:title NOT LIKE 'book'
... WHERE dc:title LIKE 'nuxeo' AND dc:title NOT LIKE 'book'
... WHERE dc:title LIKE 'nuxeo' AND NOT dc:title LIKE 'book'
... WHERE dc:title LIKE '+nuxeo -book'