Back to mnoGoSearch site

mnoGoSearch cluster

Introduction

Starting from the version 3.3.0, mnoGoSearch provides a clustered solution, which allows to scale search on several computers, extending database size up to several dozens or even hundreds million documents.

How it works

A typical cluster consists of several database machines and a single front-end machine. The front-end machine receives HTTP requests from a user's browser, forwards search queries to the database machines using HTTP protocol, receives back a limited number of the best top search results (using a simple XML format, based on OpenSearch specifications) from every database machine, then parses and merges the results ordering them by score, and displays the results applying HTML template. This approach distributes operations with high CPU and hard disk consumption between the database machines in parallel, leaving simple merge and HTML template processing functions to the the front-end machine.

Operations done on the database machines

In a clustered mnoGoSearch installation, all hard operations are done on the database machines.

This is an approximate distribution chart of time spent on different search steps:

How a typical XML response looks like


<rss>
  <channel>

    <!-- General search results information block -->
    <openSearch:totalResults>20</openSearch:totalResults>
    <openSearch:startIndex>1</openSearch:startIndex>
    <openSearch:itemsPerPage>2</openSearch:itemsPerPage>

    <!-- Word statistics block -->
    <mnoGoSearch:WordStatList>
      <mnoGoSearch:WordStatItem order="0" count="300" word="apache"/>
      <mnoGoSearch:WordStatItem order="1" count="103" word="web"/>
      <mnoGoSearch:WordStatItem order="2" count="250" word="server"/>
    </mnoGoSearch:WordStatList>

    <!-- document information block -->
    <item>
      <id>1</id>
      <score>70.25%</score>
      <title>Test page of the Apache HTTP Server</title>
      <link>http://hostname/index.html</link>
      <description>...to use the images below on Apache
      and Fedora Core powered HTTP servers. Thanks for using Apache ...
      </description>
      <updated>2006-12-13T18:30:02Z</updated>
      <content-length>3956</content-length>
      <content-type>text/html</content-type>
    </item>

    <!-- more items, typically 10 items total -->

  </channel>
</rss>

Operations done on the front-end machine

The front-end machine receives XML responses from every database machine. On the first query, the front-end machine requests top 10 results from every database machine. An XML response for the top 10 results page is about 5Kb. Parsing of each XML response takes less than 1% of time. Thus, a cluster consisting of 50 machines is about 50% slower than a cluster consisting of a single machine, but allows to search through a 50-times bigger collection of documents.

If the user is not satisfied with search results returned on the first page and navigates to higher pages, then the front-end machine requests ps*np results from each database machine, where ps is page size (10 by default), and np is page number. Thus, to display the 5th result page, the front-end machine requests 50 results from every database machine and has to do five-times more parsing job, which makes search on higher pages a little bit slower. But typically, users look through not more than 2-3 pages.

Cluster types

mnoGoSearch supports two cluster types. A "merge" cluster is to join results from several independent databases, each created by its own indexer.conf. A "distributed" cluster is created by a single indexer.conf, with "indexer" automatically distributing search data between database machines.

Installing and configuring a "merge" cluster

Configuring the database machines

On each database machine install mnoGoSearch using usual procedure:

  • Configure indexer.conf: Edit DBAddr - usually specifying a database installed on the local host, for example:

    
DBAddr mysql://localhost/dbname/?dbmode=blob
    

  • Add a "Server" command corresponding to a desired collection of documents - its own collection on every database machine. Index collections on every database machines by running "indexer" then "indexer -Eblob".

  • Configure search.htm by copying the DBAddr command from indexer.conf.

  • Make sure that search works in "usual" (non-clustered) mode by opening http://hostname/cgi-bin/search.cgi in your browser and typing some search query, for example the word "test", or some other word which present in the document collection.

Configuring XML interface on the database machines

Additionally to the usual installation steps, it's also necessary to configure XML interface on every database machine.

Go through the following steps:

  • cd /usr/local/mnogosearch/etc/

  • cp node.xml-dist node.xml

  • Edit node.xml by specifying the same DBAddr

  • make sure XML search returns a well-formed response (according to the above format) by opening http://hostname/cgi-bin/search.cgi/node.xml?q=test

After these steps, you will have several separate document collections, every collection indexed into its own database, and configured XML interfaces on all database machine.

Configuring the front-end machine

Install mnoGoSearch using usual procedure, then do the following additional steps:

  • cd /usr/local/mnogosearch/etc/

  • cp search.htm-dist search.htm

  • Edit search.htm by specifying URLs of XML interfaces of all database machines, adding "?${NODE_QUERY_STRING}" after "node.xml":

    
DBAddr http://hostname1/cgi-bin/search.cgi/node.xml?${NODE_QUERY_STRING}
    DBAddr http://hostname2/cgi-bin/search.cgi/node.xml?${NODE_QUERY_STRING}
    DBAddr http://hostname3/cgi-bin/search.cgi/node.xml?${NODE_QUERY_STRING}
    

You're done. Now open http://frontend-hostname/cgi-bin/search.cgi in your browser and test searches.

Note: "DBAddr file:///path/to/response.xml" is also understood - to load an XML-formatted response from a static file. This is mostly for test purposes.

Installing and configuring a "distributed" cluster

Configuring the database machines

Install mnoGoSearch on a single database machine. Edit indexer.conf by specifying multiple DBAddr commands:


DBAddr mysql://hostname1/dbname/?dbmode=blob
DBAddr mysql://hostname2/dbname/?dbmode=blob
DBAddr mysql://hostname3/dbname/?dbmode=blob
and describing web space using Realm or Server commands. For example:

#
# The entire top level domain .ru,
# using http://www.ru/ as a start point
#
Server http://www.ru/
Realm http://*.ru/*
After that, install mnoGoSearch on all other database machines and copy indexer.conf from the first database machine. Configuration of indexer is done. Now you can start "indexer" on any database machine, and then "indexer -Eblob" after it finishes. indexer will distribute data between the databases specified in the DBAddr commands.

How indexer distributes data between multiple databases

The number of the database a document is put into is calculated as a result of division of url.seed by the number of DBAddr commands specified in indexer.conf, where url.seed is calculated using hash(URL).

Thus, for indexer.conf having three DBAddr command, distribution is done as follows:

  • URLs with seed 0..85 go to the first DBAddr

  • URLs with seed 85..170 go to the second DBAddr

  • URLs with seed 171..255 go to the third DBAddr

Prior to version 3.3.0, indexer could also distribute data between several databases, but the distribution was done using reminder of division url.seed by the number of DBAddr commands.

The new distribution style, introduced in 3.3.0, simplifies manual redistribution of an existing clustered database when adding a new DBAddr (i.e. a new database machine). Future releases will likely provide automatic tools for redistribution data when adding or deleting machines in an existing cluster, as well as more configuration commands to control distribution.

Configuring the front-end machine

Follow the same configuration instructions with the "merge" cluster type.

Cluster limitations