• The index of the content to be searched
• The act of actually performing the search
• The reporting of the search results

Many people, I believe, only really think about these last two, but it’s really the index that’s key to the success of any search engine, just like a good index at the back of a book makes it possible for a reader to quickly find what they’re looking for.

As far as search engines go, the gold standard is Apache Lucene. Lucene has been a reliable and popular search engine of choice for years now. Although Lucene is written in Java, the Zend_Search_Lucene module, part of the Zend Framework, is a great PHP port of the software. In a previous post, I explained how to integrate Zend_Search_Lucene into a Yii-based Web application. The focus in that post is really on getting the two different frameworks to work together. This is easily accomplished for two reasons:

1. Yii supports third-party tools nicely
2. The Zend Framework can be used piecemeal

So that previous post on Yii and Zend_Search_Lucene walks you through the Yii Controller and View files you’d create to perform a search and report upon the results, something that Zend_Search_Lucene does easily. Creating the index itself is the actual challenge, then, and one that I don’t feel is adequately documented elsewhere. In this post, I explain how to use Zend_Search_Lucene to create a search index of a site.

There are different ways you can create an index of any Web site. One option is to use a spider: provide it with a starting URL and it will then crawl through the site, indexing content, noting and following links, etc. Unfortunately—and this is something that’s not obvious to those new to Lucene, Lucene is not a spider and cannot crawl your site (the same goes for Zend_Search_Lucene). Lucene is simply a search engine. One option, which I have played around with, is to use a separate crawler that will create a Lucene-compatible index for you. The most popular such tool is perhaps Nutch, also from Apache. Nutch is an excellent crawler, but it’s written in Java and getting it to output Lucene-compatible indexes isn’t easy. I worked on that for some time and often felt like I was spinning my wheels (in part because I have no formal Java training). In the end, I decided to go another route…

Running a spider through a site (over HTTP) makes sense particularly when no internal access to the content is available. However, with a database-driven site, some of the site’s content can be discovered in another, easier way: in the database itself! In fact, unless a site has static pages, probably all of the site’s content can be found in the database. From database queries alone, you can generate a custom index. I’ll explain how to do that, but stepping back a bit…

Lucene stores information about content in a series of files on the server. In my Yii Controller, say SearchController.php, I identify the location of the index files as a private variable, so that the location will be available in multiple methods:

private $_indexFile =  '/runtime/search';

Yii’s runtime directory is where the Yii application stores information as the site runs, such as logs, so it’sa logical place to put the index files (i.e., the application already writes data there).

Whether you’re creating an index for the first time or using it to execute a search query, you must create an object of type Zend_Search_Lucene:

$index = new Zend_Search_Lucene($this->_indexFile, true);

The first argument is the location on the server where the files should be stored. The second argument of true says that a new index should be created. You wouldn’t use this argument to perform a search, naturally. Note that this code does assume you’re using the Zend Framework in some capacity and have already included the appropriate framework files. Also, this code was written using Zend Framework version 1.9.

Once the index directory has been identified, the next step is to add documents to the index, which is the important part. One option is to use the Zend_Search_Lucene_Document_HTML class. Call the class’s loadHTMLFile() method, passing it the URL of the page to load:

$url = 'http://www.example.com/somepage.php';
$doc = Zend_Search_Lucene_Document_Html::loadHTMLFile($url);

Then you can add that document to the index:

$index->addDocument($doc);

These simple steps will work, but this requires that you know what every URL to be indexed is (again, getting back to the issue of creating a spider or crawler). Also, this approach requires the loading of the full HTML page for each document being added, meaning that the indexer has to read through all the extraneous HTML, CSS, and JavaScript, then find only the content that counts. Instead, you can get the content that counts directly from the database.

Tip: You can create your own spider, if you wanted, as the Zend_Search_Lucene_Document_Html object will have a getLinks() method that returns every link found within the HTML document.

On a recent project I did, for each page in the site, the page-specific content was stored as a record in a pages table. When the user views the page in the browser, the page-specific content is retrieved and displayed within a context (the overall site template). Nothing else in that template needs to be indexed as it is all common to every page, so I can just index the data stored in the database instead. To do that, I first use the Yii Model to fetch every database record:

$model = Pages::model()->findAll();

Next, loop through each returned record:

foreach ($model as $m) {

Within the loop, each record needs to be added to the index as a new document. And here’s where you need to think about how the search and the search results should work. The two basic questions are:

• What content should be search-able?
• What content should be displayed on the search results?

In my example, the search-able content should be the page-specific content and its title. The search results should show the page title, a preview of its content, and be linked to the page itself. With that in mind, here’s how my loop indexed each returned record:

$url = 'http://www.example.com/index.php/pages/show/id/' . $m->id;
$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('link', $url));
$doc->addField(Zend_Search_Lucene_Field::Text('title', $m->pageTitle));
$content = $this->clean_content($m->content);
$doc->addField(Zend_Search_Lucene_Field::Unstored('content', $content));
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('preview', $this->preview_content($content)));
$index->addDocument($doc);

First, I create the URL so that the search results will be able to link tot he page. The URL for these particular records will be pages/show/id/X: show the Page whose ID value is X. Next, create a new Zend_Search_Lucene_Document object, which is the most generic search document class. From there, you add “fields” to the document. Each field can be of a specific type, can be given a unique name, and be assigned a value.

Tip: The Zend_Search_Lucene_Document_Html class is just a Zend_Search_Lucence_Document with pre-defined fields

The field types are all identified using Zend_Search_Lucene_Field::Something. They’re all listed in the Zend documentation and you’ll want to look those up before you get too involved with implementing this yourself. There are several important properties about the different field types. In particular, you want to pay attention to whether the field will be indexed or not and stored or not. For example, there’s no need to index the URL, because it won’t be meaningful as a search item, but it must be stored (to appear in the search results), so that field is of type UnIndexed. Conversely, the page’s title needs to be both stored and indexed, so it’s of type Text. The page’s content must be indexed but shouldn’t be stored (because that’d be inefficient), so it uses a field type of Unstored. Finally, I want to store but not index a preview of the document, so I use the UnIndexed type there again.

Again, this is where you’re actually creating the custom index: you decide what should be indexed and stored, in other words, what’s important. You might also store, for example, the modification date of the content (but not index the date). You’ll also see in that code that I’m running the content through two user-defined functions: clean_content() and preview_content(). Here are how those are defined:

// Function for returning a preview of the content:
// The preview is the first XXX characters.
private function preview_content($data, $limit = 400) {
    return substr($data, 0, $limit) . '...';
} // End of preview_content() function.
// Function for stripping junk out of content:
private function clean_content($data) {
    return strip_tags($data);
}

The clean_content() function strips out all the tags. This allows me to index just the text itself, without any HTML. Although defining a function that just invokes a single PHP function is generally a no-no, I went ahead and did that here in case I later decide to change how content is “cleaned”. The second function returns a subsection of the content, to be used as a preview. The preview will appear in the search results and is, by default, the first 400 characters of the content followed by ellipses.
Once a new document has been created, the last line (in the loop above) adds the document to the index.

After all of the content has been indexed (and you’ve completed the foreach loop), you can optimize and commit the index:

$index->optimize();
$index->commit();

And that’s it! A custom index has now been created based upon data stored in the database. Is it a perfect replication of Google? No. But it works well as a custom index for a Web site. Once you understand how to create your own index, you can expand it, of course. In the site I worked on, there was a contacts table that stored lists of people (displayed on a single page in the site). To index that content, I retrieved all the contacts, and looped through them. For the title value, I used each person’s full name. For the content to be indexed, I concatenated several pieces of data about each contact: their name, title, organization, personal description, and so forth.

In this particular project, I also indexed PDFs stored on the server, a topic I’ll write up in a separate post.

I hope this was helpful. Thanks for reading and, as always, let me know if you have any questions or comments!

Next, download the Zend Framework. You’ll want to download the latest full package, even though you’ll only use a bit of it. After the download has completed, expand the ZIP or TAR.GZ file (whichever format you choose to download the framework in). The result will be a folder named ZendFramework-x.y.z. (where x.y.z represent the full version number). Within that folder, go into library/Zend and copy Exception.php to protected/vendors/Zend. This is the file that the Zend Framework uses to report problems, so you’ll want to include it while developing and debugging Zend_Lucene with Yii. Also copy the Search folder to protected/vendors/Zend. You’ll end up with a structure like this:

In terms of the MVC architecture, the Zend Framework provides the Model to be used by this search process, but the Controller and View will still be done using Yii. First, let’s write a new Controller for searching:

class SearchController extends CController
{
    private $_indexFiles = '../runtime/search';
    public function actionIndex() {}
    public function actionCreate() {}
    public function actionSearch() {}
    public function actionUpdate() {}
}

As with all Yii Controllers, this one extends the base CController class. Within this Controller the various methods are defined, corresponding to the actions that’ll be taken in the search process. The index action is the default and is for accessing the search page without performing an actual search (e.g., clicking on a link to go to the search page). The create action will be used to generate the search database: the series of files that Lucene needs to perform its searches. The search action is for handling submission of the search form (i.e., it does the actual searching). Finally, the update action is for updating the Lucene database files when necessary (like when the site content changes). The class also has one private variable that stores the location on the server of Lucene database files. I chose to put them in a search folder found within runtime (protected/runtime/search). This class member is good to have as multiple methods will need this information but I create it as a private variable as it’s not necessary (nor should it be accessed) outside of the class. As a naming convention, some like to use underscores at the front of private class variables.

Within three of the methods (not actionIndex()), the Controller will use Zend_Lucene. In order to do so, this script needs access to the Zend files, so import the contents of the vendors directory at the top of this script, just before the class definition begins:

Yii::import('application.vendors.*');

Then, include the Lucene.php page, found within the Zend Framework Search folder:

require_once('Zend/Search/Lucene.php');

Now this Yii Controller can create objects of type Zend_Search_Lucene, which is defined in that file. The actions will use that object type to perform the searches. To start, the index action just renders the index View:

public function actionIndex()
{
    $this->render('index');
}

Presumably the index View file just shows the search form. The search form, by the way, should have an action attribute of www.example.com/index.php/search/search, so that it calls the search action of the search Controller. The form should contain a text input with the name terms.

The update action would be used by an administrator to update the search database. Perhaps it’d be called automatically after some content is generated or once per hour or day. It would destroy the existing search database and then invoke the actionCreate() method. The Lucene database can’t just be updated for whatever content changed; you need to destroy and recreate it instead. It really wouldn’t matter what View this action renders, depending upon what you want the admin to see. Maybe the View would just show a message indicating that the database has been updated.

The create action is an important one, and is where real knowledge of Lucene comes into play. The shell of it would look like so:

public function actionCreate() {
    $index = new Zend_Search_Lucene($this->_indexFile, true);
    // Add documents to the database.
    $index->commit();
    $this->render('create');
}

First, a Zend_Search_Lucene object is created (again, this is where Yii is making use of a class defined outside of Yii; it’s a sweet thing). The first argument provided when creating the object is the location of the database files. This is represented by the Controller variable, accessible in $this->_indexFile. The second argument indicates that a fresh database should be created. Next up, you add content to the database. This is complicated and well beyond the scope of what I’m writing here. I’ll try to discuss this, in brief, in a separate post, but I’d recommend you read as much as you can online first. In a very minimalistic way, you could add a single HTML page to the search database by doing this:

$url = 'http://www.example.com/index.php/page/show/id/1';
$doc = Zend_Search_Lucene_Document_Html::loadHTMLFile($url);
$index->addDocument($doc);

Finally the database has to be saved, by invoking the commit() method. And then some View is rendered. As this action would also only be likely called by an administrator or cron, it doesn’t matter much what the View contains.

Lastly, there’s the search action. This action would check for search terms, run the search against the database, then send the results on to a View:

public function actionSearch() {
    if (isset($_GET['terms'])) {
        $index = new Zend_Search_Lucene($this->_indexFile);
        $results = $index->find($_GET['terms']);
        $this->render('search', array('results' => $results));
    } else {
        $this->render('index');
    }
}

First the method checks for the presence of search terms in the URL. Then it creates a Zend_Search_Lucene object, which is necessary for both creating and using the search database. This time only the location of the search database is passed when creating the object. The object’s find() method is invoked for performing the search (it can be that simple!). Then the search View is rendered, passing it the results. If no search terms were passed to this page, the index View is rendered instead. As for the search results View, a basic version to get you started might look like this:

<h2>Search Results for "<?php echo CHtml::encode($_GET['terms']); ?>"</h2>
<?php if ($results): ?>
    <?php foreach($results as $result): ?>
        <p><?php echo CHtml::encode($result->title); ?></p>
    <?php endeach; ?>
<?php else: ?>
    <p class="error">No results matched your search terms.</p>
<?php endif; ?>

That’s largely the logic and structure of a search results View. It displays the provided search terms and checks for results. If there were some, each result title is printed. In a real application, you’d likely link the title to a URL or whatever but I don’t want to get too messy here. If you do print_r($result), you’ll see a bunch of information there that you can use.

So that’s the steps you need to take to get started using Zend_Lucene within your Yii application. These steps provide functionality; mastering Lucene is how you make this more professional. I’ll try to write more about defining a Lucene search database in subsequent posts towards that end. If you have any comments, questions, or requests, let me know.

Thanks,

Larry