Finite State Automata in Lucene

2012-05-15T10:51:25Z

Lucene Revolution 2012 is now done, and the talk Robert and I gave went well! We showed how we are using automata (FSAs and FSTs) to make great improvements throughout Lucene. You can view the slides

Spatial Solr Plugin 1.0-RC4

2012-05-15T09:21:49Z

I am pleased to announce the latest release of our Spatial Solr Plugin, v1.0-RC4. This release is a backwards compatible with RC3, and contains the following changes: PDF documentation has been improved to remove inconsistencies in request parameter and source code package names SpatialFilter now includes hashCode and equals implementations, facilitating storage of the filter in...

Indexing your Samba/Windows network shares using Solr

2012-04-24T10:54:19Z

Many of JTeam's clients want to search the content of their existing network shares as part of their Enterprise Search infrastructure. Over the last couple of years, more and more people are switching to Apache Lucene / Solr as their preferred, open source search solution. However, many still have the misconception that it is not possible to index the content of other enterprise content systems, like Microsoft Sharepoint and Samba / Windows...

Faceting & result grouping

2012-04-10T18:43:26Z

Result grouping and faceting are in essence two different search features. Faceting counts the number of hits for specific field values matching the current query. Result grouping groups documents together with a common property and places these documents under a group. These groups are used as the hits in the search result. Usually result grouping and faceting are used together and a lot of times the results get misunderstood. The main reason is that when using grouping people...

Result grouping made easier

2012-04-03T09:45:18Z

Lucene has result grouping for a while now as a contrib in Lucene 3.x and as a module in the upcoming 4.0 release. In both releases the actual grouping is performed with Lucene Collectors. As a Lucene user you need to use various of these Collectors in searches. However these Collectors have many constructor arguments. So they can become quite cumbersome to use grouping in pure Lucene apps. The example below illustrates this. Result grouping using the...

Query time joining in Lucene

2012-02-06T15:24:41Z

Recently query time joining has been added to the Lucene join module in the Lucene svn trunk. The query time joining will be included in the Lucene 4.0 release and there is a possibility that it will also be included in Lucene 3.6. Lets say we have articles and comments. With the query time join you can store these entities as separate documents. Each comment and article can be updates without...

Uwe Says: is your Reader atomic?

2012-02-06T11:55:49Z

Since Day 1 Lucene exposed the two fundamental concepts of reading and writing an index directly through IndexReader & IndexWriter. However, the API didn’t reflect reality; from the IndexWriter perspective this was desirable but when reading the index this caused several problems in the past. In reality a Lucene index isn’t a single index while logically treated as a such. The latest developments in Lucene trunk try to expose reality for type-safety and performance, but before I go into...

Simon says: Single Byte Norms are Dead!

2012-01-19T19:19:12Z

Apache Lucene turned 10 last year with a limitation that bugged many many users from day one. You may know Lucene’s core scoring model is based on TF/IDF (Vector Space Model). Lucene encapsulates all related calculations in a class called...

Introduction to Apache Tika

2012-01-04T14:57:28Z

Apache Tika is a toolkit for parsing and extracting metadata and structured content from various document formats. Tika does not attempt to reinvent the wheel by implementing their own parsers but rather reuse existing libraries for parsing. This way they are able to support many document formats out-of-the-box. This blog post provides an introduction to Apache Tika, including a simple tutorial on how to use Tika to parse some common document...

Gimme all resources you have - I can use them!

2012-01-04T14:54:36Z

Exploiting full IO and CPU concurrency when indexing with Apache Lucene During the last year Apache Lucene has been improved an extreme amount with outstanding improvements such as 100 times faster FuzzyQueries, new Term-Dictionary implementation,...

Lucene / Solr and deep paging

2011-12-27T09:39:51Z

On the solr-user mailing list I’ve seen several questions about the performance impact when retrieving results from pages that are well beyond the first few results. This is also known as deep paging. The problem they are encountering is increase of search time and memory usage. Actually this is a well known problem and even Google seems to suffer from it. When a search for a specific term in my 1M index with Solr I immediately notice the increase in search time...

A look back at 2010

2011-12-27T09:38:52Z

A year has passed and it's time to reflect. Quite a bit has happened at 2010... some things that bothered me and others that inspired me. I thought it would be a cool exercise to try and list some of them... you know, just so in a few years I'll look back at it and reminisce....

Announcing Dutch Lucene User Group

2011-12-27T09:34:51Z

In the last 3 years we've witnessed the rise of open source enterprise search. Of course it was always there, and Apache Lucene in particular was there since, well... the previous century. But in the last 3 years the interest in this area has grown dramatically and the install/user base of the different Lucene related projects (Lucene Java and Solr in particular) has grown at an amazing rate. Today, the Lucene ecosystem is booming - there's a high demand for expertise in this field, yet...

Being at the fore of Apache Solr and Lucene Development

2011-12-27T09:31:50Z

JTeam has always prided itself on being at the forefront of custom software development and exceeding customer expectations. This has been further confirmed with the release of the most voted for features for Apache Solr 1.5, the next version to be developed. As highlighted by Grant Ingersoll, the Solr committer who released the figures,...

Spatial Lucene 2.0

2011-12-27T09:31:04Z

In a number of blog entries we have spoken about the spatial search functionality that we have been developing here at Jteam. In the last two weeks, I have had a chance to contribute much of this work back to the Apache Lucene project with the goal of furthering the development of Lucene's open source spatial search support. If you want to dive immediately into the code, then jump to LUCENE-2139, if you want more details, then...

Language analysis comparable to Fast / Endeca for Solr

2011-12-27T09:29:55Z

Good, solid language analysis is a very important asset for the quality of your search results. It is one of the features that for instance Microsoft Fast and Endeca are using as one of their unique selling points. However, you can get the same powerful analysis when using Apache Solr to implement your search. The thing is that both Ms Fast and Endeca did not implement their language analysis themselves. They use an existing, commercial solution called the Rosetta...

Enterprise Search using Solr and Lucene

2011-12-27T09:28:47Z

The Enterprise Search market has long been dominated by commercial vendors and their products (e.g. Autonomy and Fast). We at JTeam feel that this era is finally over. At least for certain customers and requirements, there is finally a good Open Source alternative: Apache Solr, which is the Enterprise Search server based on Apache Lucene. In this blog post we'll give our view on enterprise search and...

State of Solr

2011-12-27T09:28:16Z

What happened to Solr 1.5? what is Solr 3.1? and what about Solr Cloud? In the last few months, there have been many changes to Solr that can leave users confused about which version to use, what features each version provides, and when (and if) they will be released. This blog entry will try to clarify the State of Solr. Solr 1.5 With Solr 1.4 released in November '09, Solr 1.5 carried on where 1.4 left off. Most development was related to bugs that...

Solr and Lucene 3.1 Release

2011-12-27T09:24:23Z

The new release of Solr and Lucene 3.1, available here and here, is the first major release for Solr in almost two years and the first joint release of both projects. With each project having resolved several hundred issues leading to the release, lets take a look at the major improvements and new features including the much sought after spatial search, UIMA...

SSP 1.0 Video Tutorial

2011-12-27T09:23:44Z

Although SSP v1.0 has been replaced by the simpler 2.0 version, some of you out there are probably still using 1.0 version. Because we like to provide as much assistance as we can to our users, we've decided to publish a video tutorial I created on how to configure and use SSP v1.0. It walks you through the general concepts of spatial search, how to configure the plugin in both your schema.xml and solrconfig.xml and then shows...

Solr blogs