Search: Divergent Sources, Simple Organization

Search tools that can filter mass quantities of data while connect people to the differing, yet relevant information – quickly, easily, and accurately – are highly valuable. At the same time, companies are under pressure to manage and organize data interactions and records in a timely manner — making technologies like Federated search a hot commodity. However, as with any "hot commodity" there are concerns. Here are some of them:

User Concerns

• A search system that requires too much information from the user about where to look, and how it should search, defeats the federated search model and inhibits usability.
• Federated search may return faceted results. These results must be organized in a way that makes sense to the user. The system must present the user with a visually hierarchical way of organizing the faceted or categorized results.
• Also there is the concern of different users needing different security credential to access different data sources. What is needed is a way to provide a single sign-on capability. This concern is normally understood at the outset of a project. The need for different security credentials should be transparent to the user.
• Users think, “I know what I’m looking for, so should the application.” Innovative linguistic patterns and algorithms should determine the user’s intent. For instance, if a user searched for a person whose last name is “Johnson”, linguistic patterns and algorithms should determine that this textural string is indeed a name of a person. Similarly, if a user submits the query “Process payroll”, it is likely that he or she wants to visit the location in the application where payroll processing takes place. Develop technological methods to identify navigational queries.

Development Concerns

• Scalability: A scalable search system must meet the demand of data indexing and query processing loads today and also handle increased bandwidth under a bigger load tomorrow.
• Diverse content require diverse relevance models. Determine how the system can possibly know the intent of the user.
• What search features are required: Federated search requires advanced features such as algorithmic sentiment expressions, clustering, entity extraction, faceted classifications, and other linguistic pattern algorithms.
• Connectivity: Search engines rely on connectors to hook into databases and application. Sometimes these are in different business applications. Applying search across line-of-business application or different databases and data sources may introduce complex connectivity issues.
• An inherent problem with joining together data is that different sources may require different security credentials and different retrieval algorithms. Connectors that access secure sites are required to handle security individually and different than a wide-open site. Retrieval algorithms for relational databases may different than the algorithms need to retrieve indexed documents.
• Should we purchase a third-party search tool or develop a Federated search system internally.
• How to develop a system that is adaptable, easily extensible and maintainable (this means putting as much information as possible in a “configuration layer” that is easily managed and extended to handle new situations and differing situation. This separates the mechanics of generating search queries and formatting results from the details of the query language and native data formats. A configuration layer or tool will provide the “mappings” between the results and the source).
• Administrative overhead. Enterprise search must be quick enough to deploy and easy enough to manage that the cost of installing and maintaining it won’t exceed the benefit.