Quickly view any managed property value in SharePoint 2013 using Search REST API

Search REST API in SharePoint 2013 makes it very easy to quickly check the value stored in the search index for any managed property. For example, let’s say you have a SharePoint list and one of the custom columns in that list is a publishing image field. You need to display the image in search results but are not sure what type of value is stored in the search index. Is it an absolute url to the image? A relative url? Maybe an entire image HTML tag?

A quick way to check exactly what it is by using the Search REST API. First, let’s run a simple Search REST API query and see what managed property values are returned.

http://intranet.contoso.com/_api/search/query?querytext='contenttype:banner'

Since only standard managed properties are included by default, add the selectproperties parameter to the query.

http://intranet.contoso.com/_api/search/query?querytext='contenttype:banner'&selectproperties='BannerImageOWSIMGE'

Similar to querytext and selectproperties, many other KeywordQuery object model properties can be used as parameters for Search REST API calls.

Crawler Impact Rules in SharePoint 2010

In SharePoint 2010, you can control the content crawl rate at the search service application level by using Crawler Impact Rules. By default, the number of simultaneous requests changes dynamically based on the server hardware and utilization. Crawler impact rules are often used to throttle the request rate for external websites. You can manage crawler impact rules in Central Administration > Search Service Application > Crawler Impact Rules.

Let’s watch the Filtering Threads and Idle Threads from the OSS Search Gatherer category in Performance Monitor during a content crawl before any crawler impact rules are defined.

In my development VM, I see that the number of filtering threads stays at around 20 and the number of idle threads around 12 for the duration of the crawl which means that an average of 8 threads are being used by the gatherer process.

Next, let’s create a new crawler impact rule to limit the number of simultaneous requests to 2 using the * site name wildcard to apply the rule to all sites.

If we keep an eye on the performance counters this time, we can see that now the difference between the number of filtering and idle threads during a crawl equals to 2 (11 filtering threads and 9 idle threads in the example below).

The performance counters confirmed that our new crawler impact rule is working. Be careful when you delete a crawler impact rule though! I’ve seen it a number of times in different SharePoint farms that SharePoint remembers and keeps using the last deleted crawler impact rule. I verified it by monitoring the performance counters – the deleted crawler impact rule remains in effect until the the SharePoint Server Search 14 service is restarted (or a new rule is added that overrides the deleted rule settings). So remember – restart the SharePoint Server Search 14 windows service after deleting a crawler impact rule!

Using Content Enrichment Web Service Callout in SharePoint 2013 Preview

SharePoint 2013 Preview release intoduced a new functionality called content enrichment web service callout. It provides the ability to inspect and manipulate managed property values for each item before it’s added to the search index. Prior to SharePoint 2013, the only way to accomplish something similar was in FAST Search for SharePoint by extending the item processing pipeline. Clients using SharePoint server search were out of luck as the functionality was not available to them.

The process of building and configuring a web service callout is relatively straight forward. These are the high-level steps to follow:

  1. Build a web service by implementing the IContentProcessingEnrichmentService interface. Add logic to manipulate managed property values.
  2. Run PowerShell commands to configure the callout web service endpoint address, input and output managed properties, trigger condition and a few other things.
  3. Execute a full crawl.

In this blog post I’ll show an example of developing a web service that populates a new managed property value which is then used as a refiner on the search results page. Let’s say we have a number of project sites in SharePoint where each site contains information about a specific bike model.

Each bike model belongs to a product category such as Mountain Bikes, Road Bikes and Touring Bikes. We’d like to be able to refine search results by product category but unfortunately that metadata is not available in SharePoint at this point. What we are going to do next is create a new managed property called ProductCategory and build a web service to populate the managed property values based on our custom business logic. The ProductCategory managed property can then be used as a refiner on the search results page.

To create the managed property, navigate to Central Administration > Search Service Application > Search Schema > New Managed Property.

  • Property name: ProductCategory
  • Type: Text
  • Searchable: checked
  • Queryable: checked
  • Retrievable: checked
  • Refinable: Yes – active
  • Token Normalization: checked

In Visual Studio 2012, create the web service: New Project > Visual C# > WCF > WCF Service Application.

Delete the Service1 created by default or rename it to EnrichmentService. Delete the IService1 or IEnrichmentService interface.

Add an assembly reference to C:\Program Files\Microsoft Office Servers\15.0\Search\Applications\External\microsoft.office.server.search.contentprocessingenrichment.dll.

Open EnrichmentService.svc.cs, add the following using statements:

using Microsoft.Office.Server.Search.ContentProcessingEnrichment;
using Microsoft.Office.Server.Search.ContentProcessingEnrichment.PropertyTypes;

Replace the class implementation:

public class EnrichmentService : IContentProcessingEnrichmentService
{
    private Dictionary<string, string> productModels = new Dictionary<string, string>()
    {
        {"mountain-100", "Mountain Bikes"},
        {"mountain-500", "Mountain Bikes"},
        {"road-150", "Road Bikes"},
        {"road-450", "Road Bikes"},
        {"touring-1000", "Touring Bikes"},
        {"touring-2000", "Touring Bikes"}
    };

    public ProcessedItem ProcessItem(Item item)
    {
        ProcessedItem processedItem = new ProcessedItem();
        processedItem.ItemProperties = new List<AbstractProperty>();

        AbstractProperty pathProperty = item.ItemProperties.Where(p => p.Name == "Path").FirstOrDefault();
        if (pathProperty != null)
        {
            Property<string> pathProp = pathProperty as Property<string>;
            if (pathProp != null)
            {
                foreach (var productModel in productModels)
                {
                    if (pathProp.Value.Contains(productModel.Key))
                    {
                        Property<string> modelProp = new Property<string>()
                        {
                            Name = "ProductCategory",
                            Value = productModel.Value
                        };
                        processedItem.ItemProperties.Add(modelProp);
                    }
                }
            }
        }

        return processedItem;
    }
}

Now the web service is ready and the next step is to configure SharePoint to call the web service during the crawl. That is done using PowerShell. To minimize the performance impact of the web service callout, we only want it to be called under a certain condition – this condition is defined in the Trigger property. More information about the syntax can be found in the Trigger expression syntax article on MSDN. The expected input and output managed properties are configured via the InputProperties and OutputProperties. When debugging the web service, the DebugMode property value can be set to $true in which case SharePoint will ignore the InputProperties value and will send all available managed properties for each item to the service. Any managed property values returned by the web service in debug mode are ignored by SharePoint.

$ssa = Get-SPEnterpriseSearchServiceApplication
$config = New-SPEnterpriseSearchContentEnrichmentConfiguration
$config.DebugMode = $false
$config.Endpoint = "http://localhost:64401/EnrichmentService.svc"
$config.FailureMode = "WARNING"
$config.InputProperties = "Path"
$config.OutputProperties = "ProductCategory"
$config.SendRawData = $false
$config.Trigger = 'StartsWith(Path,"http://intranet.contoso.com/adventureworks/models/")'
Set-SPEnterpriseSearchContentEnrichmentConfiguration –SearchApplication $ssa –ContentEnrichmentConfiguration $config

Finally, launch the web EnrichmentService created earlier and start a new full crawl. Once the crawl is complete, the ProductCategory managed property should be populated and searchable:

The final step is to add a Product Category search refiner. Edit the search results page, edit the Refinement web part, click the Choose Refiners… button within the Properties for Search Refinement section, select the ProductCategory managed property in the Available refiners list and press the Add > button. Move the ProductCategory to the top of the Selected refiners list, then scroll down and set the Display name to Product Category and save your changes.

Run a search for “bike” and you should now be able to refine the search results by the product categories:

References: