By creating data quality Rules, corrective action can be taken against data that does not adhere to a strict criteria.

To access the Rules functionality, click Setup, Extensions then Rules.

Click the New button to begin the process.

Creation Process

Enter the name that will be given to the Rule in the Name textbox. This name must be unique.

The Description textbox enables a detailed description to be added to Rules, aiding users find created examples.

Use the Type drop-down list to select the required Rule type. The following types are available:

Enabling the Invert option inverts the rule, essentially adding NOT to the beginning of the statement.

The Action option enables the choice of action taken if Rules are broken. Validation allows the specification of a Corrective Action message. This message will create a new field in the Index containing the specified text whenever the rule is broken. Users can then use manual data entry to correct the discrepancy.

Selecting Update as the action allows a calculation to be specified whenever a rule is broken. This automatically updates each value that breaks the rule according to the specified calculation.

The Severity option allows the assigned severity, either Major or Minor, to be applied when the Rule is broken. Severity is used to sort Rules in the front-end through the creation of specific fields, such as ‘Minor Rules Failed’ or ‘Rules Passed Count Major’, for example.

The Test box allows a Rule to be tested prior to its creation. Enter a value in the textbox and click Test to result in a tick if the value has passed, or a cross if it has failed.

To complete the process and save changes, click Create Rule. To return to the previous screen and discard changes, click Cancel.

Rules Data Source

After creating Rules, a Rules-based Data Source can be built in the Data Sources screen that allows a collection of Rules to be applied simultaneously.

To access the Data Source Screen, click Search Engine, then Data Sources.

Select Rules from the Type drop-down list to begin the creation process.

Creation Process

Enter the name that will be given to the Data Source in the Name textbox. This name must be unique.

Use the Tags textbox to add associated search terms to the data source. This allows access via customised strings when using the search bar.

Select the Data Source Group that the rules will be applied to from the Data Source Group drop-down list. Following the selection, a Rules search bar and a table populated with the Data Source Group fields will be revealed below.

Select one or more fields using the relevant checkbox, and click the search bar above the table. This will reveal all available Rules that can be applied to the data, displayed along with their description. Results can be filtered by searching for the specific name or description text. Click a Rule to select it, and remove a selected rule by clicking the relevant X icon.

Once the required rules have been selected for the checked field in the table, click Apply to add the selected rules to the Rules column of the table.

Applying an Update Rule will display a prompt to select the fields used in the calculation. The Output Field can either be a new field by entering the desired field name in the textbox and clicking Add, or an existing field by selecting it from the drop-down list.

After assigning the desired rules to the fields, an Index can be created alongside the Rules-Based Data Source by enabling the Create Index checkbox. This will reveal the Add to Search Engine option, allowing the resulting Index to be added to a specific Search Engine.

Click Create Data Source to complete the process, or Cancel to discard all changes.

Created Fields

Creating a Rules Data Source adds the following fields:

Any fields with an Update Rule applied will have an automatically populated Updated field.

Any fields with a Validation Rule applied will have a field specifying the action to be taken.

Rules Summary Data Source

A Rules Summary Data Source can be built in the Data Sources screen to provide an audit of all Rules and the outcomes of each data set run with rules applied.

To access the Data Sources Screen, click Search Engine, then Data Sources.

Select Rules Summary from the Type drop-down list to begin the creation process.

Creation Process

Enter the name that will be given to the Data Source in the Name textbox. This name must be unique.

Use the Tags textbox to add associated search terms to the data source. This allows access via customised strings when using the search bar.

To create an Index can be created alongside the Data Source, enable the Create Index checkbox. This will reveal the Add to Search Engine option, allowing the resulting Index to be added to a specific Search Engine.

Click Create Data Source to complete the process, or Cancel to discard all changes.

Created Fields

Rules Summary Data Sources contain a row for each time a Rule has been run. The following fields are created:

Modifying Rules

When making changes to Rules, it is important to ensure the relevant steps are taken to refresh the data.

To modify created Rules, navigate to the Rules screen, by clicking Setup, Extensions then Rules, and click the Edit icon next to the relevant Rule. Make the relevant changes and click Modify Rule to save any changes, or Cancel to discard any changes.

To add or remove rules from a Data Source, navigate back to the Data Sources screen, by clicking Search Engine then Data Sources, and click the Edit icon next to the relevant Rules Data Source. Make the relevant changes and click Modify Data Source to save any changes, or Cancel to discard any changes.

Finally, navigate to the Data Source Groups screen by clicking Search Engine, then Data Source Groups. Click the Build icon next to the relevant Data Source to rebuild using the latest iteration of any Rule configuration.

If the build method has been set to Cumulative, rebuilding a Rules Data Source will result in zero records. Click the Build icon again to apply new Rule changes and repopulate the data.

Rules Promotion

Once a Rules Index has been created, the Rules Promotion Data Source allows an administrator to specify the maximum number of errors before an Index is prevented from being made available to end users. If the maximum number of minor or major errors is breached, the new version of the data will not be available in the front-end and the previous version will remain in place until the errors are corrected and the data refreshed.

To use this functionality, navigate to the Data Sources screen by clicking Search Engine then Data Sources and select Rules Promotion from the Type drop-down list. Select a previously created Rules Index from the Data Source Group drop-down list and specify the error thresholds using the Maximum Minor Error and Maximum Major Error textboxes. Enable the Percent checkbox for either option to instead use the entered number as a percentage value.

After entering the required information, an Index can be created alongside the Data Source by enabling the Create Index checkbox. This will reveal the Add to Search Engine option, allowing the resulting Index to be added to a specific Search Engine.

Click Create Data Source to complete the process, or Cancel to discard all changes.