Viglet Turing ES: Developer Guide

Preface

Viglet Turing ES (https://viglet.com/turing) is an open source solution (https://github.com/openturing), which has Semantic Navigation and Chatbot as its main features. You can choose from several NLPs to enrich the data. All content is indexed in Solr as search engine.

1. More Documentation

Technical documentation on Turing ES is available at https://docs.viglet.com/turing.

2. Open Source Development

You can collaborate with Turing, participating in its development. Below are the steps to create your Turing environment.

2.1. Development Structure

2.1.1. Frameworks

Turing ES was developed using Spring Boot (https://spring.io/projects/spring-boot) for its backend.

The UI is currently using AngularJS (https://angularjs.org), but a new UI is being developed using Angular 12 (https://angular.io) with Primer CSS (https://primer.style/css).

In addition to Java, you also need to have Git (https://git-scm.com/downloads) and NodeJS (https://nodejs.org/en/download/) installed.

2.1.2. Databases

By default it uses the H2 database (https://www.h2database.com), but can be changed to other databases using Spring Boot properties. It comes bundled with OpenNLP (https://opennlp.apache.org/) in the same JVM.

2.1.3. Programming Language and Deploy

It uses Java 21 (https://adoptium.net/temurin/releases/?package=jdk&version=21) and its deployment is done with Maven and works on Unix and Windows.

2.1.4. Docker

To use Semantic Navigation and Chatbot you must have a Solr (https://solr.apache.org) service available. If you prefer to work with all the services Turing depends on, you can use docker-compose (https://docs.docker.com/compose/install) to start these services, we use the Docker Desktop (https://www.docker.com/products/docker-desktop) installed on computer.

2.1.5. IDE

You can use Spring Tools 4 for Eclipse (https://spring.io/tools) or Eclipse (https://www.eclipse.org/downloads/) or Visual Studio Code (https://code.visualstudio.com/) or IntelliJ (https://www.jetbrains.com/pt-br/idea/) as IDEs.

2.2. Download

Use the git command line to download the repository to your computer.

2.2.1. Turing Server and Connectors

git clone https://github.com/openturing/turing.git

2.3. Run during Development

To run Turing ES, execute the following lines:

2.3.1. Turing Server

Development

With UI

cd turing
mvn spring-boot:run -pl turing-app

Without update UI

cd turing
mvn spring-boot:run -pl turing-app -Dskip.npm

New Turing UI

Start the Turing Server using dev-ui profile

cd turing
mvn spring-boot:run -pl turing-app -Dskip.npm  -Dspring-boot.run.profiles=dev-ui

And start one of the components of turing-ui

cd turing/turing-ui

## Console
ng serve console

## Search
ng serve sn

## Chatbot
ng serve converse

## Chatbot
ng serve welcome

Important

You need start the Turing Server and Solr first.

Build

cd turing
mvn clean install
mvn package -pl turing-app

2.3.2. Java SDK

Development

cd turing-java-sdk
mvn package
java -cp build/libs/turing-java-sdk-all.jar com.viglet.turing.client.sn.sample.TurSNClientSample

Important

You need start the Turing Server and Solr first.

2.3.3. Build

cd turing-java-sdk
mvn package

Or use the jitpack into your project at https://jitpack.io/#openturing/turing-java-sdk

2.3.4. WEM Listener

cd turing
mvn package -pl turing-wem

For development, copy the turing-wem/build/libs/turing-wem-all.jar into WEM_DIR/libs and test the listener using turing-wem command line.

Important

You need start the Turing Server and Solr first and restart WEM

2.3.5. Database Connector

cd turing
mvn package -pl turing-db

Important

You need start the Turing Server and Solr first and restart WEM

2.3.6. Filesystem Connector

cd turing
mvn package -pl turing-filesystem

Important

You need start the Turing Server and Solr first and restart WEM

2.3.7. Nutch

Nutch 1.20

cd turing/
mvn package -pl turing-nutch1_20

For development, copy the files of turing-nutch/nutch1_18/build/extracted_dist to APACHE_NUTCH1_18/plugins/indexer-viglet-turing

Important

You need start the Turing Server and Solr first.

2.4. URLs

2.4.1. Turing Server

Administration Console: http://localhost:2700. (admin/admin)
Semantic Navigation Sample: http://localhost:2700/sn/Sample.

2.4.2. New Turing UI

Welcome http://localhost:4200/welcome
Console http://localhost:4200/console
Search Page http://localhost:4200/sn/template?_setsite=Sample&_setlocale=en_US
Converse http://localhost:4200/converse

2.4.3. Docker Compose

Administration Console: http://localhost. (admin/admin)
Semantic Navigation Sample: http://localhost/sn/Sample.
Solr: http://localhost:8983

2.4.4. Code Quality

You can check the quality of Turing Code at:

SonarCloud at https://sonarcloud.io/organizations/viglet-turing/projects
Github Actions at https://github.com/openturing/turing/actions
Github Security at https://github.com/openturing/turing/security/code-scanning
Codecov at https://app.codecov.io/gh/openturing/turing

3. REST API

3.1. API Overview

Turing ES offers a variety of robust, convenient, and simple RESTful Web service APIs to integrate data from Turing to any external system. Through Turing’s API, your developers can create Web applications to interact directly with data that resides in Turing. Among the available features include RESTful APIs using JSON format, authentication via API Key invoking the existing user-level governance and security model built into Turing as well as a developer to manage access to API documentation and API keys. We have APIs to deliver search and cognitive features.

3.2. OpenAPI 3.0

The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to HTTP APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with a minimal amount of implementation logic.

An OpenAPI definition can then be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases.

Turing OpenAPI 3.0 is available at http://localhost:2700/v3/api-docs.

3.3. Swagger

Swagger allows you to describe the structure of your APIs so that machines can read them. The ability of APIs to describe their own structure is the root of all awesomeness in Swagger. Swagger does this by asking your API to return a YAML or JSON that contains a detailed description of your entire API. This file is essentially a resource listing of your API which adheres to OpenAPI Specification.

You can access the Turing API documentation and test it directly using Swagger at http://localhost:2700/swagger-ui.html.

3.4. Generate an API Key

3.4.1. Step 1

3.4.2. Step 2

Access API Token Section.

3.4.3. Step 3

Create a new API Token with Title and Description.

3.4.4. Step 4

Will be create a new random API Token.

3.5.1. Search

Search on the semantic navigation site.

API Endpoint

GET|POST http://localhost:2700/api/sn/{{siteName}}/search

Headers

API headers include the mandatory information you send along with the request URL and body. This information helps provide insights into request context and authorization credentials that, in turn, allows access to protected resources.

Key	Value	Description
Key	api-key	API key helps authenticate the application with the server. For generating API key, refer to Generate an API Key
Content-Type	application/json	Content-Type is a representation header that determines the type of data (media/resource) present in the request body
Accept	application/json	Determines the acceptable response type from the server

Key

Value

Description

Key

api-key

API key helps authenticate the application with the server. For generating API key, refer to Generate an API Key

Content-Type

application/json

Content-Type is a representation header that determines the type of data (media/resource) present in the request body

application/json

Determines the acceptable response type from the server

Path Parameters

Attribute	Required / Optional	Description	Example
siteName	Required	Site Name	Sample

Attribute

Required / Optional

Description

Example

siteName

Required

Site Name

Sample

Query String

Attribute Required / Optional Description Example

Attribute	Required / Optional	Description	Example
q	Required	Search Query.	q=foo
p	Required	Page Number, first page is 1. DEFAULT: 1	p=1
sort	Required	Sort values: `relevance`, `newest` and `oldest`. Or FIELD: SORT DEFAULT: relevance	sort=relevance or sort=title:asc
fq[]	Optional	Query Field. Filter by field using default operator in configuration or using the `fqOperator`(request) or `fq.op` (query string), using the following pattern: FIELD: VALUE.	fq[]=title:bar
fq.and[]	Optional	Query Field. Filter `AND` by field, using the following pattern: FIELD: VALUE.	fq.and[]=title:bar
fq.or[]	Optional	Query Field. Filter `OR` by field, using the following pattern: FIELD: VALUE.	fq.or[]=title:bar
fq.op	Optional	Query Field Operator. Filter OR by field, Operator values: `AND` and `OR`.	fq.op=AND
rows	Optional	Number of rows that query will return. DEFAULT: Behavior Configuration	rows=10
_setlocale	Required	Locale of Semantic Navigation Site	_setlocale=en_US
nfpr	Optional	Disable Auto Correction DEFAULT: 0	nfpr=1
group	Optional	Group by attribute	group=type

Required

Search Query.

q=foo

Required

Page Number, first page is 1. DEFAULT: 1

p=1

sort

Required

Sort values: relevance, newest and oldest. Or FIELD: SORT DEFAULT: relevance

sort=relevance or sort=title:asc

fq[]

Optional

Query Field. Filter by field using default operator in configuration or using the fqOperator(request) or fq.op (query string), using the following pattern: FIELD: VALUE.

fq[]=title:bar

fq.and[]

Optional

Query Field. Filter AND by field, using the following pattern: FIELD: VALUE.

fq.and[]=title:bar

fq.or[]

Optional

Query Field. Filter OR by field, using the following pattern: FIELD: VALUE.

fq.or[]=title:bar

fq.op

Optional

Query Field Operator. Filter OR by field, Operator values: AND and OR.

fq.op=AND

rows

Optional

Number of rows that query will return. DEFAULT: Behavior Configuration

rows=10

_setlocale

Required

Locale of Semantic Navigation Site

_setlocale=en_US

nfpr

Optional

Disable Auto Correction DEFAULT: 0

nfpr=1

group

Optional

Group by attribute

group=type

Request Parameters

Important

These parameters have precedence over the query string.

Attribute Type Required / Optional Description

Attribute	Type	Required / Optional	Description
query	String	Required	Search Query.
page	Integer	Required	Page Number, first page is 1. DEFAULT: 1
sort	String	Required	Sort values: `relevance`, `newest` and `oldest`. Or FIELD: SORT. DEFAULT: relevance
fq	String	Optional	Query Field. Filter by field using default operator in configuration or using the `fqOperator`(request) or `fq.op` (query string), using the following pattern: FIELD: VALUE.
fqAnd	String	Optional	Query Field. Filter `AND` by field, using the following pattern: FIELD: VALUE.
fqOr	String	Optional	Query Field. Filter `OR` by field, using the following pattern: FIELD: VALUE.
fqOperator	String	Optional	Query Field Operator. Filter OR by field, Operator values: `AND` and `OR`.
rows	Integer	Optional	Number of rows that query will return. DEFAULT: Behavior Configuration
locale	String	Required	Locale of Semantic Navigation Site
group	String	Optional	Group by attribute
targetingRules	String[]	Optional	Targeting Rules. Restrict search based in: FIELD: VALUE.
targetingRulesWithCondition	Map<String, String[]>	Optional	Targueting Rules with Condition
targetingRulesWithConditionAND	Map<String, String[]>	Optional	Targueting Rules `AND` with Condition
targetingRulesWithConditionOR	Map<String, String[]>	Optional	Targueting Rules `OR` with Condition
populateMetrics	Boolean	Optional	If populate metrics
userId	String	Optional	User Id
disableAutoComplete	Boolean	Optional	Disable Auto Correction. DEFAULT: false

query

String

Required

Search Query.

page

Integer

Required

Page Number, first page is 1. DEFAULT: 1

sort

String

Required

Sort values: relevance, newest and oldest. Or FIELD: SORT. DEFAULT: relevance

String

Optional

Query Field. Filter by field using default operator in configuration or using the fqOperator(request) or fq.op (query string), using the following pattern: FIELD: VALUE.

fqAnd

String

Optional

Query Field. Filter AND by field, using the following pattern: FIELD: VALUE.

fqOr

String

Optional

Query Field. Filter OR by field, using the following pattern: FIELD: VALUE.

fqOperator

String

Optional

Query Field Operator. Filter OR by field, Operator values: AND and OR.

rows

Integer

Optional

Number of rows that query will return. DEFAULT: Behavior Configuration

locale

String

Required

Locale of Semantic Navigation Site

group

String

Optional

Group by attribute

targetingRules

String[]

Optional

Targeting Rules. Restrict search based in: FIELD: VALUE.

targetingRulesWithCondition

Map<String, String[]>

Optional

Targueting Rules with Condition

targetingRulesWithConditionAND

Map<String, String[]>

Optional

Targueting Rules AND with Condition

targetingRulesWithConditionOR

Map<String, String[]>

Optional

Targueting Rules OR with Condition

populateMetrics

Boolean

Optional

If populate metrics

userId

String

Optional

User Id

disableAutoComplete

Boolean

Optional

Disable Auto Correction. DEFAULT: false

Regular Search

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&p=1&fq[]=state:WA&sort=relevance&rows=10&nfpr=0&_setlocale=en_US' \
  -H 'Content-Type: application/json'

curl -X 'POST' \
  'http://localhost:2700/api/sn/Sample/search' \
  -H 'Key: {{Enter your API KEY}}' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "foobar",
  "fq": "state:WA",
  "sort": "relevance",
  "rows": 10,
  "locale": "en_US"
}'

Example - Response

{
    "pagination": [
        {
            "type": "CURRENT",
            "text": "1",
            "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance",
            "page": 1
        },
        {
            "type": "PAGE",
            "text": "2",
            "href": "/api/sn/Sample/search?q=foobar&p=2&_setlocale=en_US&sort=relevance",
            "page": 2
        },
        {
            "type": "PAGE",
            "text": "3",
            "href": "/api/sn/Sample/search?q=foobar&p=3&_setlocale=en_US&sort=relevance",
            "page": 3
        },
        {
            "type": "PAGE",
            "text": "4",
            "href": "/api/sn/Sample/search?q=foobar&p=4&_setlocale=en_US&sort=relevance",
            "page": 4
        },
        {
            "type": "NEXT",
            "text": "Next",
            "href": "/api/sn/Sample/search?q=foobar&p=2&_setlocale=en_US&sort=relevance",
            "page": 2
        },
        {
            "type": "LAST",
            "text": "Last",
            "href": "/api/sn/Sample/search?q=foobar&p=480&_setlocale=en_US&sort=relevance",
            "page": 480
        }
    ],
    "queryContext": {
        "count": 4795,
        "index": "Sample",
        "limit": 10,
        "offset": 0,
        "page": 1,
        "pageCount": 480,
        "pageEnd": 10,
        "pageStart": 1,
        "responseTime": 51,
        "query": {
            "queryString": "foobar",
            "sort": "relevance",
            "locale": "en_US"
        },
        "defaultFields": {
            "title": "title",
            "date": "publicationDate",
            "description": "texts",
            "text": "text",
            "image": "image",
            "url": "url"
        }
    },
    "results": {
        "document": [
            {
                "source": "https://www.example.com/0df43c64ebeeb710VgnVCM100000d701210aRCRD",
                "elevate": false,
                "metadata": [
                    {
                        "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=site%3ASample Site",
                        "text": "Sample Site"
                    },
                    {
                        "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=categories%3AInova%C3%A7%C3%A3o",
                        "text": "Inovation"
                    },
                    {
                        "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=type%3ANews",
                        "text": "News"
                    }
                ],
                "fields": {
                    "author": "john",
                    "description": "Loren",
                    "title": "Loren",
                    "type": "News",
                    "url": "https://www.example.com/0df43c64ebeeb710VgnVCM100000d701210aRCRD",
                    "state": [
                        "MA"
                    ],
                    "modificationDate": "2021-09-16T13:28:54.000+00:00",
                    "site": "Sample Site",
                    "texts": [
                        "..."
                    ],
                    "provider": "Sample Site",
                    "_version_": 1765073243578826752,
                    "id": "0df43c64ebeeb710VgnVCM100000d701210aRCRD",
                    "categories": [
                        "Inovation"
                    ],
                    "source_apps": [
                        "WEM"
                    ],
                    "publicationDate": "2021-09-16T13:28:59.000+00:00"
                }
            }
        ]
    },
    "groups": [],
    "widget": {
        "facet": [
            {
                "facets": [
                    {
                        "count": 2343,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=type%3AArticle",
                        "label": "Article"
                    },
                    {
                        "count": 485,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=type%3ANews",
                        "label": "News"
                    },
                    {
                        "count": 267,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=type%3AVideo",
                        "label": "Video"
                    },
                    {
                        "count": 171,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=type%3AFaq",
                        "label": "Faq"
                    }
                ],
                "label": {
                    "lang": "en",
                    "text": "Types"
                },
                "name": "type",
                "description": "Content Type Field",
                "type": "STRING",
                "multiValued": false
            },
            {
                "facets": [
                    {
                        "count": 4764,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=site%3ASample Site",
                        "label": "Sample Site"
                    }
                ],
                "label": {
                    "lang": "en",
                    "text": "Sites"
                },
                "name": "site",
                "description": "Site Name",
                "type": "STRING",
                "multiValued": false
            },
            {
                "facets": [
                    {
                        "count": 2603,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=categories%3AInova%C3%A7%C3%A3o",
                        "label": "Inovation"
                    },
                    {
                        "count": 300,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=categories%3AINOVA%C3%87%C3%83O",
                        "label": "Inovation"
                    }
                ],
                "label": {
                    "lang": "en",
                    "text": "Categories"
                },
                "name": "categories",
                "description": "Categories",
                "type": "STRING",
                "multiValued": true
            },
            {
                "facets": [
                    {
                        "count": 2145,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=state%3ANA",
                        "label": "NA"
                    },
                    {
                        "count": 325,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=state%3ASC",
                        "label": "SC"
                    },
                    {
                        "count": 297,
                        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=state%3APE",
                        "label": "PE"
                    }
                ],
                "label": {
                    "lang": "en",
                    "text": "States"
                },
                "name": "state",
                "description": "State",
                "type": "STRING",
                "multiValued": true
            }
        ],
        "facetToRemove": null,
        "similar": null,
        "spellCheck": {
            "correctedText": false,
            "usingCorrectedText": true,
            "original": {
                "text": "foobar",
                "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&nfpr=1"
            },
            "corrected": {
                "text": "",
                "link": "/api/sn/Sample/search?q=&p=1&_setlocale=en_US&sort=relevance"
            }
        },
        "locales": [
            {
                "locale": "en_US",
                "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance"
            }
        ],
        "spotlights": []
    }
}

Targeting Rules

Search on the semantic navigation site adding segmentation, restricting content based on defined segments.

Example - Request

curl -X 'POST' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&p=1&fq%5B%5D=state%3AWA&sort=relevance&rows=10&nfpr=0&_setlocale=en_US' \
  -H 'Content-Type: application/json' \
  -H 'Key: {{Enter your API KEY}}' \
  -d '{
  "userId": "string",
  "populateMetrics": true,
  "targetingRules": [
    "department:HR",
    "profile:Principal"
  ]
}'

Example - Request

curl -X 'POST' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&p=1&fq%5B%5D=state%3AWA&sort=relevance&rows=10&nfpr=0&_setlocale=en_US' \
  -H 'Content-Type: application/json' \
  -H 'Key: {{Enter your API KEY}}' \
  -d '{
  "userId": "string",
  "populateMetrics": true,
  "targetingRulesCondition": {
        "type:News": [
            "department:HR",
            "profile:Principal"
        ],
        "type:Article": [
            "department:IT",
            "profile:Consultant"
        ]
   },
   "targetingRulesConditionAND": {
        "type:Event": [
            "department:HR",
            "profile:Manager"
        ],
        "type:Report": [
            "department:IT",
            "profile:Coordinator"
        ]
   },
   "targetingRulesConditionOR": {
        "type:Faq": [
            "department:HR",
            "profile:Junior"
        ],
        "type:Text": [
            "department:IT",
            "profile:Junior"
        ]
   }
}'

Group By

Search on the semantic navigation site by grouping the search into groups.

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&p=1&group=type&sort=relevance&rows=10_setlocale=en_US' \
  -H 'Content-Type: application/json'

Example - Request

curl -X 'POST' \
  'http://localhost:2700/api/sn/Sample/search' \
  -H 'Key: {{Enter your API KEY}}' \
  -H 'Content-Type: application/json' \
  -d '{
  "query": "foobar",
  "page": 1,
  "group": "type",
  "sort": "relevance",
  "rows": 10,
  "locale": "en_US"
}'

Example - Response

{
  "pagination": [
    {
      "type": "CURRENT",
      "text": "1",
      "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type",
      "page": 1
    }
  ],
  "queryContext": {
    "count": 4795,
    "index": "Sample",
    "limit": 10,
    "offset": 0,
    "page": 1,
    "pageCount": 480,
    "pageEnd": 10,
    "pageStart": 1,
    "responseTime": 73,
    "query": {
      "queryString": "foobar",
      "sort": "relevance",
      "locale": "en_US"
    },
    "defaultFields": {
      "title": "title",
      "date": "publicationDate",
      "description": "texts",
      "text": "text",
      "image": "image",
      "url": "url"
    }
  },
  "results": { "document": [] },
  "groups": [
    {
      "name": "News",
      "count": 485,
      "page": 1,
      "pageCount": 49,
      "pageEnd": 10,
      "pageStart": 1,
      "limit": 10,
      "results": {
        "document": [
          {
            "source": "https://www.example.com/ufs/ma/noticias/premio-nacional-de-foobar-recebe-inscricoes-ate-2-de-outubro,0df43c64ebeeb710VgnVCM100000d701210aRCRD",
            "elevate": false,
            "metadata": [
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=state%3AMA",
                "text": "MA"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=site%3ASample Site",
                "text": "Sample Site"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=categories%3AInova%C3%A7%C3%A3o",
                "text": "Inovation"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=type%3ANews",
                "text": "News"
              }
            ],
            "fields": {
              "author": "john",
              "description": "...",
              "title": "...",
              "type": "News",
              "url": "https://www.example.com/0df43c64ebeeb710VgnVCM100000d701210aRCRD",
              "state": ["MA"],
              "modificationDate": "2021-09-16T13:28:54.000+00:00",
              "site": "Sample Site",
              "texts": [
                "..."
              ],
              "provider": "Sample Site",
              "_version_": 1765073243578826752,
              "id": "0df43c64ebeeb710VgnVCM100000d701210aRCRD",
              "categories": ["Inovation"],
              "source_apps": ["WEM"],
              "publicationDate": "2021-09-16T13:28:59.000+00:00"
            }
          },
          {
            "source": "https://www.example.com/ufs/1e929c110e49c710VgnVCM100000d701210aRCRD",
            "elevate": false,
            "metadata": [
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=state%3AMA",
                "text": "MA"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=site%3ASample Site",
                "text": "Sample Site"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=categories%3AInova%C3%A7%C3%A3o",
                "text": "Inovation"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=type%3ANews",
                "text": "News"
              }
            ],
            "fields": {
              "author": "john",
              "description": "...",
              "title": "...",
              "type": "News",
              "url": "https://www.example.com/1e929c110e49c710VgnVCM100000d701210aRCRD",
              "state": ["MA"],
              "modificationDate": "2021-10-18T20:01:04.000+00:00",
              "site": "Sample Site",
              "texts": [
                "..."
              ],
              "provider": "Sample Site",
              "_version_": 1765073243259011072,
              "id": "1e929c110e49c710VgnVCM100000d701210aRCRD",
              "categories": ["Inovation"],
              "source_apps": ["WEM"],
              "publicationDate": "2021-10-18T20:01:11.000+00:00"
            }
          }

        ]
      },
      "pagination": [
        {
          "type": "CURRENT",
          "text": "1",
          "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=News",
          "page": 1
        },
        {
          "type": "PAGE",
          "text": "2",
          "href": "/api/sn/Sample/search?q=foobar&p=2&_setlocale=en_US&sort=relevance&fq[]=News",
          "page": 2
        },
        {
          "type": "PAGE",
          "text": "3",
          "href": "/api/sn/Sample/search?q=foobar&p=3&_setlocale=en_US&sort=relevance&fq[]=News",
          "page": 3
        },
        {
          "type": "PAGE",
          "text": "4",
          "href": "/api/sn/Sample/search?q=foobar&p=4&_setlocale=en_US&sort=relevance&fq[]=News",
          "page": 4
        },
        {
          "type": "NEXT",
          "text": "Next",
          "href": "/api/sn/Sample/search?q=foobar&p=2&_setlocale=en_US&sort=relevance&fq[]=News",
          "page": 2
        },
        {
          "type": "LAST",
          "text": "Last",
          "href": "/api/sn/Sample/search?q=foobar&p=49&_setlocale=en_US&sort=relevance&fq[]=News",
          "page": 49
        }
      ]
    },
    {
      "name": "Article",
      "count": 2343,
      "page": 1,
      "pageCount": 235,
      "pageEnd": 10,
      "pageStart": 1,
      "limit": 10,
      "results": {
        "document": [
          {
            "source": "https://www.example.com/dc926de4cced1810VgnVCM100000d701210aRCRD",
            "elevate": false,
            "metadata": [
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=state%3ANA",
                "text": "NA"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=site%3ASample Site",
                "text": "Sample Site"
              },
              {
                "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=type%3AArticle",
                "text": "Article"
              }
            ],
            "fields": {
              "author": "john",
              "htmls": [
                "..."
              ],
              "description": "...",
              "title": "...",
              "type": "Article",
              "url": "https://www.example.com/dc926de4cced1810VgnVCM100000d701210aRCRD",
              "state": ["NA"],
              "modificationDate": "2022-07-12T14:00:58.000+00:00",
              "site": "Sample Site",
              "texts": [
                "..."
              ],
              "provider": "Sample Site",
              "_version_": 1765071551980371968,
              "id": "dc926de4cced1810VgnVCM100000d701210aRCRD",
              "categories": ["Inovation"],
              "source_apps": ["WEM"],
              "publicationDate": "2022-07-12T14:03:16.000+00:00"
            }
          }
        ]
      },
      "pagination": [
        {
          "type": "CURRENT",
          "text": "1",
          "href": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&fq[]=Article",
          "page": 1
        },
        {
          "type": "PAGE",
          "text": "2",
          "href": "/api/sn/Sample/search?q=foobar&p=2&_setlocale=en_US&sort=relevance&fq[]=Article",
          "page": 2
        },
        {
          "type": "PAGE",
          "text": "3",
          "href": "/api/sn/Sample/search?q=foobar&p=3&_setlocale=en_US&sort=relevance&fq[]=Article",
          "page": 3
        },
        {
          "type": "PAGE",
          "text": "4",
          "href": "/api/sn/Sample/search?q=foobar&p=4&_setlocale=en_US&sort=relevance&fq[]=Article",
          "page": 4
        },
        {
          "type": "NEXT",
          "text": "Next",
          "href": "/api/sn/Sample/search?q=foobar&p=2&_setlocale=en_US&sort=relevance&fq[]=Article",
          "page": 2
        },
        {
          "type": "LAST",
          "text": "Last",
          "href": "/api/sn/Sample/search?q=foobar&p=235&_setlocale=en_US&sort=relevance&fq[]=Article",
          "page": 235
        }
      ]
    }

  ],
  "widget": {
    "facet": [
      {
        "facets": [
          {
            "count": 2343,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=type%3AArticle",
            "label": "Article"
          },
          {
            "count": 485,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=type%3ANews",
            "label": "News"
          }
        ],
        "label": { "lang": "en", "text": "Types" },
        "name": "type",
        "description": "Content Type Field",
        "type": "STRING",
        "multiValued": false
      },
      {
        "facets": [
          {
            "count": 4764,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=site%3ASample Site",
            "label": "Sample Site"
          }
        ],
        "label": { "lang": "en", "text": "Sites" },
        "name": "site",
        "description": "Site Name",
        "type": "STRING",
        "multiValued": false
      },
      {
        "facets": [
          {
            "count": 2603,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=categories%3AInova%C3%A7%C3%A3o",
            "label": "Inovation"
          },
          {
            "count": 1106,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=categories%3AEmpreendedorismo",
            "label": "Empreendedorismo"
          }
        ],
        "label": { "lang": "en", "text": "Categories" },
        "name": "categories",
        "description": "Categories",
        "type": "STRING",
        "multiValued": true
      },
      {
        "facets": [
          {
            "count": 2145,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=state%3ANA",
            "label": "NA"
          },
          {
            "count": 325,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=state%3ASC",
            "label": "SC"
          },
          {
            "count": 297,
            "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&fq[]=state%3APE",
            "label": "PE"
          }
        ],
        "label": { "lang": "en", "text": "States" },
        "name": "state",
        "description": "State",
        "type": "STRING",
        "multiValued": true
      }
    ],
    "facetToRemove": null,
    "similar": null,
    "spellCheck": {
      "correctedText": false,
      "usingCorrectedText": true,
      "original": {
        "text": "foobar",
        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type&nfpr=1"
      },
      "corrected": {
        "text": "",
        "link": "/api/sn/Sample/search?q=&p=1&_setlocale=en_US&sort=relevance&group=type"
      }
    },
    "locales": [
      {
        "locale": "en_US",
        "link": "/api/sn/Sample/search?q=foobar&p=1&_setlocale=en_US&sort=relevance&group=type"
      }
    ],
    "spotlights": []
  }
}

Dates

Search on the semantic navigation site between Dates

Important

You will use fq[], fq.and[] and fq.or[] (query string) or fq, fqAnd, fqOr (request body) to define the dates using the expression with the following date format: yyyy-MM-dd’T’HH:mm:ss’Z

Between Dates

fq[]=attribute:[InitialDate TO EndDate].

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&_setlocale=en_US&fq[]=modificationDate:[2023-07-23T00:00:00Z+TO+2023-11-30T00:00:00Z]' \
  -H 'Content-Type: application/json'

From a date

fq[]=attribute:[InitialDate TO *].

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&_setlocale=en_US&fq[]=modificationDate:[2023-07-23T00:00:00Z+TO+*]' \
  -H 'Content-Type: application/json'

Until a date

fq[]=attribute:[* TO EndDate].

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/search?q=foobar&_setlocale=en_US&fq[]=modificationDate:[*+TO+2023-11-30T00:00:00Z]' \
  -H 'Content-Type: application/json'

3.5.2. Search Latest

Returns the last terms searched by the user in the search.

API Endpoint

POST http://localhost:2700/api/sn/{{siteName}}/search/latest

Headers

Key	Value	Description
Key	api-key	API key helps authenticate the application with the server. For generating API key, refer to Generate an API Key
Content-Type	application/json	Content-Type is a representation header that determines the type of data (media/resource) present in the request body
Accept	application/json	Determines the acceptable response type from the server

Key

Value

Description

Key

api-key

API key helps authenticate the application with the server. For generating API key, refer to Generate an API Key

Content-Type

application/json

Content-Type is a representation header that determines the type of data (media/resource) present in the request body

application/json

Determines the acceptable response type from the server

Path Parameters

Attribute	Required / Optional	Description	Example
siteName	Required	Site Name	Sample

Attribute

Required / Optional

Description

Example

siteName

Required

Site Name

Sample

Query String

Attribute	Type	Required / Optional	Description	Default	Example
q	String	Required	Search Query.	-	q=foo
rows	Integer	Optional	Number of rows that query will return.	5	rows=10
_setlocale	Boolean	Required	Locale of Semantic Navigation Site	-	_setlocale=en_US

Attribute

Type

Required / Optional

Description

Default

Example

String

Required

Search Query.

q=foo

rows

Integer

Optional

Number of rows that query will return.

rows=10

_setlocale

Boolean

Required

Locale of Semantic Navigation Site

_setlocale=en_US

Request Parameters

Attribute	Type	Required / Optional	Description	Example
userId	String	Required	User Id	userId=john

Attribute

Type

Required / Optional

Description

Example

userId

String

Required

User Id

userId=john

Example - Request

curl -X POST \
  'http://localhost:2700/api/sn/Sample/search/latest?rows=5&_setlocale=en_US' \
  -H 'Content-Type: application/json'
  -H 'Key: {{Enter your API KEY}}'
  -d '{
  "userId": "john"
  }'

Example - Response

[
  "foo",
  "bar"
]

3.5.3. Search Locales

List all locales on the semantic navigation site.

API Endpoint

GET http://localhost:2700/api/sn/{{siteName}}/search/locales

Path Parameters

Attribute	Required / Optional	Description	Example
siteName	Required	Site Name	Sample

Attribute

Required / Optional

Description

Example

siteName

Required

Site Name

Sample

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/search/locales' \
  -H 'Content-Type: application/json'

Example - Response

[
  {
    "locale": "en_US",
    "link": "/api/sn/Sample/search?_setlocale=en_US"
  },
  {
    "locale": "pt_BR",
    "link": "/api/sn/Sample/search?_setlocale=pt_BR"
  }
]

3.5.4. Auto Complete

Returns a term array that starts with the query value.

API Endpoint

GET http://localhost:2700/api/sn/{{siteName}}/ac

Path Parameters

Attribute	Required / Optional	Description	Example
siteName	Required	Site Name	Sample

Attribute

Required / Optional

Description

Example

siteName

Required

Site Name

Sample

Query String

Attribute	Type	Required / Optional	Description	Example
q	String	Required	Search Query.	q=foo
rows	Integer	Optional	Number of rows that query will return.	rows=10
_setlocale	Boolean	Required	Locale of Semantic Navigation Site	_setlocale=en_US

Attribute

Type

Required / Optional

Description

Example

String

Required

Search Query.

q=foo

rows

Integer

Optional

Number of rows that query will return.

rows=10

_setlocale

Boolean

Required

Locale of Semantic Navigation Site

_setlocale=en_US

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/ac?q=dis&rows=10&_setlocale=en_US' \
  -H 'Content-Type: application/json'

Example - Response

[
  "disc",
  "discovery"
  "disco"
  "disney"
]

3.5.5. Spell Check

Corrects the text based on the semantic navigation site search database in a given language.

API Endpoint

GET http://localhost:2700/api/sn/{{siteName}}/{{locale}}/spell-check

Path Parameters

Attribute	Type	Required / Optional	Description	Example
sitename	String	Required	Site Name.	Sample
locale	String	Required	Locale of Semantic Navigation Site	en_US

Attribute

Type

Required / Optional

Description

Example

sitename

String

Required

Site Name.

Sample

locale

String

Required

Locale of Semantic Navigation Site

en_US

Query String

Attribute	Type	Required / Optional	Description	Example
q	String	Required	Search Query.	q=foo
rows	Integer	Optional	Number of rows that query will return.	rows=10
_setlocale	Boolean	Required	Locale of Semantic Navigation Site	_setlocale=en_US

Attribute

Type

Required / Optional

Description

Example

String

Required

Search Query.

q=foo

rows

Integer

Optional

Number of rows that query will return.

rows=10

_setlocale

Boolean

Required

Locale of Semantic Navigation Site

_setlocale=en_US

Example - Request

curl -X 'GET' \
  'http://localhost:2700/api/sn/Sample/en_US/spell-check?q=fuu' \
  -H 'Content-Type: application/json'

Example - Response

{
  "correctedText": true,
  "usingCorrectedText": false,
  "original": {
    "text": "fuu",
    "link": "/api/sn/Sample/en_US/spell-check?q=fuu&nfpr=1"
  },
  "corrected": {
    "text": "foo",
    "link": "/api/sn/Sample/en_US/spell-check?q=foo"
  }
}

3.6. Cognitive

3.6.1. Spell Check

Corrects text based on the given language.

API Endpoint

GET http://localhost:2700/api/cognitive/spell-checker/{{locale}}

Headers

Key	Value	Description
Key	api-key	API key helps authenticate the application with the server. For generating API key, refer to Generate an API Key
Content-Type	application/json	Content-Type is a representation header that determines the type of data (media/resource) present in the request body
Accept	application/json	Determines the acceptable response type from the server

Key

Value

Description

Key

api-key

API key helps authenticate the application with the server. For generating API key, refer to Generate an API Key

Content-Type

application/json

Content-Type is a representation header that determines the type of data (media/resource) present in the request body

application/json

Determines the acceptable response type from the server

Path Parameters

Attribute	Type	Required / Optional	Description	Example
locale	String	Required	Locale of Semantic Navigation Site	en_US

Attribute

Type

Required / Optional

Description

Example

locale

String

Required

Locale of Semantic Navigation Site

en_US

Query String

Attribute	Type	Required / Optional	Description	Example
text	String	Required	Text to validated	text=fuu ber

Attribute

Type

Required / Optional

Description

Example

text

String

Required

Text to validated

text=fuu ber

Example - Request

curl -X GET \
  "http://localhost:2700/api/cognitive/spell-checker/en_US?text=urange" \
  -H "Accept: application/json" \
  -H "Key: {{Enter your API KEY}}"

Example - Response

[
  "range",
  "orange",
  "grange",
  "Grange",
  "Orange",
  "u range"
]

Preface

1. More Documentation

2. Open Source Development

2.1. Development Structure

2.1.1. Frameworks

2.1.2. Databases

2.1.3. Programming Language and Deploy

2.1.4. Docker

2.1.5. IDE

2.2. Download

2.2.1. Turing Server and Connectors

2.3. Run during Development

2.3.1. Turing Server

Development

With UI

Without update UI

New Turing UI

Build

2.3.2. Java SDK

Development

2.3.3. Build

2.3.4. WEM Listener

2.3.5. Database Connector

2.3.6. Filesystem Connector

2.3.7. Nutch

Nutch 1.20

2.4. URLs

2.4.1. Turing Server

2.4.2. New Turing UI

2.4.3. Docker Compose

2.4.4. Code Quality

3. REST API

3.1. API Overview

3.2. OpenAPI 3.0

3.3. Swagger

3.4. Generate an API Key

3.4.1. Step 1

3.4.2. Step 2

3.4.3. Step 3

3.4.4. Step 4

3.5. Semantic Navigation

3.5.1. Search

API Endpoint

Headers

Path Parameters

Query String

Request Parameters

Regular Search

Example - Request

Example - Response

Targeting Rules

Example - Request

Example - Request

Group By

Example - Request

Example - Request

Example - Response

Dates

Between Dates

Example - Request

From a date

Example - Request

Until a date

Example - Request

3.5.2. Search Latest

API Endpoint

Headers

Path Parameters

Query String

Request Parameters

Example - Request

Example - Response

3.5.3. Search Locales

API Endpoint

Path Parameters

Example - Request

Example - Response

3.5.4. Auto Complete

API Endpoint

Path Parameters