Semantria API 3.0€¦ · API will respond with the verbal representation of the sentiment....

Semantria API 3.0.1 Upgrade guide

Version 1.05 Date 02/20/2013 Status Final version Authors George Kozlov


Semantria API 3.0.1 Upgrade guide of 14

Summary

Semantria’s API 3.0 is the most significant release since Semantria API 1.0. Throughout the past year, we

have been tirelessly collecting feedback from our customers and the rapidly growing Natural Language

Processing market. Taking all this knowledge into account, we have developed a brand new text analytics

API which now better suits our clients’ stringent requirements.

Semantria’s API 3.0 upgrade is the logical next step from our previous release, API 2.1, which has been

successfully adopted by the market and by thousands of happy customers. Many of the existing features

have been enhanced with new properties, effectively easing use and increases possibilities. We have also

added a host of new features, making Semantria the best text analytics API available, delivering endless

flexibility for any user. Finally, we have improved our existing API principles to make them easier to

integrate with and use.

Here are the new features that make their debut in Semantria API 3.0:

A completely new infrastructure that elevates performance to a new level and won’t be

dependent on frequent configuration switching.

A new cost system which will let our customers choose which edition and features they wish to

use.

A new statistics end-point which gives users statistical information on usage as a whole or per

configuration.

Low-level analysis, POS tagging (Part Of Speech Markup) of whole texts.

Relationship detection between extracted entities, both from named and user entities.

A new language detection feature, which detects the language of the source text and responds

with the language name.

Sentiment-bearing phrase detection; Semantria can now suggest which phrases from the source

text may affect sentiment results. A great tool for adjusting sentiment.

Sentiment polarity is now available with the API out-of-the-box. Along with sentiment scores, the

API will respond with the verbal representation of the sentiment.

Reworked configuration API. To ease the life of developers, we’ve scrapped the previous “update”

approach in favor of an approach which is more relevant to REST, separate API calls.

HTTP compression for outgoing data. Users now have the ability to perform run-time data

compression.

Source text response. Semantria can now respond with the original source text along with

analytical results.

Fixed many known and reported bugs.



Compatibility

The new API 3.0 infrastructure supports both API 2.1 and API 3.0 protocols on the same infrastructure; it

is backwards compatible. This means that all existing customers using API 2.1 should not experience any

problems. It is important to note that while the new infrastructure supports both API protocols, the new

API 3.0 protocol is not compatible with the old API 2.1 protocol. While technically very similar, there are

significant differences in configuration management which make them incompatible with one another.

Every new Semantria API release exposes a new, unique link to keep them separated:

API 2.1 is available through the following URL: https://api21.semantria.com

API 3.0 is available through the following URL: https://api30.semantria.com

All API end-points can be divided into three logical groups: Information, Configuration and Data

Information API end-point have mostly remained untouched. We have increased the number of

return fields for the subscription end-point and reordered them logically. API 3.0 introduces a new

Statistics end-point designed to give users statistical information on API usage within a specific

timeframe.

Configuration API end-points have received the most changes. API 2.1 offered two configuration

end-points for every configuration entity, such as Configurations, Blacklists, Categories, Queries,

Entities, and Sentiment-Bearing Phrases. The new API 3.0 now offers 3 methods.

Data API end-points have not changed with API 2.1; users can use this aspect of the API the same

way the always have.

Additionally, API 3.0 significantly increases the amount of available analysis output. We’ve added two

completely new types of output: POS tagging and Entities Relation. Existing output types have been

enhanced with new fields also.

Configuration changes

Semantria’s API 2.1 configuration end-points were built with heavy influence from Google API. One

common “mutate” API call allowed users to perform many operations with different objects using the

same API call. The idea was to use the POST method to pass added, removed and updated records in one

complex object.

The approach of API 3.0 has changed to be more RESTful, offering two different methods, instead of the

single one.

According to REST principles, there should be should be one method for adding, one method for updating

and one method for removing records. Although Semantria’s 2.1 release didn’t follow this convention, API

3.0 now does.

https://api21.semantria.com/

https://api30.semantria.com/



API 2.1 approach

In API 2.1, the GET HTTP method was used for acquiring data from the server.

URL: https://api21.semantria.com/configurations.format Format: JSON & XML

MIME type: Content-Type: application/json or application/xml Request type: GET

Authentication: Required Limit: Per system calls limit and expiration date according to user subscription.

Sample request JSON:

GET https://api21.semantria.com/configurations.json

Sample response JSON:

HTTP/1.0 200 Request accepted and served. [ { “conf_id” : “23498367”, “name” : “A test configuration” } ]

The POST HTTP method was used for updating data on the server.


MIME type: Content-Type: application/json or application/xml Request type: POST


Request body: Parameters should be passed within JSON or XML object

added: List of the new configurations which should be added on the server. optional removed: List of the configurations which should be removed from the server. optional


POST https://api21.semantria.com/configurations.json { “added” : [ { “conf_id” : “b1df666c-df5e-44c3-86f6-d1f331024f19”, “name” : “A test configuration” } ], “removed” : [ “b1df666c-df5e-44c3-86f6-d1f331024f19” ] }

https://api21.semantria.com/configurations.format





HTTP/1.0 200 Request accepted and served.

API 3.0 approach

In API 3.0, the GET HTTP method remains the same.


MIME type: Content-Type: application/json or application/xml Request type: GET



GET https://api30.semantria.com/configurations.json


HTTP/1.0 200 Request accepted and served. [ { “conf_id” : “b1df666c-df5e-44c3-86f6-d1f331024f19”, “name” : “A test configuration” } ]

The POST HTTP method is used for updating and adding records onto the server.


MIME type: Content-Type: application/json or application/xml Request type: POST



List of the configurations to be added/updated on the server. required


POST https://api30.semantria.com/configurations.json [ { “config_id” : “b1df666c-df5e-44c3-86f6-d1f331024f19”, “name” : “A test configuration” } ]







The DELETE HTTP method is used for removing and adding of records onto the server.


MIME type: Content-Type: application/json or application/xml Request type: DELETE



List of the configurations to be removed on the server. required


DELETE https://api30.semantria.com/configurations.json [ “cd2e7341-a3c2-4fb4-9d3a-779e8b0a5eff”, “b1df666c-df5e-44c3-86f6-d1f331024f19” ]



Note: All changes listed above are implemented in Semantria SDKs and listed for explanation purposes.

On an SDK level, all methods are available through the session object, which is gateway to Semantria API.

Another important change introduced in API 3.0 is the change to the “sentiment” endpoint, which is

intended for sentiment adjustment. It has been renamed to “phrase”.

Sentiment adjustment end-point in API 2.1: https://api21.semantria.com/sentiment.json

Sentiment adjustment end-point in API 3.0: https://api30.semantria.com/phrases.json

Default values for newly created configurations have been changed from API 2.1 to API 3.0. In short,

custom configurations created in API 2.1 will be different from those created in API 3.0; they will have

different settings out of the box. Configurations are still completely adjustable and customizable.


https://api21.semantria.com/sentiment.json

https://api30.semantria.com/phrases.json

Semantria settings changes

Renamed field

New field

Semantria API 2.1 Semantria API 3.0 Description

configuration configuration config_id config_id Unique configuration identifier. name name Configuration name. is_primary is_primary Identifies whether the current configuration is primary or not. auto_responding auto_response Defines whether or not the service should respond with processed results automatically. language language Defines target language that will be used for task processing. chars_threshold chars_threshold Defines the threshold for alphanumeric characters in the text in percent. callback callback Defines a callback URL for automatic data responding. document document Includes options related to document processing mode. phrases_limit phrases_limit Limits the number of responded sentiment-bearing phrases for document. possible_phrases_limit Limits the number of possible sentiment-bearing phrases to be responded. doc_themes_limit themes_limit Limits the number of document themes responded from the service. named_entities_limit named_entities_limit Limits the number of named entities responded from the service. user_entities_limit user_entities_limit Limits the number of user entities responded from the service. entity_themes_limit entity_themes_limit Limits the number of entity themes responded from the service. named_relations_limit Limits the number of relations between named entities responded from the service. user_relations_limit Limits the number of relations between user entities responded from the service. concept_topics_limit concept_topics_limit Limits the number of concept topics responded from the service. query_topics_limit query_topics_limit Limits the number of query topics responded from the service. summary_limit summary_limit Limits the number of sentences for the document summary feature. detect_language Switches on language detection feature. pos_types Defines parts-of-speech which will be responded by the server. collection collection Includes options related to collection processing mode. facets_limit facets_limit Limits the number of facets which will be responded from the server. facet_atts_limit facet_atts_limit Limits the number of facets which will be responded from the server. facet_mentions_limit Limits the number of mentions of facet responded from the server. themes_limit themes_limit Limits the number of document themes responded from the service. named_entities_limit named_entities_limit Limits the number of named entities responded from the service. user_entities_limit user_entities_limit Limits the number of user entities responded from the service. concept_topics_limit concept_topics_limit Limits the number of concept topics responded from the service. query_topics_limit query_topics_limit Limits the number of query topics responded from the service.

Output data changes

Semantria’s API 3.0 output data has been enhanced from that of API 2.1. Output structure has most

remained the same, though new fields and output data types have been added. The new data output

types debuting in API 3.0 are Entity Relations and POS Tagging.

Entity Relations are logical relationships between entities that have been extracted by the engine.

Relationship detection are available with named entities and user entities exclusively, meaning that

relations can either only be found within named entities, or within user entities at one time; relations

between named and user entities are not available. Relations are extracted into a separate group and not

directly connected to extracted entity. Matching entities with relations must be done manually. Relations

cannot be adjusted by the users, for now; however, Semantria has a robust collection of known relations.

POS Tagging is a low-level analysis tool that dissects texts for parts of speech and logical sentences. This is

useful for low-level text analysis and extracting specific parts of a speech from a source text. Semantria is

capable of marking the source text with POSE markers and respond to all or a defined list of tags selected

by the user.

The available POS types distinguishable by Semantria are as follows:

All – Return whole text marked with POS tags.

Noun - Common, plural, proper, possessive, personal nouns, pronouns.

Verb - Modal, base, future, past verbs.

Adjective - Comparative, superlative and regular adjectives, adverbs.

Determiner - Determiners, predetermines.

Misc - Data/time, foreign words, coordinating conjunction, etc.

Twitter - URLs, emails mentions, hash tags, smiles, etc.

Note: Semantria also marks every extracted part of speech with more specific markers, like NN, NNP, VP,

etc. A detailed list of parts of speech markers can be found on Lexalytics’ web site.

http://dev.lexalytics.com/wiki/pmwiki.php?n=Main.POSTags

http://dev.lexalytics.com/wiki/pmwiki.php?n=Main.POSTags

Semantria output changes for document processing mode


document document summary summary A summary of the document text. sentiment_score sentiment_score A sentiment analysis of the document text. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”. language Detected language of source text. phrases phrases Returns sentiment-bearing phrases of the document. title title The text of the sentiment-bearing phrase. type Type of phrase; can be either “possible” or “detected” value. sentiment_score sentiment_score The sentiment score associated with this phrase. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”. is_negated is_negated Specifies whether the phrase has been negated or not. negating_phrase negating_phrase If the phrase has been negated, this gives the negating phrase. themes themes Returns themes of the document. title title The text of the document theme. is_about is_about An indicator specifying whether theme is the main theme of the document. evidence evidence A measure of the content on which the sentiment score for the theme is based. strength_score strength_score A measure of the strength of the theme within the document. sentiment_score sentiment_score The sentiment score for the document theme. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”. entities entities Returns the named entities and user defined entities from the text. entity_type entity_type Normalized entity title based on existing entity normalization rules. is_about is_about Type of the entity; can be either “named” or “user”. evidence evidence Specifies whether the document is about this entity. confident confident A measure of how much evidence the sentiment score was based on. title title An indicator specifying whether or not the confidence queries matched for this entity. type type Type of the entity (Company, Person, Place, Product, etc.). sentiment_score sentiment_score Sentiment related to the entity. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”. themes themes Returns list of themes related to this entity. title title The text of the entity theme. is_about is_about An indicator specifying if the theme is contained within the summary of the document. evidence evidence A measure of the content on which the sentiment score for the theme is based. strength_score strength_score A measure of the strength of the theme within the document. sentiment_score sentiment_score The sentiment score for the entity theme. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”.



relations Returns relations which represent a connection between one or more Entity objects. type A label describing the nature of the relationship. relation_type Type of relation according to extracted entities. Can be either named or user value. confidence_score A measure of confidence in the relationship extraction. extra Extra information that has been extracted about the relationship. entities Returns entities which presents parent relationship. title Normalized entity title based on existing entity normalization rules. entity_type Type of the entity (Company, Person, Place, Product, etc.). details Returns sentences from the original document with POS tags within. is_impoerative Represents whether sentence (group of words) is imperative or not. is_polar Represents whether or not sentence content contains sentiment polarity. words Returns list of words grouped by the parent sentence. tag POS marker of the word. title The original word representation in text. stemmed The stemmed form of the word. sentiment_score The sentiment score associated with the word. topics topics Returns the concept and query defined topics determined for the text. title title The topic title, which is its label in the text. type type Type of the topic; can be either “concept” or “query”. hitcount hitcount Number of query terms that are hit within the document based on query topic. strength_score strength_score Strength of the concept topic matches with the document content. sentiment_score sentiment_score The sentiment score for document content associated with the query topic. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”.



Semantria output changes for collection processing mode


collection collection facets facets Returns the facets extracted across all documents in the collection. label label The text of the facet. count count The count of occurrences of the facet in the text. negative_count negative_count The count of negative occurrences of the facet across the documents. neutral_count neutral_count The count of neutral occurrences of the facet across the documents. positive_count positive_count The count of positive occurrences of the facet across the documents. attributes attributes Returns the attributes associated with this facet. label label The text of the attribute. count count The count of occurrences of the attribute across the documents. mentions Returns genuine forms of facet mentioned across the documents. label The text of the mention. is_negated Indicates whether or not the mention has been negated. negating_phrase If the mention has been negated, this gives the negating phrase. indexes List of document indexes within collection where the mention appeared. themes themes Returns themes across the documents. title title The text of the theme across the documents. phrases_count phrases_count Count of sentiment-bearing phrases was used in calculating theme's sentiment. themes_count themes_count Count of themes across the documents that got rolled into this one. sentiment_score sentiment_score Sentiment score for theme’s sentences across the documents. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”. enities entities Returns the named entities and user defined entities from the text. title title Normalized form of the entity. It is the normalized entity title. type type Type of the entity; can be either “named” or “user”. entity_type entity_type Type of the entity (Company, Person, Place, Product, etc.). count count The count of occurrences of the entity across the documents. negative_count negative_count The count of negative occurrences of the facet across the documents. neutral_count neutral_count The count of neutral occurrences of the facet across the documents. positive_count positive_count The count of positive occurrences of the facet across the documents. topics topics Returns the concept and query defined topics determined for the text. title title The topic title, which is its label in the text. type type Type of the topic; can be either “concept” or “query”. hitcount hitcount Number of query terms that are hit within the document based on query topic. sentiment_score sentiment_score The sentiment score for documents content associated with the query topic. sentiment_polarity Verbal representation of sentiment score. Can be “negative”, “positive” or “neutral”.

SDK changes

Wrappers (SDKs) are flexible gateways to connect to Semantria’s API. They implement all available

functionality and share usage best practices of the API. Many SDKs written in a variety of programming

languages are available. All of them share common usage and integration ideas; in other words, all of

them are founded on the same set of principles. New wrappers intended for API 3.0 have not changed in

regards to data retrieval and processing approaches, however, they do reflect all the changes made in

regards to Configuration data end-points and new output details.

The most critical change relate to the removal of proxy objects. In wrappers for API 2.1, proxies were the

main objects through which users performed any updates. Users needed to fill proxy objects with all

changed objects and then push it to Semantria. In API 3.0, proxy objects have been replaced with sessions

becoming exposed dedicated methods for adding, removing and changing configuration entities.

API 2.1 SDK sample on Python

# Creates JSON serializer instance serializer = semantria.JsonSerializer() # Initializes new session with the serializer object and the keys. session = semantria.Session(consumerKey, consumerSecret, serializer) # Creates a sample Configuration for adding conf = { “name” : “Sample”, “auto_responding” : True } # Creates and fills the proxy object for Configurations update proxy = session.createUpdateProxy() proxy[“added”].append(conf) proxy[“removed”].append(“e882d177-2561-40be-a786-d86c243d0199”)

# Updates Configurations on Semantria service side session.updateConfigurations(proxy)

API 3.0 SDK sample on Python

# Creates JSON serializer instance serializer = semantria.JsonSerializer() # Initializes new session with the serializer object and the keys. session = semantria.Session(consumerKey, consumerSecret, serializer) # Creates a sample Configuration for adding conf = { “name” : “Sample”, “auto_response” : True } # Adds Configurations on Semantria service side session.addConfigurations([conf]) # Remove Configurations on Semantria side. session.removeConfigurations([“e882d177-2561-40be-a786-d86c243d0199”])

Note: The above approach is applicable to all configuration entities. All of them expose four different

methods, GET, ADD, UPDATE and REMOVE operations. The Blacklist configuration entity is the only entity



which does not follow this convention. It exposes three methods, instead of four. The UPDATE operation

was unnecessary.

Available Session methods

Session Information methods getStatus() Returns Semantria service status. getSubscription() Returns Semantria subscriber details. getStatistics() Returns subscriber related statistics.

Configuration methods getConfigurations() Returns list of available server-side configurations. addConfigurations() Adds list of provided configurations on server. updateConfigurations() Updates list of provided configurations on server. removeConfigurations() Remove list of provided configurations on server. getBlacklist() Returns all blacklisted items available on server. addBlacklist() Adds list of provided blacklisted items on server. removeBlacklist() Remove list of provided blacklisted items on server. getCategories() Returns list of available server-side categories. addCategories() Adds list of provided categories on server. updateCategories() Updates list of provided categories on server. removeCategories() Remove list of provided categories on server. getQueries() Returns list of available server-side queries. addQueries() Adds list of provided queries on server. updateQueries() Updates list of provided queries on server. removeQueries() Remove list of provided queries on server. getEntities() Returns list of available server-side entities. addEntities() Adds list of provided entities on server. updateEntities() Updates list of provided entities on server. removeEntities() Remove list of provided entities on server. getPhrases() Returns list of available server-side sentiment-bearing phrases. addPhrases() Adds list of provided sentiment-bearing phrases on server. updatePhrases() Updates list of provided sentiment-bearing phrases on server. removePhrases() Remove list of provided sentiment-bearing phrases on server.

Data related methods queueDocument() Queues document on server for processing. queueBatch() Queue batch of documents on server for processing. queueCollection() Queue collection of documents on server for processing. getDocument() Requests single document or its status from server by its ID. getCollection() Requests single collection or its status from server by its ID. cancelDocumnet() Cancels single document on server by its ID. cancelCollection() Cancels single document on server by its ID. getProcessedDocuments() Requests batch of processed documents from server. getProcessedCollections() Requests batch of processed collections from server.



All new SDKs have been completely reworked for API 3.0, talking into about the specific platforms on

which they will be running. However, most API methods have remained the same, easing the transition to

API 3.0.

Java, .Net and C++ wrappers still support both XML and JSON serialization, whereas wrappers for scripting

languages, such as Python, Ruby, PHP and JavaScript, have lost XML support in new release. After

investigating usage data, we noticed that none of our customers were using XML serialization in scripting

languages, so we decided to drop XML serialization support.

For customers who required XML serialization support in scripting languages, they may easily change their

existing SDK, which is distributed as source code, and perform XML requests if needed. The SDKs offer

“onRequest” and “onResponse” methods, which are intended for debugging purposes and allow users to

obtain raw response and request before sending and receiving from the API. This way can be used to

acquire raw XML functionality on an SDK level, if needed.

Compression

Semantria API 3.0 introduces HTTP compression support on a protocol level. This allows for a significant

decrease in data to be passed through the network. HTTP compression is only available for response data;

request data will always remain uncompressed. Response data can sometimes be up to ten times larger

than request data, making compression a valuable feature.

Compression will require additional CPU (processor) resources from the clients to decompress response

data. To accommodate as many users as possible, compression is set of “off” by default in all of

Semantria’s SDKs. To switch on compression, users only need to adjust one Session object:

Session.useCompression = true

Semantria API 3.0€¦ · API will respond with the verbal representation of the sentiment....

Documents

Transcript of Semantria API 3.0€¦ · API will respond with the verbal representation of the sentiment....