Apply API Governance to RESTful Service APIs using WSO2 Governance Registry and WSO2 API Manager
API Reference · 3.1 Overview of RESTful APIs HUAWEI CLOUD APIs comply with RESTful API design...
Transcript of API Reference · 3.1 Overview of RESTful APIs HUAWEI CLOUD APIs comply with RESTful API design...
Cloud Data Migration
API Reference
Issue 10
Date 2018-08-03
HUAWEI TECHNOLOGIES CO., LTD.
Copyright © Huawei Technologies Co., Ltd. 2018. All rights reserved.No part of this document may be reproduced or transmitted in any form or by any means without prior writtenconsent of Huawei Technologies Co., Ltd. Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei and thecustomer. All or part of the products, services and features described in this document may not be within thepurchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,and recommendations in this document are provided "AS IS" without warranties, guarantees orrepresentations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.
Huawei Technologies Co., Ltd.Address: Huawei Industrial Base
Bantian, LonggangShenzhen 518129People's Republic of China
Website: http://e.huawei.com
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
i
Contents
1 Before You Start............................................................................................................................. 1
2 API Overview................................................................................................................................. 2
3 API Use............................................................................................................................................ 43.1 Overview of RESTful APIs............................................................................................................................................43.2 Obtaining Request Authentication Information............................................................................................................. 83.2.1 Token Authentication...................................................................................................................................................83.3 Obtaining a Project ID....................................................................................................................................................93.4 API Use Case................................................................................................................................................................10
4 API Description........................................................................................................................... 174.1 Cluster Management.....................................................................................................................................................174.1.1 Creating a Cluster...................................................................................................................................................... 174.1.2 Querying the Cluster List.......................................................................................................................................... 214.1.3 Querying Cluster Details........................................................................................................................................... 234.1.4 Restarting a Cluster................................................................................................................................................... 284.1.5 Deleting a Cluster...................................................................................................................................................... 304.1.6 Stopping a Cluster..................................................................................................................................................... 314.1.7 Starting a Cluster....................................................................................................................................................... 324.2 Link Management.........................................................................................................................................................334.2.1 Creating a Link.......................................................................................................................................................... 344.2.2 Link Parameter Description.......................................................................................................................................394.2.2.1 Link to a Relational Database.................................................................................................................................394.2.2.2 Link to OBS............................................................................................................................................................424.2.2.3 Link to OSS on Alibaba Cloud...............................................................................................................................434.2.2.4 Link to Qiniu Cloud Object Storage (KODO)........................................................................................................444.2.2.5 Link to HDFS......................................................................................................................................................... 454.2.2.6 Link to HBase.........................................................................................................................................................484.2.2.7 Link to CloudTable................................................................................................................................................. 504.2.2.8 Link to Hive............................................................................................................................................................514.2.2.9 Link to an FTP or SFTP Server.............................................................................................................................. 524.2.2.10 Link to MongoDB/DDS....................................................................................................................................... 534.2.2.11 Link to Redis/DCS................................................................................................................................................544.2.2.12 Link to NAS..........................................................................................................................................................55
Cloud Data MigrationAPI Reference Contents
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
ii
4.2.2.13 Link to Kafka........................................................................................................................................................564.2.2.14 Link to DIS........................................................................................................................................................... 574.2.2.15 Link to Elasticsearch............................................................................................................................................ 584.2.2.16 Link to DLI...........................................................................................................................................................594.2.3 Querying a Link.........................................................................................................................................................604.2.4 Modifying a Link.......................................................................................................................................................634.2.5 Deleting a Link.......................................................................................................................................................... 664.3 Job Management...........................................................................................................................................................674.3.1 Creating a Job............................................................................................................................................................ 674.3.2 Source Job Parameters...............................................................................................................................................714.3.2.1 From a Relational Database....................................................................................................................................714.3.2.2 From OBS/OSS...................................................................................................................................................... 734.3.2.3 From HDFS............................................................................................................................................................ 784.3.2.4 From Hive...............................................................................................................................................................814.3.2.5 From HBase/CloudTable........................................................................................................................................ 824.3.2.6 From FTP/SFTP/NAS............................................................................................................................................ 834.3.2.7 From HTTP/HTTPS............................................................................................................................................... 884.3.2.8 From MongoDB/DDS............................................................................................................................................ 884.3.2.9 From Redis/DCS.....................................................................................................................................................894.3.2.10 From DIS.............................................................................................................................................................. 914.3.2.11 From Kafka........................................................................................................................................................... 924.3.2.12 From Elasticsearch/Cloud Search Service............................................................................................................934.3.3 Destination Job Parameters....................................................................................................................................... 944.3.3.1 To a Relational Database........................................................................................................................................ 944.3.3.2 To OBS................................................................................................................................................................... 964.3.3.3 To HDFS...............................................................................................................................................................1014.3.3.4 To Hive................................................................................................................................................................. 1024.3.3.5 To HBase/CloudTable...........................................................................................................................................1034.3.3.6 To FTP/SFTP/NAS...............................................................................................................................................1054.3.3.7 To DDS................................................................................................................................................................. 1074.3.3.8 To DCS................................................................................................................................................................. 1084.3.3.9 To Elasticsearch/Cloud Search Service................................................................................................................ 1104.3.3.10 To DLI.................................................................................................................................................................1114.3.4 Job Parameter Description....................................................................................................................................... 1124.3.5 Querying a Job......................................................................................................................................................... 1164.3.6 Modifying a Job.......................................................................................................................................................1204.3.7 Starting a Job........................................................................................................................................................... 1224.3.8 Stopping a Job......................................................................................................................................................... 1244.3.9 Querying Job Status.................................................................................................................................................1264.3.10 Querying Job Execution History........................................................................................................................... 1304.3.11 Deleting a Job........................................................................................................................................................ 131
5 Common Parameters................................................................................................................. 133
Cloud Data MigrationAPI Reference Contents
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
iii
5.1 Error Code.................................................................................................................................................................. 1335.2 Status Code................................................................................................................................................................. 141
A Change History......................................................................................................................... 145
Cloud Data MigrationAPI Reference Contents
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
iv
1 Before You Start
This document provides detailed information about application programming interfaces(APIs) of Cloud Data Migration (CDM), including API description, syntax, parameters, andexamples. You can switch to your desired content according to Table 1-1.
Table 1-1 Document overview
If You Want to Know Link
API constituents and list API Overview
Representational State Transfer (REST)message bodies, calling methods, andexamples
API Use
API description API Description
Common parameters Common Parameters
Cloud Data MigrationAPI Reference 1 Before You Start
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
1
2 API Overview
CDM provides Huawei-developed APIs. You can perform the following operations withCDM APIs.
Type API Description
ClusterManagementAPIs
Creating a Cluster Creates a cluster.
Querying the Cluster List Queries and displays the cluster list.
Querying Cluster Details Queries cluster details.
Restarting a Cluster Restarts a cluster.
Deleting a Cluster Deletes a cluster.
Stopping a Cluster Stops a specified cluster.
Starting a Cluster Starts a specified cluster.
LinkManagementAPIs
Creating a Link Connects to a data source.
Querying a Link Queries the link list.
Modifying a Link Modifies the link configuration.
Deleting a Link Deletes a link.
Job ManagementAPIs
Creating a Job Creates a table/file migration job.
Querying a Job Queries and displays the job list.
Modifying a Job Modifies the job configuration.
Starting a Job Starts a data migration job.
Stopping a Job Stops a running job.
Querying Job Status Queries and displays the runningstatus of a job.
Querying Job ExecutionHistory
Queries and displays the historicalstatus of job execution.
Cloud Data MigrationAPI Reference 2 API Overview
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
2
Type API Description
Deleting a Job Deletes a job.
Cloud Data MigrationAPI Reference 2 API Overview
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
3
3 API Use
3.1 Overview of RESTful APIsHUAWEI CLOUD APIs comply with RESTful API design specifications. RepresentationalState Transfer (REST) allocates Uniform Resource Identifiers (URIs) to dispersed resourcesso that the resources can be located. Applications on clients use unified resource locators(URLs) to obtain the resources.
The URL is in the following format: https://Endpoint/uri. uri indicates the resource path.That is, the API access path.
HUAWEI CLOUD APIs use HTTP as the transfer protocol. Requests/Responses aretransferred by using JSON packets, with media type represented by application/json.
A RESTful API request or response consists of the following five parts:
l Request URIl Request Headerl Request Bodyl Response Headerl Response Body
Sending a Request
You can send a request based on the constructed request body using any of the following threemethods:
l cURLcURL is a command-line tool used to perform URL operations and transfer information.It can serve as an HTTP client to send HTTP requests to the server end and receiveresponse messages. cURL is suitable for use in API tuning scenarios. For moreinformation about cURL, visit https://curl.haxx.se/.
l CodeYou can call APIs through code to assemble, send, and process requests.
l REST clients
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
4
Both Mozilla and Google provide a graphical browser plug-in, that is, REST client, tosend and process requests.
– For Mozilla Firefox, see Firefox REST Client.
– For Google Chrome, see Postman Interceptor.
Request URI
Although the request URI is included in the request header, most languages or frameworksrequire that it be transferred separately from the request message. Therefore, the request URIis listed independently.
A request URI consists of the following:
{URI-scheme} :// {Endpoint} / {resource-path} ? {query-string}
Table 3-1 Parameters in a URI
Parameter Description
URI-scheme Protocol used to transfer requests. CDM APIs use https.
Endpoint Domain name or IP address of the server bearing the REST serviceendpoint. Obtain this value from Regions and Endpoints.
resource-path API access path for performing a specified operation. Obtain thevalue from the URI of the API, for example, v3/auth/tokens.
Query string (Optional) For example, API version or resource selection criteria.
Request Header
A request header consists of the following two parts:
l HTTP methods, which are also called operations or actions, specify the type ofoperations that you are requesting. Table 3-2 describes methods supported by CDMRESTful APIs.
Table 3-2 HTTP methods
Method Description
GET Requests the server to return a specific resource.
PUT Requests the server to update a specific resource.
POST Requests the server to add resources or perform special operations.
DELETE Requests the server to delete a specific resource, for example, todelete an object.
HEAD Allows the client to query the server for the headers of a givenresource without actually downloading the resource itself.
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
5
Method Description
PATCH Requests the server to apply partial modifications to a resource.If the resource does not exist, a resource can be created using thePATCH method.
l (Optional) Additional fields in the request header required by a specific URI and HTTPmethod.
For details about common request headers in token authentication, see Table 3-3. Fordetails about the request authentication information, see Obtaining RequestAuthentication Information.
Table 3-3 Common request headers in token authentication
Parameter Description Mandatory Example Value
x-sdk-date Time when therequest is sent. Thetime is inYYYYMMDD'T'HHMMSS'Z' format.The value is thecurrent GreenwichMean Time (GMT)time of the system.
No 20150907T101459Z
Authorization AuthenticationinformationThe value can beobtained from therequest signingresult.
No SDK-HMAC-SHA256Credential=ZIRRKMTWPTQFQI1WKNKB/20150907//ec2/sdk_request,SignedHeaders=content-type;host;x-sdk-date,Signature=55741b610f3c9fa3ae40b5a8021ebf7ebc2a28a603fc62d25cb3bfe6608e1994
Host Information aboutthe requested server.The value can beobtained from theURL of the serviceAPI inhostname[:port]format. If the portnumber is notspecified, the defaultport is used. Thedefault port numberfor https is 443.
No code.test.comorcode.test.com:443
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
6
Parameter Description Mandatory Example Value
Content-type Request body MIMEtype
Yes application/json
Content-Length
Length of therequest body. Theunit is byte.
This parameteris mandatoryfor POST andPUT requestsbut must beleft blank forGET requests.
3495
X-Project-Id Project ID. Thisparameter is used toobtain the token forthe project.This parameter ismandatory for arequest from a DeCor multi-project user.
No e9993fc787d94b6c886cbaa340f9c0f4
X-Auth-Token User token Yes -
X-Language Request language Yes zh_cn
NOTE
For details about other parameters in the header, see the HTTP protocol documentation.
(Optional) Request BodyA request body is generally sent in a structured format (for example, JSON or XML),corresponding to Content-type in the request header, and is used to transfer content otherthan the request header.
If the request body contains Chinese characters, convert the Chinese characters into theUTF-8 encoding format.
Response HeaderA response header consists of the following two parts:
l An HTTP status code, from 2xx success code to 4xx or 5xx error code, or the status codedefined by the service. See Status Code.
l Additional fields in the response header required by a specified response, for example,the Content-type response header. For details about common response headers, seeTable 3-4.
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
7
Table 3-4 Common response headers
Parameter Description Example Value
Date Standard HTTP header, whichspecifies the date and time at whichthe message was sent. The format isdefined by RFC 822.
Mon, 12 Nov 200715:55:01 GMT
Server Standard HTTP header, whichincludes the software informationthat the server uses to process therequest.
Apache
Content-Length Standard HTTP header, whichspecifies the size of the entity body,in decimal number of bytes, sent tothe recipient.
-
Content-Type Standard HTTP header, whichspecifies the media type of theentity body sent to the recipient.
application/json
(Optional) Response Body
The response body is generally returned in a structured format (for example, JSON or XML),corresponding to Content-type in the response header, and is used to transfer content otherthan the response header.
3.2 Obtaining Request Authentication InformationCDM supports token authentication in API calling. You can use tokens for authentication.
3.2.1 Token Authentication
Scenarios
To use token authentication, obtain the user token, and add X-Auth-Token to the requestheader of the service API when making an API call.
This section describes how to make an API call using token authentication.
Procedure1. Send POST https://Endpoint of IAM/v3/auth/tokens to obtain the endpoint of IAM and
the region name in the message body.
See Regions and Endpoints.
If the service region name is ALL, replace Endpoint of IAM with the endpoint of CNNorth-Beijing1.
The following code provides a sample request.
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
8
NOTE
Replace the items in italic in the following example with actual ones. For details, see the Identityand Access Management API Reference.
{ "auth": { "identity": { "methods": [ "password" ], "password": { "user": { "name": "Username", "password": "Password", "domain": { "name": "Domain name" } } } }, "scope": { "project": { "id": "0215ef11e49d4743be23dd97a1561e91" //This ID is used as an example. } } }}
2. Obtain the token by performing the operations in section "Obtaining a User Token" inthe Identity and Access Management API Reference. The token value is the X-Subject-Token value in the response header.
3. Make a call to a service API, add X-Auth-Token to the message header, and set thevalue of X-Auth-Token to the token obtained in 2.
3.3 Obtaining a Project IDA project ID is required for some URLs when an API is called. It can be project_id ortenant_id because project_id has the same meaning as tenant_id in this document. Beforecalling an API, you need to obtain a project ID on the console. The steps are as follows:
1. Log in to the management console.2. Hover the cursor on the username and select My Credential from the drop-down list.
On the page that is displayed, view the project ID in the project list.
Figure 3-1 Viewing the project ID
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
9
3.4 API Use CaseThis section describes how to use CDM APIs with an example of creating a table/filemigration job by making API calls.
The process of creating a CDM job to migrate data from the local MySQL database to DataWarehouse Service (DWS) on the cloud is as follows:
1. Call the API in Token Authentication to obtain the user token, which will be put intothe request header for authentication in a subsequent request.
2. Create a cluster and obtain the cluster ID.
NOTE
If you have created a cluster, skip the steps for creating a cluster and directly use the createdcluster ID.
a. Obtain the project ID.b. Call the API in Creating a Cluster to create a cluster.c. Call the API in Querying Cluster Details to obtain the cluster ID.
3. Call the API in Creating a Link to create the MySQL link and DWS link.4. Call the API in Creating a Job to create a job of migrating data from MySQL to DWS.5. Call the API in Starting a Job to execute the job.
Prerequisitesl You have created a VPC, subnet, and elastic IP address (EIP), and obtained the IDs of
the VPC, subnet, and security group. For details, see the Virtual Private Cloud APIReference.
l You have obtained the endpoints of IAM and CDM. For details, see Regions andEndpoints.
l You can access the local MySQL database using a public IP address, and have obtainedthe IP address, port number, database name, username, and password of the MySQLdatabase. The account has the read, write, and delete permissions on the MySQLdatabase.
l You have purchased a DWS instance and obtained the IP address, port number, databasename, username, and password for accessing DWS. The account has the read, write, anddelete permissions on the database installed on DWS.
Creating a JobThe following values are examples (replace them based on the actual situation).
l IAM endpoint: iam_endpointl CDM endpoint: cdm_endpointl Project ID: 1551c7f6c808414d8e9f3c514a170f2el VPC ID (the same as that of DWS): 6b47302a-bf79-4b20-bf7a-80987408e196l Subnet ID (the same as that of DWS): 63bdc3cb-a4e7-486f-82ee-d9bf208c8f8cl Security group ID (the same as that of DWS): 005af77a-cce5-45ac-99c7-2ea50ea8addf
Perform the following operations to create a job.
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
10
1. Before calling other APIs, obtain the token and set it as an environment variable.curl -H "Content-Type:application/json" https://{iam_endpoint}/v3/auth/tokens -X POST -d '{ "auth": { "identity": { "methods": [ "password" ], "password": { "user": { "name": "testname", "domain": { "name": "testname" }, "password": "Password" } } }, "scope": { "project": { "name": "cn-north-1" } } }}' -v -k
The value of X-Subject-Token in the response header is the token.X-Subject-Token:MIIDkgYJKoZIhvcNAQcCoIIDgzCCA38CAQExDTALBglghkgBZQMEAgEwgXXXXX...
Run the following command to set the token as an environment variable for future use:export X-Auth-Token = MIIDkgYJKoZIhvcNAQcCoIIDgzCCA38CAQExDTALBglghkgBZQMEAgEwgXXXXX...
2. Call the API in Creating a Cluster to create a cluster. The following values areexamples:
– Cluster name: cdm-ab82
– Cluster flavor: cdm.medium
– The VPC, subnet, and security group are the same as those of DWS, and the EIP isautomatically bound.
curl -X POST -H 'Content-Type:application/json;charset=utf-8' -H "X-Auth-Token:$Token" -d '{ "cluster": { "name": "cdm-ab82", "vpcId": "6b47302a-bf79-4b20-bf7a-80987408e196", "instances": [{ "specCode": "cdm.medium", "nics": [{ "net-id": "63bdc3cb-a4e7-486f-82ee-d9bf208c8f8c", "securityGroupId": "005af77a-cce5-45ac-99c7-2ea50ea8addf" }], "availability_zone": "cn-north-1b", "type": "cdm", "publicIP": { "publicBindType": "auto_assign" } }] }}' https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters -v -k
If status code 200 is returned, the cluster is successfully created.
3. Call the API in Querying the Cluster List to query cluster information, obtain thecluster ID, and set the cluster ID to a global variable.
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
11
curl -X GET -H 'Content-Type:application/json;charset=utf-8' -H "X-Auth-Token:$Token" https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters -k –vThe response is as follows:{ "clusters": [{ "version": "1.0.0", "updated": "2017-09-05T08:38:25", "name": "cdm-ab82", "created": "2017-09-05T08:38:25", "id": "bae65496-643e-47ca-84af-948672de7eeb", "status": "200", "isFrozen": "0", "statusDetail": "Normal", "actionProgress": { }, "config_status": "In-Sync" }]}If the value of status is 200, the cluster is successfully created. The cluster ID isbae65496-643e-47ca-84af-948672de7eeb.Run the following command to set the cluster ID to a global variable for future use:export ID = bae65496-643e-47ca-84af-948672de7eeb
4. Call the API in Creating a Link to create the MySQL link mysql_link. The followingvalues are examples:– Public IP address: 1xx.120.85.24– Port number: 3306– Database name: DB_name– Login username: username– Password: DB_passwordcurl -X POST -H "Content-Type:application/json" -H "X-Auth-Token:$Token" -d '{ "links": [{ "enabled": true, "update-user": null, "name": "mysql_link", "link-config-values": { "configs": [ { "name": "linkConfig", "inputs": [ { "name": "linkConfig.databaseType", "value": "MYSQL" }, { "name": "linkConfig.host", "value": "1xx.120.85.24" }, { "name": "linkConfig.port", "value": "3306" }, { "name": "linkConfig.database", "value": "DB_name" }, { "name": "linkConfig.username", "value": "username" }, { "name": "linkConfig.password",
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
12
"value": "DB_password" }, { "name": "linkConfig.fetchSize", "value": "100000" }, { "name": "linkConfig.usingNative", "value": "true" } ] } ] }, "connector-name": "generic-jdbc-connector", "creation-date": 1496654788622, "update-date": 1496654788622, "creation-user": null }]}' https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/cdm/link -k -vIf status code 200 is returned, the link is successfully created.
5. Call the API in Creating a Link to create the DWS link dws_link. The following valuesare examples:– IP address of the database: 10.120.85.24– Port number: 3306– Database name: DWS– Login username: user_dws– Password: dws_passwordcurl -X POST -H "Content-Type:application/json" -H "X-Auth-Token:$Token" -d '{ "links": [{ "enabled": true, "update-user": null, "name": "dws_link", "link-config-values": { "configs": [ { "name": "linkConfig", "inputs": [ { "name": "linkConfig.databaseType", "value": "DWS" }, { "name": "linkConfig.host", "value": "10.120.85.24" }, { "name": "linkConfig.port", "value": "3306" }, { "name": "linkConfig.database", "value": "DWS" }, { "name": "linkConfig.username", "value": "user_dws" }, { "name": "linkConfig.password", "value": "dws_password" },
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
13
{ "name": "linkConfig.fetchSize", "value": "100000" }, { "name": "linkConfig.usingNative", "value": "true" } ] } ] }, "connector-name": "generic-jdbc-connector", "creation-date": 1496654789622, "update-date": 1496654789622, "creation-user": null }]}' https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/cdm/link -k -vIf status code 200 is returned, the link is successfully created.
6. After the two links are created, call the API in Creating a Job to create a migration job.The following is a sample job:– The job name is mysql2dws.– The name of the MySQL database from which data is exported is default, and the
name of the exported table is mysql_tbl. The job is split into multiple tasks by idand the tasks are executed concurrently.
– The name of the database on DWS to which the data is imported is public, and thetable name is cdm_all_type. Do not clear the data in the table before import.
– If no table in the local MySQL database exists in the database on DWS, CDMautomatically creates the table on DWS.
– The field list loaded to DWS is id&gid&name.– When the job extracts data, a number of three Extractor tasks are concurrently
executed.curl -X POST -H "Content-Type:application/json" -H "X-Cluster-ID:$ID" -H "X-Auth-Token:$Token" -d '{ "jobs": [{ "job_type": "NORMAL_JOB", "name": "mysql2dws", "from-link-name": "mysql_link", "from-connector-name": "generic-jdbc-connector", "to-link-name": "dws_link", "to-connector-name": "generic-jdbc-connector", "from-config-values": { "configs": [{ "name": "fromJobConfig", "inputs": [{ "name": "fromJobConfig.schemaName", "value": "default" }, { "name": "fromJobConfig.tableName", "value": "mysql_tbl" }, { "name": "fromJobConfig.partitionColumn", "value": "id" }] }] },"to-config-values": { "configs": [
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
14
{ "inputs": [ { "name": "toJobConfig.schemaName", "value": "public" }, { "name": "toJobConfig.tablePreparation", "value": "CREATE_WHEN_NOT_EXIST" }, { "name": "toJobConfig.tableName", "value": "cdm_all_type" }, { "name": "toJobConfig.columnList", "value": "id&gid&name" }, { "name": "toJobConfig.shouldClearTable", "value": "false" } ], "name": "toJobConfig" } ] }, "driver-config-values": { "configs": [{ "name": "throttlingConfig", "inputs": [{ "name": "throttlingConfig.numExtractors", "value": "3" }] }] } }]}' https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/cdm/job -k -vIf status code 200 is returned, the job is successfully created.
7. Call the API in Starting a Job to execute the job.curl -X GET -H 'Content-Type:application/json;charset=utf-8' -H "X-Cluster-ID:$ID" -H "X-Auth-Token:$Token" https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/cdm/job/mysql2dws/start -k -vThe response is as follows:{ "submissions": [{ "progress": 1, "job-name": "mysql2dws", "status": "BOOTING", "creation-date": 1496905778725, "creation-user": "cdm" }]}
8. Call the API in Querying Job Status to query the job status.curl -X GET -H 'Content-Type:application/json;charset=utf-8' -H "X-Cluster-ID:$ID" -H "X-Auth-Token:$Token" https://{cdm_endpoint}/cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/mysql2dws/status -k -vIf the job status is successfully queried, the response is as follows:{ "submissions": [{ "progress": 0, "job-name": "mysql2dws", "status": "SUCCEEDED",
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
15
"creation-date": 1496906582997, "creation-user": "cdm", "isStopingIncrement": "", "last-update-date": 1496906580005, "is-execute-auto": false, "last-udpate-user": "cdm", "isDeleteJob": false, "isIncrementing": false, "external-id": "job_local1127970451_0009", "counters": { "org.apache.sqoop.submission.counter.SqoopCounters": { "BYTES_WRITTEN": -1, "TOTAL_FILES": -1, "BYTES_READ": -1, "FILES_WRITTEN": -1, "TOTAL_SIZE": -1, "FILES_READ": -1, "ROWS_WRITTEN": 80, "ROWS_READ": 80 } } }]}
NOTE
l BYTES_WRITTEN: number of written bytes
l BYTES_READ: number of read bytes
l TOTAL_FILES: total number of files
l FILES_WRITTEN: number of written files
l FILES_READ: number of read files
l ROWS_WRITTEN: number of rows that are successfully written
l ROWS_READ: number of rows that are successfully read
Cloud Data MigrationAPI Reference 3 API Use
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
16
4 API Description
4.1 Cluster Management
4.1.1 Creating a Cluster
FunctionThis API is used to create a CDM cluster. Creating a CDM cluster is an asynchronousoperation. You can query the creation progress and result by periodically calling the API inQuerying Cluster Details.
URIl URI format
POST /cdm/v1.0/{project_id}/clustersl Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID. For detailsabout how to obtain theproject ID, see Obtaininga Project ID.
Requestl Sample request
POST /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters{ "cluster": { "name": "cdm-ab82", "vpcId": "6b47302a-bf79-4b20-bf7a-80987408e196", "instances": [{ "specCode": "cdm.medium", "nics": [{ "net-id": "63bdc3cb-a4e7-486f-82ee-d9bf208c8f8c", "securityGroupId": "005af77a-cce5-45ac-99c7-2ea50ea8addf"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
17
}], "availability_zone": "cn-north-1b", "type": "cdm", "publicIP": { "publicBindType": "auto_assign" } }] }}
l Parameter description
Parameter Mandatory Type Description
cluster Yes List Cluster object. Fordetails, see Descriptionof the clusterparameter.
l Description of the cluster parameter
Parameter Mandatory Type Description
name Yes String Cluster name
vpcId Yes String VPC ID, which is usedfor configuring the clusternetwork
instances Yes List Node list. For details, seeDescription of theinstances parameter.
l Description of the instances parameter
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
18
Parameter Mandatory Type Description
specCode Yes String Instance flavor. The optionsare as follows:l cdm.small: 2 vCPUs with
4 GB memory, applicableto Proof of Concept(PoC) verification anddevelopment tests
l cdm.medium: 4 vCPUswith 8 GB memory,applicable to migration ofa single database tablewith fewer than 10million pieces of data
l cdm.large: 8 vCPUs with16 GB memory,applicable to migration ofa single database tablewith 10 million pieces ofdata or more
l cdm.xlarge: 16 vCPUswith 32 GB memory,applicable to TB-leveldata migration requiring10GE high-speedbandwidth
nics Yes List NIC list. A maximum of twoNICs are supported. Fordetails, see Description ofthe nics parameter.
availability_zone Yes String Currently, CDM supports thefollowing AZs:l CN North-Beijng1:
– cn-north-1a– cn-north-1b
l CN East-Shanghai2:– cn-east-2a– cn-east-2b– cn-east-2c
l CN South-Guangzhou:– cn-south-1a– cn-south-2b– cn-south-1c
type Yes String Node type. Currently, onlycdm is available.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
19
Parameter Mandatory Type Description
publicIP No Dictionary EIP. This parameter is notrequired when no EIP isbound. For details, seeDescription of the publicIPparameter.
l Description of the nics parameter
Parameter Mandatory Type Description
net-id Yes String Subnet ID
securityGroupId Yes String Security group ID
l Description of the publicIP parameter
Parameter Mandatory Type Description
publicBindType Yes String Currently, this parameter hasonly one value.auto_assign indicates that theelastic IP address isdynamically allocated.
Responsel Sample response
{ "id": "7d85f602-a948-4a30-afd4-e84f47471c15", "name": "cdm-ab82"}
l Parameter description
Parameter Mandatory Type Description
id Yes String Cluster ID
name Yes Sring Cluster name
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
20
Return Value Description
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.1.2 Querying the Cluster List
Function
This API is used to query the CDM cluster list.
URIl URI format
GET /cdm/v1.0/{project_id}/clustersl Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
Request
Sample requestGET /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters
Responsel Sample response
{ "clusters": [{ "version": "1.0.0", "updated": "2017-09-05T08:38:25", "name": "cdm-c018", "created": "2017-09-05T08:38:25", "id": "bae65496-643e-47ca-84af-948672de7eeb", "status": "200", "isFrozen": "0", "statusDetail": "Normal", "actionProgress": { }, "config_status": "In-Sync" }]}
l Parameter description
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
21
Parameter Mandatory Type Description
clusters Yes List Cluster list. For details,see Description of theclusters parameter.
l Description of the clusters parameter
Parameter Mandatory Type Description
version Yes String Cluster version
updated Yes String Time when a cluster isupdated. The format isYYYY-MM-DDThh:mm:ssZ(ISO 8601).
name Yes String Cluster name
created Yes String Time when a cluster iscreated. The format isYYYY-MM-DDThh:mm:ssZ(ISO 8601).
id Yes String Cluster ID
status Yes String Cluster status. The optionsare as follows:l 100: Creatingl 200: Runningl 300: Failedl 303: Creation failedl 400: Deletedl 800: Frozenl 900: Stoppedl 910: Stoppingl 920: Starting
isFrozen Yes String Whether a cluster is frozen.The options are as follows:l 0: Nol 1: Yes
statusDetail No String Cluster status description
actionProgress No Dictionary Cluster operation progress.For details, see Descriptionof the actionProgressparameter.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
22
Parameter Mandatory Type Description
config_status No String Cluster configurationstatus. The options are asfollows:l In-Sync: Configuration
synchronizedl Applying: Being
configuredl Sync-Failure:
Configuration failed
l Description of the actionProgress parameter
Parameter Mandatory Type Description
CREATING No String Cluster creation progress. Forexample, 29%.
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.1.3 Querying Cluster Details
Function
This API is used to query CDM cluster details.
URIl URI format
GET /cdm/v1.0/{project_id}/clusters/{cluster_id}
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
23
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
RequestSample requestGET /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb
Responsel Sample response
{ "version": "1.0.0", "instances": [{ "flavor": { "id": "fb8fe666-6734-4b11-bc6c-43d11db3c745" }, "volume": { "type": "LOCAL_DISK", "size": 100 }, "status": "200", "actions": ["REBOOTING"], "type": "cdm", "id": "635dce67-3df8-4756-b4c7-90e45e687367", "name": "cdm-c018", "isFrozen": "0", "config_status": "In-Sync" }], "updated": "2017-09-05T08:38:25", "name": "cdm-c018", "created": "2017-09-05T08:38:25", "id": "bae65496-643e-47ca-84af-948672de7eeb", "status": "200", "actions": ["REBOOTING"], "isFrozen": "0", "statusDetail": "Normal", "actionProgress": { }, "config_status": "In-Sync"}
l Parameter description
Parameter Mandatory Type Description
version Yes String Cluster version
instances Yes List Cluster node information.For details, see Descriptionof the instances parameter.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
24
Parameter Mandatory Type Description
updated Yes String Time when a cluster isupdated. The format isYYYY-MM-DDThh:mm:ssZ(ISO 8601).
name Yes String Cluster name
created Yes String Time when a cluster iscreated. The format is YYYY-MM-DDThh:mm:ssZ (ISO8601).
id Yes String Cluster ID
status Yes String Cluster status. The optionsare as follows:l 100: Creatingl 200: Runningl 300: Failedl 303: Creation failedl 400: Deletedl 800: Frozenl 900: Stoppedl 910: Stoppingl 920: Starting
actions No String Cluster operation status list.The options are as follows:l REBOOTING:
Restartingl RESTORING:
Restoringl REBOOT_FAILURE:
Restart failed
isFrozen Yes String Whether a cluster is frozen.The options are as follows:l 0: Nol 1: Yes
statusDetail No String Cluster status description
actionProgress No List Cluster operation progress.For details, see Descriptionof the actionProgressparameter.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
25
Parameter Mandatory Type Description
config_status No String Cluster configuration status.The options are as follows:l In-Sync: Configuration
synchronizedl Applying: Being
configuredl Sync-Failure:
Configuration failed
l Description of the instances parameter
Parameter Mandatory Type Description
flavor Yes Dictionary VM flavor of a node. Fordetails, see Description ofthe flavor parameter.
volume Yes Dictionary Disk information of a node.For details, see Descriptionof the volume parameter.
status Yes String Node status. The options areas follows:l 100: Creatingl 200: Normall 300: Failedl 303: Creation failedl 400: Deletedl 800: Frozen
actions No String list Node operation status list.The options are as follows:l REBOOTING: Restartingl RESTORING: Restoringl REBOOT_FAILURE:
Restart failed
type Yes String Node type. Currently, onlycdm is available.
id Yes String VM ID
name Yes Strng VM name
isFrozen Yes String Whether the node is frozen.The options are as follows:l 0: Nol 1: Yes
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
26
Parameter Mandatory Type Description
config_status No String Node configuration status.The options are as follows:l In-Sync: Configuration
synchronizedl Applying: Being
configuredl Sync-Failure:
Configuration failed
l Description of the flavor parameter
Parameter Mandatory Type Description
id Yes String VM flavor ID
l Description of the volume parameter
Parameter Mandatory Type Description
type Yes String Type of disks on the node. Onlylocal disks are supported.
size Yes Integer Size of the disk on the node(GB)
l Description of the actionProgress parameter
Parameter Mandatory Type Description
CREATING No String Cluster creation progress. Forexample, 29%.
Return Valuel Normal
200
l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
27
Return Value Description
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.1.4 Restarting a Cluster
FunctionThis API is used to restart a CDM cluster. Restarting a CDM cluster is an asynchronousoperation. You can query the restart result by periodically calling the API in QueryingCluster Details.
URIl URI format
POST /cdm/v1.0/{project_id}/clusters/{cluster_id}/actionl Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
Requestl Sample request
POST /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/action{ "restart": { "type": "cdm" }}
l Parameter description
Parameter Mandatory Type Description
restart Yes Dictionary Cluster restart. For detailsabout how to define clusterrestart, see Description ofthe restart parameter.
l Description of the restart parameter
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
28
Parameter Mandatory Type Description
type Yes String Type of the cluster to berestarted. The parameter can beset to cdm only.
restartMode No Enumeration Restart mode. The options are asfollows:l IMMEDIATELY:
Immediate restartl GRACEFULL: Graceful
restartl FORCELY: Forcible restartThe default value isIMMEDIATELY.
restartLevel No Enumeration Restart level. The options are asfollows:l SERVICE: Service restartl VM: VM restartThe default value is SERVICE.
restartDelayTime
No Integer Restart delay, in seconds
Responsel Sample response
{ "jobId": "ff8080815e59d92d015e5b27ccb0004d"}
l Parameter description
Parameter Mandatory Type Description
jobId No String Job ID
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
29
Return Value Description
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.1.5 Deleting a Cluster
Function
This API is used to delete a specified CDM cluster. Deleting a CDM cluster is anasynchronous operation. You can query the deletion result by periodically calling the API inQuerying Cluster Details.
URIl URI format
DELETE /cdm/v1.0/{project_id}/clusters/{cluster_id}l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
Request
Sample requestDELETE /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920
Responsel Sample response
{ "jobId": "ff8080815e55125a015e552eddba001a"}
l Parameter description
Parameter Mandatory Type Description
jobId No String Job ID
Return Valuel Normal
200
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
30
l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.1.6 Stopping a Cluster
Function
This API is used to stop a specified CDM cluster. Stopping a CDM cluster is an asynchronousoperation. You can query the stop result by periodically calling the API in Querying ClusterDetails. A stopped cluster cannot execute migrations jobs or access the Job Managementpage. You are charged lower fees for a stopped cluster than a running cluster. When a CDMcluster is no longer used, you can call this API to save costs.
URIl URI format
POST /cdm/v1.0/{project_id}/clusters/{cluster_id}/actionl Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
Requestl Sample request
POST /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/action{ "stop": { }}
l Parameter description
Parameter Mandatory Type Description
stop Yes Dictionary Stopping a cluster
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
31
Responsel Sample response
{ "jobId": "ff8080815e59d92d015e5b27ccb0004d"}
l Parameter description
Parameter Mandatory Type Description
jobId No String Job ID
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.1.7 Starting a Cluster
Function
This API is used to start a specified CDM cluster. Starting a CDM cluster is an asynchronousoperation. You can query the start result by periodically calling the API in Querying ClusterDetails. When a cluster is stopped, you can call this API to start the cluster.
URIl URI format
POST /cdm/v1.0/{project_id}/clusters/{cluster_id}/actionl Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
32
Requestl Sample request
POST /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/bae65496-643e-47ca-84af-948672de7eeb/action{ "start": { }}
l Parameter description
Parameter Mandatory Type Description
start Yes Dictionary Starting a cluster
Responsel Sample response
{ "jobId": "ff8080815e59d92d015e5b27ccb0004d"}
l Parameter description
Parameter Mandatory Type Description
jobId No String Job ID
Return Valuel Normal
200
l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.2 Link Management
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
33
4.2.1 Creating a Link
Function
When you create links for connectors, required parameters of links vary with connectors. Forexample, to create a link for the JDBC connector, you must enter the IP address, port number,username, and password of the database server.
URIl URI format
POST /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/link?validate={validate}
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
Requestl Sample request
POST /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/link
{ "links": [{ "enabled": true, "update-user": null, "name": "mysql_link", "link-config-values": { "configs": [ { "name": "linkConfig", "inputs": [ { "name": "linkConfig.databaseType", "value": "MYSQL" }, { "name": "linkConfig.host", "value": "10.120.85.24" }, { "name": "linkConfig.port", "value": "3306" }, { "name": "linkConfig.database", "value": "DB_name" }, { "name": "linkConfig.username", "value": "username" }, { "name": "linkConfig.password", "value": "DB_password" },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
34
{ "name": "linkConfig.fetchSize", "value": "100000" }, { "name": "linkConfig.usingNative", "value": "true" } ] } ] }, "connector-name": "generic-jdbc-connector", "creation-date": 1496654788622, "update-date": 1496654788622, "creation-user": null }]}
l Parameter description
Parameter Mandatory Type Description
links Yes List Link list. For details, seeDescription of the linksparameter.
l Description of the links parameter
Parameter Mandatory Type Description
enabled No Boolean Whether to activate a link.The default value is true.
update-user No String User who updated a link
name Yes String Link name
link-config-values Yes Dictionary Link configuration. Fordetails, see Description ofthe link-config-valuesparameter.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
35
Parameter Mandatory Type Description
connector-name Yes String Connector name. Theoptions are as follows:l generic-jdbc-connector:
relational databaseconnector
l obs-connector:HUAWEI CLOUD OBSconnector or AlibabaCloud OSS connector
l hdfs-connector: HDFSconnector
l hive-connector: Hiveconnector
l hbase-connector: HBaseconnector
l ftp-connector: FTPconnector
l sftp-connector: SFTPconnector
l mangodb-connector:MangoDB connector
l redis-connector: Redisconnector
l nas-connector: NASconnector
l elasticsearch-connector:Elasticsearch connector
l kafka-connector: Kafkaconnector
l dis-connector: DataIngestion Service (DIS)connector
l dli-connector: Data LakeInsight (DLI) connector
l http-connector: HTTP/HTTPS connector
l thirdparty-obs-connector: third-partyobject storage connector
creation-date No Timestamp Time when a link is created
update-date No Timestamp Time when a link is updated
creation-user No String User who created a link
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
36
Parameter Mandatory Type Description
validate No Boolean When the parameter is set totrue, the API only validateswhether the parameters arecorrectly configured, but notcreate any link.
l Description of the link-config-values parameter
Parameter Mandatory Type Description
configs Yes List Data structure of linkconfiguration items. Fordetails, see Description of theconfigs parameter.
l Description of the configs parameter
Parameter Mandatory Type Description
name Yes String Configuration item name
inputs Yes List Input parameter list. Eachelement in the list is inname,value format. Fordetails, see Description ofthe inputs parameter.For details about theparameter names, see LinkParameter Description.
l Description of the inputs parameter
Parameter Mandatory Type Description
name Yes String Parameter name
value Yes String Parameter value
Responsel Successful sample response
{ "name": "rdb_link", "validation-result": [{ }]}
l Failed sample response{ "validation-result": [{ "linkConfig": [{
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
37
"message": "Can't connect to the database with given credentials: The authentication type 12 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it is using an authentication scheme supported by the driver.", "status": "ERROR" }] }]}
l Parameter description
Parameter Mandatory Type Description
name Yes String Link name
validation-result Yes List Validation structurel If a link fails to be
created, the failurecause is returned. Fordetails, see Descriptionof the validation-resultparameter.
l If a link is successfullycreated, a blank list isreturned.
l Description of the validation-result parameter
Parameter Mandatory Type Description
linkConfig No List Validation result of linkcreation or update. Fordetails, see Description ofthe linkConfigparameter.
l Description of the linkConfig parameter
Parameter Mandatory Type Description
message No String Error message
status No Enumeration l ERRORl WARNING
Return Valuel Normal
200l Abnormal
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
38
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.2.2 Link Parameter Description
4.2.2.1 Link to a Relational Database
Description
By creating a JDBC link, you can extract data from or load data to the following relationaldatabases:l Data Warehouse Servicel RDS (MySQL)l RDS (PostgreSQL)l RDS (SQL Server)l DDMl MySQLl PostgreSQLl Microsoft SQL Serverl Oraclel IBM Db2l FusionInsight LibrAl Derecho (GaussDB)
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.databaseType", "value": "MYSQL" }, { "name": "linkConfig.host",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
39
"value": "10.120.205.30" }, { "name": "linkConfig.port", "value": "3306" }, { "name": "linkConfig.database", "value": "DB_name" }, { "name": "linkConfig.username", "value": "username" }, { "name": "linkConfig.password", "value": "Add password here" }, { "name": "linkConfig.fetchSize", "value": "100000" }, { "name": "linkConfig.usingNative", "value": "false" }, { "name": "linkConfig.useSSL", "value": "false" } ], "name": "linkConfig" } ] }, "name": "mysql_link", "connector-name": "generic-jdbc-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.databaseType
Yes Enumeration Database type. The options areas follows:l ORACLEl MYSQLl SQLSERVERl DB2l POSTGRESQLl GAUSSDBl DWS
linkConfig.host No String IP address of the database server
linkConfig.port No String Port number of the databaseserver
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
40
Parameter Mandatory Type Description
linkConfig.database
No String Database name
linkConfig.username
Yes String Username
linkConfig.password
Yes String Password
linkConfig.fetchSize
No String Number of data rows obtainedeach time
linkConfig.usingNative
No Boolean Whether to use the local APIacceleration function of thedatabaseWhen creating a MySQL link,you can use the LOAD DATAfunction of MySQL to acceleratedata import and improve theperformance of importing data tothe MySQL database.
linkConfig.useSSL
No Boolean Whether to use encryptedtransmission. You can enableSSL encrypted transmission forRelational Database Service(RDS).
linkConfig.jdbcProperties
No Map Link attribute, which specifiesthe JDBC connector attributes ofthe data source. For details abouthow to configure the linkattributes, see the JDBCconnector description of thecorresponding database.
dialect.identifierEnclose
No String Reference identifier, which is thedelimiter between the referencedtable names or column names.For details, see the productdocumentation of thecorresponding database.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
41
Parameter Mandatory Type Description
linkConfig.importMode
No Enumeration Data import mode. Thisparameter is available only whenthe database type is DWS.l GDS: In GDS mode, port
25000 is enabled on CDM,and multiple DWSDataNodes extract data fromCDM.
l COPY: In COPY mode,CDM copies data to DWS byusing the JDBC API ofDWS.
4.2.2.2 Link to OBS
DescriptionBy creating an OBS link, you can extract files from or load files to HUAWEI CLOUD OBS.Files in CSV, JSON, and binary formats are supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.storageType", "value": "OBS" }, { "name": "linkConfig.server", "value": "10.121.16.183" }, { "name": "linkConfig.port", "value": "5443" }, { "name":"linkConfig.accessKey", "value": "RSO6TTEZMJ6TTFBBAACE" }, { "name":"linkConfig.securityKey", "value":"Add password here" } ], "name": "linkConfig" } ] }, "name": "obs_link", "connector-name": "obs-connector" }
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
42
]}
Link ParametersParameter Mandatory Type Description
linkConfig.storageType
Yes String Storage class of an object
linkConfig.server Yes String IP address of the OBS server
linkConfig.port Yes String Port number
linkConfig.accessKey
Yes String AK
linkConfig.securityKey
Yes String SK
4.2.2.3 Link to OSS on Alibaba Cloud
DescriptionBy creating an OSS link, you can extract files from Object Storage Service (OSS) on AlibabaCloud. Files in CSV, JSON, and binary formats are supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.storageType", "value": "OSS" }, { "name": "linkConfig.ossEndpoint", "value": "oss-cn-qingdao.aliyuncs.com" }, { "name": "linkConfig.ossAuthType", "value": "ACCESS_KEY" }, { "name": "linkConfig.accessKey", "value": "LTAI1mR805IE0cVr" }, { "name": "linkConfig.securityKey", "value": "Add password here" } ], "name": "linkConfig" } ] },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
43
"name": "op_oss_link", "connector-name": "obs-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.storageType
Yes String Storage class of an object
linkConfig.ossEndpoint
Yes String Endpoint of OSS
linkConfig.ossAuthType
Yes Boolean Identity authentication methodl ACCESS_KEY: Use the
access key to access OSS.l STS: Use the temporary key
and security token to accessOSS.
linkConfig.accessKey
Yes String AK
linkConfig.securityKey
Yes String SK
linkConfig.securityToken
No String Temporary security tokenrequired when a temporaryaccess key is used to log in toOSS
4.2.2.4 Link to Qiniu Cloud Object Storage (KODO)
DescriptionBy creating a KODO link, you can extract files from Qiniu Cloud Object Storage (KODO).Files in CSV, JSON, and binary formats are supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.storageType", "value": "KODO" }, { "name": "linkConfig.zone", "value": "z1"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
44
}, { "name": "linkConfig.accessKey", "value": "xib1R_oeN4KBYTA-yNoV0Kd3AbUwi1yuvzbPF4LE" }, { "name": "linkConfig.securityKey", "value": "Add password here" }, { "name": "linkConfig.useCustomDomain", "value": "false" } ], "name": "linkConfig" } ] }, "name": "kodo_link", "connector-name": "thirdparty-obs-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.storageType
Yes String Storage class of an object
linkConfig.zone Yes String Storage area of KODO
linkConfig.accessKey
Yes String AK
linkConfig.securityKey
Yes String SK
linkConfig.useCustomDomain
No Boolean Whether to use the user-defineddomain name to downloadobjects
4.2.2.5 Link to HDFS
DescriptionBy creating an HDFS link, you can extract files from or load files to MRS, FusionInsight HD,or Apache Hadoop. Files in CSV, Parquet, and binary formats are supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
45
"name": "linkConfig.hadoopType", "value": "FusionInsight HD" }, { "name": "linkConfig.host", "value": "10.120.205.143" }, { "name": "linkConfig.casPort", "value": "20009" }, { "name": "linkConfig.port", "value": "28443" }, { "name": "linkConfig.authType", "value": "KERBEROS" }, { "name": "linkConfig.user", "value": "admin" }, { "name": "linkConfig.password", "value": "Add password here" }, { "name": "linkConfig.runMode", "value": "STANDALONE" } ], "name": "linkConfig" } ] }, "name": "hdfslink", "connector-name": "hdfs-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.hadoopType
Yes Enumeration Hadoop type. The options are asfollows:l MRS: link to HDFS of MRSl FusionInsight HD: link to
HDFS of FusionInsight HDl Apache Hadoop: link to HDFS
of Apache Hadoop
linkConfig.uri No String NameNode URI required for thelink to Apache Hadoop. The formatis ip:port.
linkConfig.host No String IP address of Manager required forthe link to MRS or FusionInsightHD
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
46
Parameter Mandatory Type Description
linkConfig.port No String Port number of Manager requiredfor the link to FusionInsight HD
linkConfig.casPort
No String Port number of CAS Server thatconnects to FusionInsight HDrequired for the link toFusionInsight HD
linkConfig.user No String Username for logging in toManager. If SIMPLEauthentication is used forconnecting to MRS, you do notneed to enter the username andpassword for logging in toManager.
linkConfig.password
No String Password for logging in toManager. If SIMPLEauthentication is used forconnecting to MRS, you do notneed to enter the username andpassword for logging in toManager.
linkConfig.authType
No Enumeration Authentication method. Theoptions are as follows:l Simple: for non-security model Kerberos: for security mode
linkConfig.principal
No String Account principal required forKerberos authentication. You cancontact the administrator to obtainthe account.
linkConfig.keytab
No FileContent Local absolute path of the keytabfile required for Kerberosauthentication. You can contact theadministrator to obtain the file.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
47
Parameter Mandatory Type Description
linkConfig.runMode
No Enumeration Running mode of the HDFS link.Options are as follows:l Embedded: The link instance
runs with CDM. This mode hasbetter performance.
l Standalone: The link instanceruns in an independent process.If CDM needs to connect tomultiple Hadoop data sources(MRS, Hadoop, or CloudTable)with both Kerberos and Simpleauthentication methods,Standalone prevails.
If Standalone is selected, CDMcan migrate data between HDFSsof multiple MRS clusters.
4.2.2.6 Link to HBase
DescriptionBy creating an HBase link, you can extract data from or load data to HBase of MRS,FusionInsight HD, and Apache Hadoop.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.hbaseType", "value": "MRS" }, { "name": "linkConfig.host", "value": "192.168.2.23" }, { "name": "linkConfig.authType", "value": "SIMPLE" }, { "name": "linkConfig.runMode", "value": "STANDALONE" } ], "name": "linkConfig" } ] }, "name": "mrshbase",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
48
"connector-name": "hbase-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.hbaseType
Yes Enumeration HBase type. The options are asfollows:l CloudTable: link to CloudTable
Service (CloudTable)l MRS: link to HBase of MRSl FusionInsight HD: link to HBase
of FusionInsight HDl Apache Hadoop: link to HBase
of Apache Hadoop
linkConfig.uri No String NameNode URI required for the linkto Apache Hadoop. The format isip:port.
llinkConfig.host
No String IP address of Manager required forthe link to MRS or FusionInsight HD
linkConfig.port No String Port number of Manager required forthe link to FusionInsight HD
linkConfig.casPort
No String Port number of CAS Server thatconnects to FusionInsight HDrequired for the link to FusionInsightHD
linkConfig.user No String Username for logging in to Manager.If SIMPLE authentication is used forconnecting to MRS, you do not needto enter the username and passwordfor logging in to Manager.
linkConfig.password
No String Password for logging in to Manager.If SIMPLE authentication is used forconnecting to MRS, you do not needto enter the username and passwordfor logging in to Manager.
linkConfig.authType
No Enumeration Authentication method. The optionsare as follows:l Simple: for non-security model Kerberos: for security mode
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
49
Parameter Mandatory Type Description
linkConfig.principal
No String Account principal required forKerberos authentication. You cancontact the administrator to obtain theaccount.
linkConfig.keytab
No FileContent Local absolute path of the keytab filerequired for Kerberos authentication.You can contact the administrator toobtain the file.
linkConfig.serviceType
No String Service type
linkConfig.runMode
No Enumeration Running mode of the HBase link.Options are as follows:l Embedded: The link instance
runs with CDM. This mode hasbetter performance.
l Standalone: The link instanceruns in an independent process. IfCDM needs to connect to multipleHadoop data sources (MRS,Hadoop, or CloudTable) with bothKerberos and Simpleauthentication methods,Standalone prevails.
4.2.2.7 Link to CloudTable
DescriptionBy creating a CloudTable link, you can extract data from or load data to CloudTable.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.hbaseType", "value": "CloudTable" }, { "name": "linkConfig.zookeeperQuorum", "value": "cloudtable-cdm-zk1.cloudtable.com:2181,cloudtable-cdm-zk2.cloudtable.com:2181" } ], "name": "linkConfig" }
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
50
] }, "name": "hbase_link", "connector-name": "hbase-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.hbaseType
Yes Enumeration HBase type. The options are asfollows:l CloudTable: link to CloudTablel MRS: link to MRSl FusionInsight HD: link to
FusionInsight HDl Apache Hadoop: link to Apache
Hadoop
linkConfig.zookeeperQuorum
Yes String ZooKeeper link of CloudTable. Thisparameter is mandatory for theCloudTable link.
4.2.2.8 Link to Hive
DescriptionBy creating a Hive link, you can extract data from or load data to Hive of MRS.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.host", "value": "10.120.205.230" }, { "name": "linkConfig.authType", "value": "KERBEROS" }, { "name": "linkConfig.user", "value": "cdm" }, { "name": "linkConfig.password", "value": "Add password here" } ], "name": "linkConfig"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
51
} ] }, "name": "hive_link", "connector-name": "hive-connector" } ]}
Link ParametersParameter Mandatory Type Description
llinkConfig.host
No String IP address of MRS Manager
linkConfig.authType
No Enumeration Authentication method of MRS. Theoptions are as follows:l SIMPLE: for non-security model KERBEROS: for security mode
linkConfig.principal
No String Account principal required forKerberos authentication. You cancontact the administrator to obtain theaccount.
linkConfig.keytab
No FileContent Local absolute path of the keytab filerequired for Kerberos authentication.You can contact the administrator toobtain the file.
4.2.2.9 Link to an FTP or SFTP Server
DescriptionBy creating an FTP or SFTP link, you are able to extract files from or load files to the FTP orSFTP server. Files in CSV, JSON, and binary format are supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.server", "value": "10.120.85.167" }, { "name": "linkConfig.port", "value": "22" }, { "name": "linkConfig.username", "value": "username"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
52
}, { "name": "linkConfig.password", "value": "Add password here" } ], "name": "linkConfig" } ] }, "name": "sftp_link", "connector-name": "sftp-connector" } ]}
Link ParametersParameters for creating the FTP or SFTP link are the same.
Parameter Mandatory Type Description
linkConfig.server
Yes String IP address of the FTP or SFTP server
linkConfig.port Yes String Port number of the FTP or SFTPserver
linkConfig.username
Yes String Username for logging in to the FTPor SFTP server
linkConfig.password
Yes String Password for logging in to the FTP orSFTP server
4.2.2.10 Link to MongoDB/DDS
DescriptionBy creating a MongoDB link, you can extract data from or load data to the MongoDB serveror Document Database Service (DDS).
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.serverList", "value": "10.120.84.149:27017" }, { "name": "linkConfig.database", "value": "DB_name" }, { "name": "linkConfig.userName",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
53
"value": "username" }, { "name": "linkConfig.password", "value": "Add password here" } ], "name": "linkConfig" } ] }, "name": "mongo_link", "connector-name": "mongodb-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.serverList
Yes String Server IP address list inhost1:port1;host2:port2 format
linkConfig.database
Yes String Name of the MongoDB database
linkConfig.userName
Yes String Username for logging in to theMongoDB server
linkConfig.password
Yes String Password for logging in to theMongoDB server
4.2.2.11 Link to Redis/DCS
DescriptionBy creating a Redis link, you can extract data from or load data to the Redis server. Bycreating a DCS link, you can load data to Data Cache Service (DCS), but not extract data forDCS. The data can be stored in string or hash format.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.deploymentMode", "value": "Cluster" }, { "name": "linkConfig.serverlist", "value": "10.120.84.149:7300" }, { "name": "linkConfig.password",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
54
"value": "Add password here" }, { "name": "linkConfig.dbIndex", "value": "0" } ], "name": "linkConfig" } ] }, "name": "redis_link", "connector-name": "redis-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.deploymentMode
Yes Enumeration Redis deployment mode. Theoptions are as follows:l Single: single-node deploymentl Cluster: cluster deployment
linkConfig.serverlist
Yes String Server IP address list inhost1:port1;host2:port2 format
linkConfig.password
Yes String Password for logging in to the Redisserver
linkConfig.dbIndex
Yes String Redis database index
4.2.2.12 Link to NAS
DescriptionBy creating a NAS link, you are able to extract files from or load files to the NAS server.Currently, the SMB and CIFS protocols are supported. Files in CSV, JSON, and binaryformats are supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.protocol", "value": "CIFS" }, { "name": "linkConfig.server", "value": "\\\\10.121.16.20\\data"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
55
}, { "name": "linkConfig.username", "value": "username" }, { "name": "linkConfig.password", "value": "Add password here" } ], "name": "linkConfig" } ] }, "name": "nas_link", "connector-name": "nas-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.protocol Yes String NAS file protocol. Currently,only the SMB and CIFSprotocols are supported.
linkConfig.server Yes String Shared path of the NASserver, in which \\\\ isconverted into \\
linkConfig.username Yes String Username for logging in tothe NAS server
linkConfig.password Yes String Password for logging in to theNAS server
4.2.2.13 Link to Kafka
DescriptionBy creating a Kafka link, you are able to access open source Kafka and migrate data fromKafka to other data sources as required. Currently, only data export from Kafka is supported.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.brokerList", "value": "127.0.0.1:9092" } ], "name": "linkConfig"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
56
} ] }, "name": "kafka_link", "connector-name": "kafka-connector" } ]}
Link ParametersParameter Mandatory Type Description
linkConfig.brokerList
Yes String Kafka broker list inhost1:port1,host2:port2 format
4.2.2.14 Link to DIS
DescriptionBy creating a DIS link, you can access DIS on HUAWEI CLOUD and migrate data from DISto other data sources as required.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.region", "value": "cn-north-1" }, { "name": "linkConfig.endpoint", "value": "https://dis.cn-north-1.myhuaweiclouds.com:20004" }, { "name": "linkConfig.ak", "value": "RSO6TTEZMJ6TTFBBAACE" }, "name": "linkConfig.sk", "value": "Add password here" }, { "name": "linkConfig.projectId", "value": "11d4d5af17c84660bc90b6631327d7c7" } ], "name": "linkConfig" } ] }, "name": "dis_link", "connector-name": "dis-connector" } ]}
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
57
Link Parameters
Parameter Mandatory Type Description
linkConfig.region Yes String Region where DIS resides
linkConfig.endpoint Yes String Domain name or IP addressof the DIS server
linkConfig.ak Yes String AK of the DIS server
linkConfig.sk Yes String SK of the DIS server
linkConfig.projectId Yes String Project ID
4.2.2.15 Link to Elasticsearch
Description
By creating an Elasticsearch link, you can extract data from or load data to the Elasticsearchserver or Cloud Search Service.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.host", "value": "10.66.65.91" }, { "name": "linkConfig.port", "value": "9200" }, { "name": "linkConfig.user", "value": "sqoop" }, { "name": "linkConfig.password", "value": "Add password here" }, { "name": "linkConfig.linkType", "value": "ElasticSearch" } ], "name": "linkConfig" } ] }, "name": "es_link", "connector-name": "elasticsearch-connector" } ]}
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
58
Link ParametersParameter Mandatory Type Description
linkConfig.host Yes String Domain name or IP address of theElasticsearch server
linkConfig.port Yes String Port number of the Elasticsearchserver
linkConfig.user No String For Elasticsearch that supportsusername and passwordauthentication, configure theusername and password whencreating a link.
linkConfig.password
No String Password for logging in to theElasticsearch server
linkConfig.linkType
Yes String Link type, which is used todistinguish the Elasticsearch linkfrom the Cloud Search Service link
4.2.2.16 Link to DLI
Description
By creating a DLI link, you can import data to DLI. Currently, you cannot export data fromDLI using CDM.
Sample Link{ "links": [ { "link-config-values": { "configs": [ { "inputs": [ { "name": "linkConfig.ak", "value": "GRC2WR0IDC6NGROYLWU2" }, { "name": "linkConfig.sk", "value": "Add password here" }, { "name": "linkConfig.region", "value": "cn-north-1" }, { "name": "linkConfig.projectId", "value": "c48475ce8e174a7a9f775706a3d5ebe2" } ], "name": "linkConfig" } ]
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
59
}, "name": "dli", "connector-name": "dli-connector" } ]}
Link Parameters
Parameter Mandatory Type Description
linkConfig.ak Yes String AK for logging in to the DLIdatabase
linkConfig.sk Yes String SK for logging in to the DLIdatabase
linkConfig.region
Yes String Region where DLI resides
linkConfig.projectId
Yes String Project ID of the DLI service
4.2.3 Querying a Link
Function
This API is used to query the link list.
URIl URI format
GET /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/link/{link_name}l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
link_name Yes String Link name. When theparameter is set to all, all linksare queried.
Request
Sample requestGET /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/link/sftplink
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
60
Responsel Sample response
{ "links": [ { "link-config-values": { "configs": [ { "name": "linkConfig", "inputs": [ { "size": 128, "name": "linkConfig.server", "type": "STRING", "mandatory": true, "value": "10.120.85.163" }, { "defaultValue": "22", "name": "linkConfig.port", "type": "INTEGER", "mandatory": true, "value": "22" }, { "size": 32, "name": "linkConfig.username", "type": "STRING", "mandatory": true, "value": "root" }, { "size": 32, "name": "linkConfig.password", "sensitive": true, "type": "STRING", "mandatory": true } ] } ] }, "creation-user": "cdm", "name": "sftp_link", "creation-date": 1516674482640, "connector-name": "sftp-connector", "update-date": 1516674476022, "enabled": true, "update-user": "cdm" } ]}
l Parameter description
Parameter Mandatory Type Description
links Yes List Link list. For details,see Description ofthe links parameter.
l Description of the links parameter
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
61
Parameter Mandatory Type Description
link-config-values Yes Dictionary Link configuration.For details, seeDescription of thelink-config-valuesparameter.
creation-user Yes String User who created alink
name Yes String Link name
creation-date Yes Timestamp Time when a link iscreated
connector-name Yes String Connector name
update-date Yes Timestamp Time when a link isupdated
enabled No Boolean Whether to activate alink
update-user No String User who updated alink
l Description of the link-config-values parameter
Parameter Mandatory Type Description
configs Yes List Data structure of linkconfiguration items. Fordetails, see Description of theconfigs parameter.
l Description of the configs parameter
Parameter Mandatory Type Description
name Yes String Configuration item name
inputs Yes List Parameter list. For details, seeDescription of the inputsparameter.
l Description of the inputs parameter
Parameter Mandatory Type Description
size No String Maximum length of theparameter
name Yes String Parameter name
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
62
Parameter Mandatory Type Description
type No String Parameter type
defaultValue No String Default value of theparameter
dependent No Boolean Whether the parameter isaffected by other components
value No String Parameter value
mandatory No Boolean Whether the parameter ismandatory
editable No String Whether a parameter can beedited
sensitive No Boolean Whether the parameter is apassword
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.2.4 Modifying a Link
Function
This API is used to modify a link that you have created.
URIl URI format
PUT /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/link/{link_name}l Parameter description
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
63
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
link_name Yes String Link name
l Parameter description of a message body
The parameters for modifying a link are the same as those for creating a link. For details,see Request in Creating a Link.
Request
Sample request
PUT /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/link/rdb_link{ "links": [{ "enabled": true, "update-user": null, "name": "mysql_link", "link-config-values": { "configs": [ { "name": "linkConfig", "inputs": [ { "name": "linkConfig.databaseType", "value": "MYSQL" }, { "name": "linkConfig.host", "value": "10.120.85.24" }, { "name": "linkConfig.port", "value": "3306" }, { "name": "linkConfig.database", "value": "DB_name" }, { "name": "linkConfig.username", "value": "username" }, { "name": "linkConfig.password", "value": "Add password here" }, { "name": "linkConfig.fetchSize", "value": "100000" }, { "name": "linkConfig.usingNative", "value": "true" } ] } ] },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
64
"connector-name": "generic-jdbc-connector", "creation-date": 1496654788622, "update-date": 1496654788622, "creation-user": null }]}
Responsel Successful sample response
{ "name": "mysql_link", "validation-result": [{ }]}
l Failed sample response{ "validation-result": [{ "linkConfig": [{ "message": "Can't connect to the database with given credentials: The authentication type 12 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it is using an authentication scheme supported by the driver.", "status": "ERROR" }] }]}
l Parameter description
Parameter Mandatory Type Description
name Yes String Link name
validation-result Yes List Validation structurel If a link fails to be
modified, the failurecause is returned. Fordetails, see Descriptionof the validation-resultparameter.
l If a link is successfullymodified, a blank list isreturned.
l Description of the validation-result parameter
Parameter Mandatory Type Description
linkConfig No List Validation result of linkcreation or update. Fordetails, see Description ofthe linkConfigparameter.
l Description of the linkConfig parameter
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
65
Parameter Mandatory Type Description
message No String Error message
status No Enumeration l ERRORl WARNING
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.2.5 Deleting a Link
FunctionThis API is used to delete a specified link. However, you cannot delete the links used by jobs.
URIl URI format
DELETE /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/link/{link_name}
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
link_name Yes String Link name
RequestSample request
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
66
DELETE /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/link/jdbclink
Responsel Parameter description of a successful response
None
l If a failure occurs, the HTTP response code is 500. A sample response is as follows:{ "errCode": "Cdm.0021", "externalMessage": "Given link name is in use"}
l Parameter description of a failed response
Parameter Mandatory Type Description
errCode Yes String Error code
externalMessage Yes String Error message
Return Valuel Normal
200
l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3 Job Management
4.3.1 Creating a Job
Function
This API is used to create a job. You must specify the source and destination links and jobparameters in the message body.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
67
URIl URI format
POST /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/jobl Parameter description
Table 4-1 URI parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
Requestl Sample request
POST /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job
{ "jobs": [{ "job_type": "NORMAL_JOB", "name": "mysql2hive", "from-link-name": "mysqllink", "from-connector-name": "generic-jdbc-connector", "to-link-name": "hivelink", "to-connector-name": "hive-connector", "from-config-values": { "configs": [{ "name": "fromJobConfig", "inputs": [{ "name": "fromJobConfig.schemaName", "value": "default" }, { "name": "fromJobConfig.tableName", "value": "mysql_tbl" }, { "name": "fromJobConfig.partitionColumn", "value": "id" }] }] }, "to-config-values": { "configs": [{ "name": "toJobConfig", "inputs": [{ "name": "toJobConfig.hive", "value": "hive" }, { "name": "toJobConfig.database", "value": "default" }, { "name": "toJobConfig.table", "value": "test_tbl" }] }] }, "driver-config-values": { "configs": [{
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
68
"name": "throttlingConfig", "inputs": [{ "name": "throttlingConfig.numExtractors", "value": "1" }] }] } }]}
l Parameter description
Parameter Mandatory Type Description
jobs Yes List Job list. For details, seeDescription of the jobsparameter.
l Description of the jobs parameter
Parameter Mandatory Type Description
job_type Yes String Job typel NORMAL_JOB: table/
file migrationl BATCH_JOB: entire
DB migration
name Yes String Job name
from-link-name Yes String Source link name
from-connector-name Yes String Name of the connector ofthe source link
to-link-name Yes String Destination link name
to-connector-name Yes String Name of the connector ofthe destination link
from-config-values Yes Dictionary Source link configuration.For details, see Descriptionof from-config-values.
to-config-values Yes Dictionary Destination linkconfiguration. For details,see Description of to-config-values.
driver-config-values Yes Dictionary Job parameterconfiguration. For details,see Description of driver-config-values.
l Description of from-config-values, to-config-values, and driver-config-values
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
69
Parameter Mandatory Type Description
configs Yes List The data structures of sourcelink parameters, destinationlink parameters, and jobparameters are the same.However, the inputsparameter varies. For details,see Description of theconfigs parameter.
l Description of the configs parameter
Parameter Mandatory Type Description
name Yes String Configuration item name
inputs Yes List Input parameter list. Eachelement in the list is inname,value format. Fordetails, see Description ofthe inputs parameter.l In from-config-values,
different types of sourcelinks have different inputsparameter lists. Fordetails, see Source JobParameters.
l In to-config-values,different types ofdestination links havedifferent inputs parameterlists. For details, seeDestination JobParameters.
l For details about theinputs parameter indriver-config-values, seeJob ParameterDescription.
l Description of the inputs parameter
Parameter Mandatory Type Description
name Yes String Parameter name
value Yes String Parameter value
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
70
Responsel Sample response
{ "name": "mysql2hive"}
l Parameter description
Parameter Mandatory Type Description
name Yes String Job name
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3.2 Source Job ParametersCDM allows you to export data from multiple types of data sources. Each data source has itsown job parameters. You can view specific input parameters based on the correspondingsection of the data source type.
Set the input parameters of the source link as described in Description of the inputsparameter of from-config-values in Creating a Job.
4.3.2.1 From a Relational Database
Sample JSON File"from-config-values": { "configs": [{ "inputs": [{ "name": "fromJobConfig.schemaName", "value": "default" }, { "name": "fromJobConfig.tableName", "value": "cdm_tbl" },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
71
{ "name": "fromJobConfig.partitionColumn", "value": "id" }], "name": "fromJobConfig" }] }
Parameter DescriptionParameter Mandatory Type Description
fromJobConfig.schemaName
Yes String Database mode or tablespace. Forexample, public.
fromJobConfig.tableName
Yes String Table name. For example,TBL_EXAMPLE.
fromJobConfig.whereClause
No String WHERE clauses to be extracted. Ifno WHERE clause is specified, theentire table will be extracted. Forexample, age > 18 and age <= 60.
fromJobConfig.columnList
No String List of fields to be extracted. Use& to separate field names. Forexample, id&gid&name.
fromJobConfig.partitionColumn
No String Partition field to be extracted, bywhich a job is split in multiple sub-jobs executed concurrently. Forexample, id.
fromJobConfig.regainSymbol
No String After Regain Symbol is set to aspecified field, CDM queries thetable imported from the destinationdatabase every time the job isstarted. If the table does not containthe specified field, CDM performsfull migration. If the destinationtable contains the specified fieldand the field has a value, CDMperforms incremental migration tomigrate only the data whose valueis greater than the value of thisfield.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
72
Parameter Mandatory Type Description
fromJobConfig.incrMigration
No Boolean Whether to migrate incrementaldata in MySQL Binlog mode.Currently, this mode can be used inthe table/file migration from theMySQL database to the DWSdatabase only.After this function is enabled, datain the source table and destinationtable can be synchronized in realtime. One MySQL link supportsonly one incremental migration job,and one source table supports onlyone incremental migration job.
4.3.2.2 From OBS/OSS
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.bucketName", "value": "cdm" }, { "name": "fromJobConfig.inputDirectory", "value": "/data/" }, { "name": "fromJobConfig.inputFormat", "value": "JSON_FILE" }, { "name": "fromJobConfig.jsonReferenceNode", "value": "data.list" }, { "name": "fromJobConfig.fieldSeparator", "value": "," }, { "name": "fromJobConfig.encodeType", "value": "UTF-8" }, { "name": "fromJobConfig.splitType", "value": "FILE" }, { "name": "fromJobConfig.fromFileOpType", "value": "DELETE" }, { "name": "fromJobConfig.markerFile", "value": "ok.txt" }, {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
73
"name": "fromJobConfig.filterType", "value": "WILDCARD" }, { "name": "fromJobConfig.pathFilter", "value": "*input" }, { "name": "fromJobConfig.fileFilter", "value": "*.csv" } ], "name": "fromJobConfig" } ] }
Parameter Descriptionl OBS/OSS job parameter description
Parameter Mandatory Type Description
fromJobConfig.bucketName
Yes String OBS/OSS bucket name. Forexample, cdm.
fromJobConfig.inputDirectory
Yes String Data extraction path. A single filepath can contain a maximum of 50files, which are separated byvertical bars (|). For example,FROM/example.csv|FROM/b.txt.
fromJobConfig.inputFormat
Yes Enumeration File format required for datatransmission. Currently, thefollowing file formats aresupported:l CSV_FILE: CSV formatl JSON_FILE: JSON formatl BINARY_FILE: binary formatl CARBONDATA: CarbonData
formatWhen the parameter is set toBINARY_FILE, the connector ofthe destination link must be HDFS,OBS, SFTP, FTP, or NAS.
fromJobConfig.lineSeparator
No String Line feed character in the file. Forexample, \n. By default, \n, \r, or\r\n is automatically identified.
fromJobConfig.columnList
No String Numbers of columns to beextracted. Use & to separatecolumn numbers in ascendingorder. For example, 1&3&5.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
74
Parameter Mandatory Type Description
fromJobConfig.lineSeparator
No String Line feed character. Thisparameter is valid only when thefile format is CSV_FILE. Thedefault value is \r\n.
fromJobConfig.regexSeparator
No Boolean Whether to use the regularexpression to separate fields. Thisparameter is valid only when thefile format is CSV_FILE.
fromJobConfig.regex
No String Regular expression. Thisparameter is valid only when theregular expression is used toseparate fields.
fromJobConfig.fieldSeparator
No String Field delimiter. This parameter isvalid only when the file format isCSV_FILE. The default value is ,.
fromJobConfig.quoteChar
No Boolean Whether to use the encirclingsymbol. If this parameter is set totrue, the field delimiters in theencircling symbol are regarded asa part of the string value.Currently, the default encirclingsymbol of CDM is doublequotation mark (").
fromJobConfig.firstRowAsHeader
No Boolean Whether to regard the first line asthe heading line. This parameter isvalid only when the file format isCSV_FILE. When this parameteris set to true, CDM reads the firstline as the heading line whenextracting data, which means theheading line of the CSV file isremoved in data transmission.
fromJobConfig.fromCompression
No Enumeration Compression format. Thisparameter is valid only when thefile format is CSV_FILE orJSON. The options are as follows:l None: Files in all formats are
transferred.l gzip: Files in gzip format are
transferred.l Zip: Files in Zip format are
transferred.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
75
Parameter Mandatory Type Description
fromJobConfig.jsonReferenceNode
No String Reference node. This parameter isvalid when the file format isJSON_FILE. Resolve data on theJSON node. If the datacorresponding to the node is aJSON array, the system extractsdata from the array in the samemode. Nested JSON nodes areseparated by periods (.). Forexample, data.list.
fromJobConfig.encodeType
No String Encoding type. For example,UTF_8 or GBK.
fromJobConfig.splitType
No Enumeration Job split mode. The options are asfollows:l FILE: Files are split by file
quantity.l SIZE: Files are split by file
size.
fromJobConfig.fromFileOpType
No Enumeration Source file processing mode. Aftera job is completed, operations onthe source file can be performed.The source file can be renamed ordeleted.
fromJobConfig.useMarkerFile
No Boolean Whether to start a job by a markerfile. A job is started only when amarker file for starting the jobexists in the source path.Otherwise, the job will besuspended for a period of timespecified by Suspension Period.
fromJobConfig.markerFile
No String Name of the identifier file forstarting a job. After an identifierfile is specified, the task isexecuted only when the file existsin the source path. If the file is notspecified, this function is disabledby default. For example, ok.txt.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
76
Parameter Mandatory Type Description
fromJobConfig.waitTime
No String Period of waiting for a marker file.If you set Start Job by MarkerFile to Yes but no marker fileexists in the source path, the jobfails upon suspension timeout.If you set this parameter to 0 andno marker file exists in the sourcepath, the job will fail immediately.Unit: second
fromJobConfig.filterType
No Enumeration Whether to filter files usingwildcard characters or regularexpressions. When you select aregular expression, the Javaregular expression is used.l WILDCARD: Files are filtered
by wildcard character.l REGEX: Files are filtered by
Java regular expression.
fromJobConfig.pathFilter
No String Path filter, which is used to filterdirectories in the input path. Forexample, *input.
fromJobConfig.fileFilter
No String File filter, which is used to filterfiles in the input path. Forexample, *.csv.
fromJobConfig.formats
No Datastructure
Time format. This parameter ismandatory only when the fileformat is CSV_FILE and the timefield exists in the file. For details,see Description of thefromJobConfig.formatsparameter.
l Description of the fromJobConfig.formats parameter
Parameter Mandatory Type Description
name Yes String Column number. For example, 1.
value Yes String Time format. For example, yyyy-MM-dd.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
77
4.3.2.3 From HDFS
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.inputDirectory", "value": "/user/cdm" }, { "name": "fromJobConfig.inputFormat", "value": "PARQUET_FILE" }, { "name": "fromJobConfig.fieldSeparator", "value": "," }, { "name": "fromJobConfig.encodeType", "value": "UTF-8" }, { "name": "fromJobConfig.fromCompression", "value": "NONE" }, { "name": "fromJobConfig.splitType", "value": "FILE" }, { "name": "fromJobConfig.fromFileOpType", "value": "RENAME" }, { "name": "fromJobConfig.markerFile", "value": "ok.txt" }, { "name": "fromJobConfig.filterType", "value": "REGEX" }, { "name": "fromJobConfig.pathFilter", "value": "*input" }, { "name": "fromJobConfig.fileFilter", "value": "*.csv" } ], "name": "fromJobConfig" } ] }
Parameter Descriptionl HDFS job parameter description
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
78
Parameter Mandatory Type Description
fromJobConfig.inputDirectory
Yes String Path for storing data to beextracted. For example, /data_dir.
fromJobConfig.inputFormat
Yes Enumeration
File format required for datatransmission. Currently, thefollowing file formats aresupported:l CSV_FILE: CSV formatl PARQUET_FILE: Parquet
formatl BINARY_FILE: binary
formatWhen the parameter is set toBINARY_FILE, the connectorof the destination link must beHDFS, OBS, SFTP, FTP, orNAS.
fromJobConfig.columnList
No String Numbers of columns to beextracted. Use & to separatecolumn numbers in ascendingorder. For example, 1&3&5.
fromJobConfig.lineSeparator
No String Line feed character. Thisparameter is valid only when thefile format is CSV_FILE. Thedefault value is \r\n.
fromJobConfig.fieldSeparator
No String Field delimiter. This parameter isvalid only when the file format isCSV_FILE. The default valueis ,.
fromJobConfig.encodeType
No String Encoding type. For example,UTF_8 or GBK.
fromJobConfig.firstRowAsHeader
No Boolean Whether to regard the first lineas the heading line. Thisparameter is valid only when thefile format is CSV_FILE. Whenthis parameter is set to true,CDM reads the first line as theheading line when extractingdata, which means the headingline of the CSV file is removedin data transmission.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
79
Parameter Mandatory Type Description
fromJobConfig.fromCompression
No Enumeration
Compression format. Only thesource files in specifiedcompression format aretransferred. NONE indicatesfiles in all formats aretransferred.
fromJobConfig.splitType
No Enumeration
Job split mode. The options areas follows:l FILE: Files are split by file
quantity.l SIZE: Files are split by file
size.
fromJobConfig.fromFileOpType
No Enumeration
Source file processing mode.After a job is completed,operations on the source file canbe performed. The source filecan be renamed or deleted.
fromJobConfig.markerFile
No String Name of the identifier file forstarting a job. After an identifierfile is specified, the task isexecuted only when the fileexists in the source path. If thefile is not specified, this functionis disabled by default. Forexample, ok.txt.
fromJobConfig.filterType
No Enumeration
Whether to filter files usingwildcard characters or regularexpressions. When you select aregular expression, the Javaregular expression is used.l WILDCARD: Files are
filtered by wildcardcharacter.
l REGEX: Files are filtered byJava regular expression.
fromJobConfig.pathFilter
No String Path filter, which is used to filterdirectories in the input path. Forexample, *input.
fromJobConfig.fileFilter
No String File filter, which is used to filterfiles in the input path. Forexample, *.csv.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
80
Parameter Mandatory Type Description
fromJobConfig.formats
No Datastructure
Time format. This parameter ismandatory only wheninputFormat is set toCSV_FILE and the time fieldexists in the file. For details, seeDescription of thefromJobConfig.formatsparameter.
l Description of the fromJobConfig.formats parameter
Parameter Mandatory Type Description
name Yes String Column number. For example, 1.
value Yes String Time format. For example, yyyy-MM-dd.
4.3.2.4 From Hive
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.hive", "value": "hive" }, { "name": "fromJobConfig.database", "value": "sqoop" }, { "name": "fromJobConfig.table", "value": "invites" }, { "name": "fromJobConfig.columnList", "value": "foo&bar&ds" } ], "name": "fromJobConfig" } ] }
Parameter Description
Parameter Mandatory Type Description
fromJobConfig.hive
No String Data source to be extracted. Forexample, hive.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
81
Parameter Mandatory Type Description
fromJobConfig.database
No String Database from which data isextracted. For example, default.
fromJobConfig.table
Yes String Name of the table from which data isextracted. For example,TBL_EXAMPLE.
fromJobConfig.columnList
No String Numbers of columns to be extracted.Use & to separate column numbersin ascending order. For example,1&3&5.
4.3.2.5 From HBase/CloudTable
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.table", "value": "sqooptest" } ], "name": "fromJobConfig" } ] }
Parameter Descriptionl HBase job parameter description
Parameter Mandatory Type Description
fromJobConfig.table
Yes String Name of the table from which data isextracted. For example,TBL_EXAMPLE.
fromJobConfig.columnFamilies
No String Column family to which the data tobe extracted belongs
fromJobConfig.columns
No String Columns to be extracted. Use & toseparate column numbers and : toseparate column families andcolumns. For example,cf1:c1&cf2:c2.
fromJobConfig.isSplit
No Boolean Whether to split the rowkey. Forexample, true.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
82
Parameter Mandatory Type Description
fromJobConfig.delimiter
No String Delimiter used for splitting rowkeys.If this parameter is not set, row keyswill not be split. For example,vertical bars (|).
fromJobConfig.startTime
No String Minimum timestamp of the timerange (the time point is included).The format is yyyy-MM-ddhh:mm:ss.Only the data created at this point intime and later is extracted.
fromJobConfig.endTime
No String Maximum timestamp of the timerange (the time point is notincluded). The format is yyyy-MM-dd hh:mm:ss.Only the data created before thispoint in time is extracted.
fromJobConfig.formats
No Datastructure
Time format. This parameter ismandatory when the field is a timetype field.
l Description of the fromJobConfig.formats parameter
Parameter Mandatory Type Description
name Yes String Column number. For example, 1.
value Yes String Time format. For example, yyyy-MM-dd.
4.3.2.6 From FTP/SFTP/NAS
Sample JSON File "from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.inputDirectory", "value": "/from/bigdata" }, { "name": "fromJobConfig.inputFormat", "value": "CSV_FILE" }, { "name": "fromJobConfig.columnList", "value": "1&2&3&4&5&6&7&8&9&10&11&12" }, { "name": "fromJobConfig.fieldSeparator",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
83
"value": "," }, { "name": "fromJobConfig.encodeType", "value": "UTF-8" }, { "name": "fromJobConfig.splitType", "value": "SIZE" }, { "name": "fromJobConfig.filterType", "value": "WILDCARD" } ], "name": "fromJobConfig" } ] }
Parameter Description
Source link job parameters of FTP, SFTP, and NAS are the same.
l FTP, SFTP, or NAS job parameter description
Parameter Mandatory Type Description
fromJobConfig.inputDirectory
Yes String Data extraction path. A singlefile path can contain a maximumof 50 files, which are separatedby vertical bars (|). For example,FROM/example.csv|FROM/b.txt.
fromJobConfig.inputFormat
Yes Enumeration File format required for datatransmission. Currently, thefollowing file formats aresupported:l CSV_FILE: CSV formatl JSON_FILE: JSON formatl BINARY_FILE: binary
formatWhen the parameter is set toBINARY_FILE, the connectorof the destination link must beHDFS, OBS, SFTP, FTP, orNAS.
fromJobConfig.lineSeparator
No String Line feed character in the file.For example, \n. By default, \n,\r, or \r\n is automaticallyidentified.
fromJobConfig.columnList
No String Numbers of columns to beextracted. Use & to separatecolumn numbers in ascendingorder. For example, 1&3&5.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
84
Parameter Mandatory Type Description
fromJobConfig.lineSeparator
No String Line feed character. Thisparameter is valid only when thefile format is CSV_FILE. Thedefault value is \r\n.
fromJobConfig.fieldSeparator
No String Field delimiter. This parameter isvalid only when the file format isCSV_FILE. The default valueis ,.
fromJobConfig.quoteChar
No Boolean Whether to use the encirclingsymbol. If this parameter is setto true, the field delimiters inthe encircling symbol areregarded as a part of the stringvalue. Currently, the defaultencircling symbol of CDM isdouble quotation mark (").
fromJobConfig.regexSeparator
No Boolean Whether to use the regularexpression to separate fields.This parameter is valid onlywhen the file format isCSV_FILE.
fromJobConfig.regex
No String Regular expression. Thisparameter is valid only when theregular expression is used toseparate fields.
fromJobConfig.firstRowAsHeader
No Boolean Whether to regard the first lineas the heading line. Thisparameter is valid only when thefile format is CSV_FILE. Whenthis parameter is set to true,CDM reads the first line as theheading line when extractingdata, which means the headingline of the CSV file is removedin data transmission.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
85
Parameter Mandatory Type Description
fromJobConfig.fromCompression
No Enumeration Compression format. Thisparameter is valid only when thefile format is CSV_FILE orJSON. The options are asfollows:l None: Files in all formats are
transferred.l gzip: Files in gzip format are
transferred.l Zip: Files in Zip format are
transferred.
fromJobConfig.jsonReferenceNode
No String Reference node. This parameteris valid when the file format isJSON_FILE. Resolve data onthe JSON node. If the datacorresponding to the node is aJSON array, the system extractsdata from the array in the samemode. Nested JSON nodes areseparated by periods (.). Forexample, data.list.
fromJobConfig.encodeType
No String Encoding type. For example,UTF_8 or GBK.
fromJobConfig.splitType
No Enumeration Job split mode. The options areas follows:l FILE: Files are split by file
quantity.l SIZE: Files are split by file
size.
fromJobConfig.fromFileOpType
No Enumeration Source file processing mode.After a job is completed,operations on the source file canbe performed. The source filecan be renamed or deleted.
fromJobConfig.useMarkerFile
No Boolean Whether to start a job by amarker file. A job is started onlywhen a marker file for startingthe job exists in the source path.Otherwise, the job will besuspended for a period of timespecified by Suspension Period.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
86
Parameter Mandatory Type Description
fromJobConfig.markerFile
No String Name of the identifier file forstarting a job. After an identifierfile is specified, the task isexecuted only when the fileexists in the source path. If thefile is not specified, this functionis disabled by default. Forexample, ok.txt.
fromJobConfig.waitTime
No String Period of waiting for a markerfile. If you set Start Job byMarker File to Yes but nomarker file exists in the sourcepath, the job fails uponsuspension timeout.If you set this parameter to 0 andno marker file exists in thesource path, the job will failimmediately.Unit: second
fromJobConfig.filterType
No Enumeration Whether to filter files usingwildcard characters or regularexpressions. When you select aregular expression, the Javaregular expression is used.l WILDCARD: Files are
filtered by wildcardcharacter.
l REGEX: Files are filtered byJava regular expression.
fromJobConfig.pathFilter
No String Path filter, which is used to filterdirectories in the input path. Forexample, *input.
fromJobConfig.fileFilter
No String File filter, which is used to filterfiles in the input path. Forexample, *.csv.
fromJobConfig.formats
No Datastructure
Time format. This parameter ismandatory only when the fileformat is CSV_FILE and thetime field exists in the file. Fordetails, see Description of thefromJobConfig.formatsparameter.
l Description of the fromJobConfig.formats parameter
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
87
Parameter Mandatory Type Description
name Yes String Column number. For example, 1.
value Yes String Time format. For example, yyyy-MM-dd.
4.3.2.7 From HTTP/HTTPS
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.inputDirectory", "value": "http://esdk.huawei.com/ilink/esdk/download/HW_456706" }, { "name": "fromJobConfig.inputFormat", "value": "BINARY_FILE" } ], "name": "fromJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
fromJobConfig.inputDirectory
Yes String URL of the file to be extractedThese connectors are used to readfiles with an HTTP/HTTPS URL,such as reading public files on thethird-party object storage systemand web disks.
fromJobConfig.inputFormat
Yes Enumeration File format required for datatransmission. Currently, only thebinary format is supported:
4.3.2.8 From MongoDB/DDS
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.database", "value": "cdm" },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
88
{ "name": "fromJobConfig.collectionName", "value": "person" }, { "name": "fromJobConfig.columnList", "value": "id&name&gender&age&score" } ], "name": "fromJobConfig" } ] }
Parameter Description
Parameter Mandatory Type Description
fromJobConfig.database
Yes String Name of the MongoDB/DDSdatabase
fromJobConfig.collectionName
Yes String Name of the MongoDB/DDScollection
fromJobConfig.columnList
No String List of fields to be extracted. Use& to separate field names. Forexample, id&gid&name.
4.3.2.9 From Redis/DCS
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.isBatchMigration", "value": "false" }, { "name": "fromJobConfig.keyPrefix", "value": "cdm_string" }, { "name": "fromJobConfig.keySeparator", "value": ":" }, { "name": "fromJobConfig.valueStoreType", "value": "STRING" }, { "name": "fromJobConfig.valueSeparator", "value": "," }, { "name": "fromJobConfig.columnList", "value": "1" } ], "name": "fromJobConfig"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
89
} ] }
Parameter Descriptionl Redis job parameter description
Parameter Mandatory Type Description
fromJobConfig.isBatchMigration
No Boolean Whether to migrate all data in thedatabase
fromJobConfig.keyPrefix
Yes String Key prefix, which is the name ofthe corresponding associationtable.Mapping between Redis and theassociation table: Name of theassociation table + delimiter is aRedis key, and a row of data inthe association table is a Redisvalue.
fromJobConfig.keySeparator
Yes String Key delimiter, which separatesthe association table and primarykey
fromJobConfig.valueStoreType
Yes String Storage mode of rows of data inthe association table on Redis.The options are string and hash.l string: indicates that a row of
data is stored as a characterstring using delimiters toseparate columns. This modereduces storage spaceoccupation.
l hash: indicates that a row ofdata is stored in columnname:column value format inthe hash table.
fromJobConfig.valueSeparator
No String Value delimiter. The defaultvalue is \tab. This parameter isvalid when valueStoreType isset to string.
fromJobConfig.columnList
No String List of fields to be extracted. Use& to separate field names. Forexample, id&gid&name.
fromJobConfig.formats
No Datastructure
Time format. For details, seeDescription of thefromJobConfig.formatsparameter.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
90
l Description of the fromJobConfig.formats parameter
Parameter Mandatory Type Description
name Yes String Column number. For example, 1.
value Yes String Time format. For example, yyyy-MM-dd.
4.3.2.10 From DIS
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.streamName", "value": "test" }, { "name": "fromJobConfig.disConsumerStrategy", "value": "FROM_LAST_STOP" }, { "name": "fromJobConfig.isPermanency", "value": "true" }, { "name": "fromJobConfig.maxPollRecords", "value": "100" }, { "name": "fromJobConfig.shardId", "value": "0" }, { "name": "fromJobConfig.separator", "value": "," } ], "name": "fromJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
fromJobConfig.streamName
Yes String DIS stream name
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
91
Parameter Mandatory Type Description
fromJobConfig.disConsumerStrategy
Yes Enumeration Used to set the initial offset whendata is pulled from DIS. Theoptions are as follows:l LATEST: maximum offset,
that is, the latest datal FROM_LAST_STOP: data
after the last stopl EARLIEST: minimum offset,
that is, the earliest data
fromJobConfig.isPermanency
Yes Boolean Whether to run permanently
fromJobConfig.maxPollRecords
No String Maximum number of requests thatcan be sent to DIS each time
fromJobConfig.shardId
Yes String ID of the DIS partition. You canenter multiple partition IDs, whichare separated by commas (,).
fromJobConfig.separator
No String Field delimiter
4.3.2.11 From Kafka
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.topicsList", "value": "test1,test2" }, { "name": "fromJobConfig.kafkaConsumerStrategy", "value": "EARLIEST" }, { "name": "fromJobConfig.isPermanency", "value": "true" } ], "name": "fromJobConfig" } ] }
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
92
Parameter DescriptionParameter Mandatory Type Description
fromJobConfig.topicsList
Yes String List of Kafka topics. Separatemultiple topics by commas (,).
fromJobConfig.kafkaConsumerStrategy
Yes Enumeration Used to set the initial offset whendata is pulled from Kafka. Theoptions are as follows:l LATEST: maximum offset,
that is, the latest datal EARLIEST: minimum offset,
that is, the earliest data
fromJobConfig.isPermanency
Yes Boolean Whether to run permanently
fromJobConfig.groupId
No String Group of the Kafka offset
fromJobConfig.maxPollRecords
No String Maximum number of requests thatcan be sent to Kafka each time
fromJobConfig.maxPollInterval
No String Maximum interval between eachpoll
fromJobConfig.separator
No String Field delimiter
4.3.2.12 From Elasticsearch/Cloud Search Service
Sample JSON File"from-config-values": { "configs": [ { "inputs": [ { "name": "fromJobConfig.index", "value": "dg" }, { "name": "fromJobConfig.type", "value": "es" }, { "name": "fromJobConfig.columnList", "value": "a1:numeric&s1:string" }, { "name": "fromJobConfig.splitNestedField", "value": "true" }, { "name": "fromJobConfig.queryString", "value": "last_name:Smith" } ],
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
93
"name": "fromJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
fromJobConfig.index
Yes String Index of the extracted data, whichis similar to the database name inthe relational database
fromJobConfig.type
Yes String Type of the extracted data, which issimilar to the table name in therelational database
fromJobConfig.columnList
No String List of fields to be extracted. Use &to separate field names. Forexample, id&gid&name.
fromJobConfig.splitNestedField
No Boolean Whether to split the JSON contentof the nested field. For example, a:{ b:{ c:1, d:{ e:2, f:3 } } } can besplit into a.b.c, a.b.d.e, and a.b.d.f.
fromJobConfig.queryString
No String Whether to use the Elasticsearchquery string to filter the sourcedata. CDM migrates only the datathat meets the filter criteria.
4.3.3 Destination Job ParametersCDM allows you to import data to multiple types of data sources. Each data source has itsown job parameters. You can view specific input parameters based on the correspondingsection of the data source type.
Set the input parameters of the source link as described in Description of the inputsparameter of to-config-values in Creating a Job.
4.3.3.1 To a Relational Database
Sample JSON File"to-config-values": { "configs": [ { "inputs": [ { "name": "toJobConfig.schemaName", "value": "public" }, { "name": "toJobConfig.tablePreparation", "value": "DO_NOTHING" }, {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
94
"name": "toJobConfig.tableName", "value": "cdm_all_type" }, { "name": "toJobConfig.columnList", "value": "Ttiny&Tsmall&Tmediumint&Tint" }, { "name": "toJobConfig.shouldClearTable", "value": "false" } ], "name": "toJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.schemaName
No String Database mode or tablespace. Forexample, public.
toJobConfig.tablePreparation
Yes Enumeration This parameter is available onlywhen both the source anddestination databases are relationaldatabases. The options for datawrite to tables are as follows:l DO_NOTHING: Do not create
the table automatically.l CREATE_WHEN_NOT_EXI
ST: If the destination databasedoes not contain the tablespecified by tableName, CDMautomatically creates the table.
l DROP_AND_CREATE:Delete the table specified bytableName, and then create thetable again.
toJobConfig.tableName
Yes String Name of the table to which data iswritten. For example,TBL_EXAMPLE.
toJobConfig.columnList
No String List of fields to be loaded. Use &to separate field names. Forexample, id&gid&name.
toJobConfig.shouldClearTable
No Boolean If this parameter is set to true, datain the target table is cleared beforedata is written into the table.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
95
Parameter Mandatory Type Description
orientation No Enumeration Storage mode. This parameter isenabled only for the DWSdatabase. When the DWS databasetable needs to be automaticallycreated, the optional data storagemodes of the table are as follows:l ROW: Data in the table is
stored in rows.l COLUMN: Data in the table is
stored in columns.
isCompress No Boolean Whether to perform compression.This parameter is enabled only forthe DWS database. When theDWS database table needs to beautomatically created, you canspecify whether to store the data inthe table after compression.
oJobConfig.useStageTable
No Boolean Whether to import data to thephase table first. If this parameteris set to true, the data is importedto the phase table before it isimported to the destination table.After the data is successfullyimported to the phase table, it isthen imported from the phase tableto the destination table. In thisway, the data that is successfullyimported to the destination tableremains in case the data importfails.
extendCharLength
No Boolean Whether to extend the length ofthe character string field. If thisparameter is set to true, the lengthof the character string field in thedestination table is three times thelength of the corresponding field inthe source table when thedestination table needs to beautomatically created.
4.3.3.2 To OBS
Sample JSON File"to-config-values": { "configs": [ {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
96
"inputs": [ { "name": "toJobConfig.bucketName", "value": "cdm01" }, { "name": "toJobConfig.kmsEncryption", "value": "false" }, { "name": "toJobConfig.outputDirectory", "value": "/oss2obs/" }, { "name": "toJobConfig.outputFormat", "value": "BINARY_FILE" }, { "name": "toJobConfig.fieldSeparator", "value": "," }, { "name": "toJobConfig.writeToTempFile", "value": "false" }, { "name": "toJobConfig.validateMD5", "value": "true" }, { "name": "toJobConfig.recordMD5Result", "value": "true" }, { "name": "toJobConfig.recordMD5Link", "value": "obslink" }, { "name": "toJobConfig.recordMD5Bucket", "value": "cdm05" }, { "name": "toJobConfig.recordMD5Directory", "value": "/md5/" }, { "name": "toJobConfig.encodeType", "value": "UTF-8" }, { "name": "toJobConfig.duplicateFileOpType", "value": "REPLACE" } ], "name": "toJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.bucketName
Yes String OBS bucket name. For example,cdm.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
97
Parameter Mandatory Type Description
toJobConfig.outputDirectory
Yes String Path to which data is written. Forexample, /data_dir.
toJobConfig.outputFormat
Yes Enumeration File format required for datawrites (except the binaryformat). Currently, the followingfile formats are supported:l CSV_FILE: Write data in
CSV format.l BINARY_FILE: Files are
directly transferred withoutresolving the content. CDMwrites the file withoutchanging the file format.
l CARBONDATA: Write datain CarbonData format.
When the parameter is set toBINARY_FILE, the connectorof the source link must beHDFS, OBS, SFTP, or FTP.
toJobConfig.fieldSeparator
No String Column delimiter. Thisparameter is valid only when thefile format is CSV_FILE. Thedefault value is ,.
toJobConfig.lineSeparator
No String Line feed character. Thisparameter is valid only when thefile format is CSV_FILE. Thedefault value is \r\n.
toJobConfig.writeFileSize
No String Whether to fragment multiplefiles by size so that the files areexported in proper size. The unitis MB. This parameter is validwhen the migration source is adatabase.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
98
Parameter Mandatory Type Description
toJobConfig.duplicateFileOpType
No Enumeration Method for processing duplicatefiles. If the name and size of afile are the same as those ofanother file, the file is regardedas a duplicate file. Duplicatefiles can be processed in thefollowing ways:l REPLACE: Replace
duplicate files.l SKIP: Skip duplicate files.l ABANDON: Stop the job
when any duplicate file isfound.
toJobConfig.kmsEncryption
No Boolean Whether to encrypt the uploadeddata using Key ManagementService (KMS). If KMSencryption is enabled, MD5verification for data cannot beperformed.
toJobConfig.kmsID
No String Key used for encryption duringdata upload. You must create akey in KMS before data upload.
toJobConfig.validateMD5
No Boolean Whether to verify the MD5value. MD5 verification cannotbe used together with KMSencryption. MD5 values can beverified only when files aretransferred in BINARY_FILEformat.Calculate the MD5 value of thesource file and verify it with theMD5 value returned by OBS. Ifan MD5 file exists on the sourceend, directly read the MD5 fileand verify the MD5 file with theMD5 value returned by OBS.
toJobConfig.recordMD5Result
No Boolean Whether to record theverification result when the MD5value is verified
toJobConfig.recordMD5Link
No String OBS link where the bucket towhich the MD5 verificationresult is written resides
toJobConfig.recordMD5Bucket
No String OBS bucket to which the MD5verification result is written
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
99
Parameter Mandatory Type Description
toJobConfig.recordMD5Directory
No String Directory to which the MD5verification result is written
toJobConfig.encodeType
No String Encoding type. For example,UTF_8 or GBK.
toJobConfig.markerFile
No String Whether to generate a markerfile with a custom name in thedestination directory after a jobis executed successfully. If youdo not specify a file name, thisfunction is disabled by default.
toJobConfig.copyContentType
No Boolean This parameter is displayed onlywhen File Format is Binary andboth the migration source anddestination are object storage.If you set this parameter to Yes,the Content-Type attribute of thesource file is copied duringobject file migration. Thisfunction is mainly used for staticwebsite migrationThe Content-Type attributecannot be written to Archivebuckets. Therefore, if you setthis parameter to Yes, themigration destination must be anon-Archive bucket.
toJobConfig.quoteChar
No Boolean This parameter is displayed onlywhen File Format is CSV. It isused when database tables aremigrated to file systems.If you set this parameter to Yesand a field in the source datatable contains a field delimiter orline separator, CDM uses doublequotation marks (") as the quotecharacter to quote the fieldcontent as a whole to prevent afield delimiter from dividing afield into two fields, or a lineseparator from dividing a fieldinto different lines. For example,if the hello,world field in thedatabase is quoted, it will beexported to the CSV file as awhole.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
100
4.3.3.3 To HDFS
Sample JSON File"to-config-values": { "configs": [ { "inputs": [ { "name": "toJobConfig.outputDirectory", "value": "/data" }, { "name": "toJobConfig.outputFormat", "value": "CSV_FILE" }, { "name": "toJobConfig.fieldSeparator", "value": "," }, { "name": "toJobConfig.duplicateFileOpType", "value": "REPLACE" }, { "name": "toJobConfig.compression", "value": "NONE" }, { "name": "toJobConfig.appendMode", "value": "true" } ], "name": "toJobConfig" } ] }
Parameter Description
Parameter Mandatory Type Description
toJobConfig.outputDirectory
Yes String Path to which data is written. Forexample, /data_dir.
toJobConfig.outputFormat
Yes Enumeration File format required for data writes(except the binary format).Currently, the following fileformats are supported:l CSV_FILE: Write data in CSV
format.l BINARY_FILE: Files are
directly transferred withoutresolving the content. CDMwrites the file without changingthe file format.
When the parameter is set toBINARY_FILE, the connector ofthe source link must be HDFS,OBS, SFTP, or FTP.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
101
Parameter Mandatory Type Description
toJobConfig.lineSeparator
No String Line feed character. This parameteris valid only when the file format isCSV_FILE. The default value is \r\n.
toJobConfig.fieldSeparator
No String Column delimiter. This parameter isvalid only when the file format isCSV_FILE. The default value is ,.
toJobConfig.duplicateFileOpType
No Enumeration Method for processing duplicatefiles. If the name and size of a fileare the same as those of anotherfile, the file is regarded as aduplicate file. Duplicate files can beprocessed in the following ways:l REPLACE: Replace duplicate
files.l SKIP: Skip duplicate files.l ABANDON: Stop the job when
any duplicate file is found.
toJobConfig.compression
No Enumeration After the file is written, select thecompression format of the file. Thefollowing compression formats aresupported:l None: Do not compress the file.l DEFLATE: Compress the file
in DEFLATE format.l gzip: Compress the file in gzip
format.l bzip2: Compress the file in
bzip2 format.l LZ4: Compress the file in LZ4
format.l Snappy: Compress the file in
Snappy format.
toJobConfig.appendMode
Yes Boolean Whether to write data when one ormultiple files exist in the loadingpath. The default value is false.
4.3.3.4 To Hive
Sample JSON File"to-config-values": { "configs": [{ "inputs": [{
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
102
"name": "toJobConfig.hive", "value": "hive" }, { "name": "toJobConfig.database", "value": "default" }, { "name": "toJobConfig.table", "value": "cdm_tbl" }], "name": "toJobConfig" }]}
Parameter Description
Parameter Mandatory Type Description
toJobConfig.hive
No String Data source to which data iswritten
toJobConfig.database
No String Name of the database to whichdata is written. For example,default.
toJobConfig.table
Yes String Name of the table to which datais written
4.3.3.5 To HBase/CloudTable
Sample JSON File "to-config-values": [ { "inputs": [ { "name": "toJobConfig.table", "value": "sqoop_bulkload" }, { "name": "toJobConfig.storageType", "value": "BULKLOAD" }, { "name": "toJobConfig.columns", "value": "cf1:a&cf2:b&cf2:c" }, { "name": "toJobConfig.rowKeyColumn", "value": "cf1:a&cf2:b" }, { "name": "toJobConfig.isOverride", "value": "true" }, { "name": "toJobConfig.algorithm", "value": "GZ" } ], "name": "toJobConfig"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
103
} ]
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.table Yes String Name of the table to which data iswritten. For example,TBL_EXAMPLE.
toJobConfig.storageType
Yes Enumeration Mode for writing data to an HBasetable. The options are as follows:l BULKLOAD: The
BULKLOAD mode isrecommended to improve theloading performance.
l PUTLIST: The PUTLISTmode is recommended onlywhen the data volume is small.
toJobConfig.columns
No String Columns to be extracted. Use & toseparate column numbers and : toseparate column families andcolumns. For example,cf1:c1&cf2:c2.
toJobConfig.rowKeyColumn
Yes String Columns serve as rowkeys. Use &to separate column numbers and :to separate column families andcolumns. For example,cf1:c1&cf2:c2.
toJobConfig.isOverride
No Boolean Whether to clear data when data isimported in BULKLOAD mode.For example, true.
toJobConfig.delimiter
No String Delimiter used for separatingcolumns when multiple columnsare used as rowkeys. For example,vertical bars (|).
toJobConfig.isRowkeyRedundancy
No Boolean Whether to write rowkey data tothe HBase column at the sametime
toJobConfig.algorithm
No Enumeration Compression algorithm used whena new HBase table is created. TheSnappy and GZ algorithms aresupported. The default value isNone.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
104
Parameter Mandatory Type Description
toJobConfig.writeToWAL
No Boolean Whether to enable Write AheadLog (WAL) of HBase. The optionsare as follows:l Yes: If the HBase server breaks
down after the function isenabled, you can replay theoperations that have not beenperformed in WAL.
l No: If you set this parameter toNo, the write performance isimproved. However, if theHBase server breaks down,data may be lost.
toJobConfig.transType
No Boolean l true: Data of the Short, Int,Long, Float, Double, andDecimal columns in the sourcedatabase is converted intoByte[] arrays (binary) andwritten into HBase. Other typesof data are written as characterstrings. If several types of datamentioned above are combinedas rowkeys, they will be writtenas character strings.This function saves storagespace. In specific scenarios, therowkey distribution is evener.
l false: All types of data in thesource database are written intoHBase as character strings.
4.3.3.6 To FTP/SFTP/NAS
Sample JSON File"to-config-values": { "configs": [ { "inputs": [ { "name": "toJobConfig.outputDirectory", "value": "/opt/data" }, { "name": "toJobConfig.outputFormat", "value": "CSV_FILE" }, { "name": "toJobConfig.fieldSeparator", "value": "," }, {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
105
"name": "toJobConfig.duplicateFileOpType", "value": "REPLACE" } ], "name": "toJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.outputDirectory
Yes String Path to which data is written. Forexample, /data_dir.
toJobConfig.outputFormat
Yes Enumeration File format required for data writes(except the binary format).Currently, the following fileformats are supported:l CSV_FILE: Write data in CSV
format.l BINARY_FILE: Files are
directly transferred withoutresolving the content. CDMwrites the file without changingthe file format.
When the parameter is set toBINARY_FILE, the connector ofthe source link must be HDFS,OBS, SFTP, or FTP.
toJobConfig.duplicateFileOpType
No Enumeration Method for processing duplicatefiles. If the name and size of a fileare the same as those of anotherfile, the file is regarded as aduplicate file. Duplicate files canbe processed in the followingways:l REPLACE: Replace duplicate
files.l SKIP: Skip duplicate files.l ABANDON: Stop the job
when any duplicate file isfound.
toJobConfig.lineSeparator
No String Line feed character. Thisparameter is valid only when thefile format is CSV_FILE. Thedefault value is \r\n.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
106
Parameter Mandatory Type Description
toJobConfig.fieldSeparator
No String Column delimiter. This parameteris valid only when the file formatis CSV_FILE. The default valueis ,.
toJobConfig.encodeType
No String Encoding type. For example,UTF_8 or GBK.
toJobConfig.writeToTempFile
No Boolean This parameter is displayed onlywhen the migration source is a filesystem (OBS/FTP/SFTP/NAS/HDFS), the migration destinationis FTP/SFP/NAS, and File Formatis Binary.The binary file is written to a .tmpfile first. After the migration issuccessful, run the rename ormove command at the migrationdestination to restore the file.
toJobConfig.recordMD5Result
No Boolean This parameter is displayed onlywhen the migration source is a filesystem (OBS/FTP/SFTP/NAS/HDFS), the migration destinationis FTP/SFP/NAS, and File Formatis Binary.An MD5 hash value is generatedfor each transferred file, and thevalue is recorded in a new .md5file. You can specify the directorywhere the MD5 value is generated.
toJobConfig.recordMD5Directory
No String Directory for storing MD5 values
toJobConfig.markerFile
No String Whether to generate a marker filewith a custom name in thedestination directory after a job isexecuted successfully. If you donot specify a file name, thisfunction is disabled by default.
4.3.3.7 To DDS
Sample JSON File"to-config-values": { "configs": [ { "inputs": [ {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
107
"name": "toJobConfig.database", "value": "demo" }, { "name": "toJobConfig.collectionName", "value": "cdmbase" }, { "name": "toJobConfig.columnList", "value": "_char&_varchar" }, { "name": "toJobConfig.isBatchMigration", "value": "false" } ], "name": "toJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.database
Yes String Name of the MongoDB/DDSdatabase
toJobConfig.collectionName
Yes String Name of the MongoDB/DDScollection
toJobConfig.columnList
No String List of fields to be extracted. Use& to separate field names. Forexample, id&gid&name.
toJobConfig.isBatchMigration
No Boolean Whether to migrate all data in thedatabase
4.3.3.8 To DCS
Sample JSON File "to-config-values": { "configs": [ { "inputs": [ { "name": "toJobConfig.isBatchMigration", "value": "false" }, { "name": "toJobConfig.shouldClearDatabase", "value": "false" }, { "name": "toJobConfig.keyPrefix", "value": "cdm_string" }, { "name": "toJobConfig.keySeparator", "value": ":" },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
108
{ "name": "toJobConfig.primaryKeyList", "value": "1" }, { "name": "toJobConfig.valueStoreType", "value": "STRING" }, { "name": "toJobConfig.valueSeparator", "value": "," }, { "name": "toJobConfig.columnList", "value": "1&2&3&4&5&6&7&8&9&10&11&12" } ], "name": "toJobConfig" } ] }
Parameter Descriptionl Parameter description
Parameter Mandatory Type Description
toJobConfig.isBatchMigration
No Boolean Whether to migrate all data in thedatabase
toJobConfig.shouldClearDatabase
No Boolean Whether to clear data beforeimport
toJobConfig.keyPrefix
Yes String Key prefix, which is similar tothe table name of a relationaldatabaseMapping between Redis and theassociation table: Name of theassociation table + delimiter is aRedis key, and a row of data inthe association table is a Redisvalue.
toJobConfig.keySeparator
Yes String Key delimiter, which separatesthe association table and primarykey
toJobConfig.primaryKeyList
Yes String List of primary keys. Use & toseparate field names. Forexample, id&gid.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
109
Parameter Mandatory Type Description
toJobConfig.valueStoreType
Yes Enumeration
Storage mode of rows of data inthe association table on Redis.The options are string and hash.l STRING: indicates that a row
of data is stored as a characterstring, in which columns inthe row are separated byvalueSeparator.
l Hash: indicates that a row ofdata is stored in columnname:column value format inthe hash table.
toJobConfig.valueSeparator
No String Value delimiter. The default valueis \tab. This parameter is validwhen valueStoreType is set tostring.
toJobConfig.columnList
No String List of fields to be written. Use &to separate field names. Forexample, id&gid&name.
toJobConfig.formats
No Datastructure
Time format. For details, seeDescription of thetoJobConfig.formatsparameter.
l Description of the toJobConfig.formats parameter
Parameter Mandatory Type Description
name Yes String Column number. For example, 1.
value Yes String Time format. For example, yyyy-MM-dd.
4.3.3.9 To Elasticsearch/Cloud Search Service
Sample JSON File"to-config-values": { "configs": [ { "inputs": [ { "name": "toJobConfig.index", "value": "cdm" }, { "name": "toJobConfig.type", "value": "type1" }, {
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
110
"name": "toJobConfig.shouldClearType", "value": "false" }, { "name": "toJobConfig.pipeLine", "value": "es_03" } ], "name": "toJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.index Yes String Index of the written data, which issimilar to the database name in therelational database
toJobConfig.type Yes String Type of the written data, which issimilar to the table name in therelational database
toJobConfig.shouldClearType
No Boolean Whether to clear data before import
toJobConfig.primaryKey
No String Primary key or unique index
toJobConfig.columnList
No String List of fields to be written. Use &to separate field names. Forexample, id&gid&name.
toJobConfig.pipeLine
No String This parameter is available onlyafter a pipeline ID is created inKibana. It is used to convert thedata format using the datatransformation pipeline of CloudSearch Service or Elasticsearchafter data is transferred to CloudSearch Service or Elasticsearch.
4.3.3.10 To DLI
Sample JSON File"to-config-values": { "configs": [ { "inputs": [ { "name": "toJobConfig.queue", "value": "cdm" }, { "name": "toJobConfig.database",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
111
"value": "sqoop" }, { "name": "toJobConfig.table", "value": "t_test1" }, { "name": "toJobConfig.columnList", "value": "string_&int_&date_&double_&boolean_&short_×tamp_&long_&smallint_&bigint_" }, { "name": "toJobConfig.shouldClearTable", "value": "false" } ], "name": "toJobConfig" } ] }
Parameter DescriptionParameter Mandatory Type Description
toJobConfig.queue
Yes String Resource queue to which data iswritten
toJobConfig.database
Yes String DLI database to which data iswritten
toJobConfig.table Yes String Name of the table to which data iswritten
toJobConfig.columnList
No String List of fields to be loaded. Use & toseparate field names. For example,id&gid&name.
toJobConfig.shouldClearTable
No Boolean Whether to clear data in theresource queue before data import
4.3.4 Job Parameter Description
Sample JSON File"driver-config-values": { "configs": [ { "inputs": [ { "name": "throttlingConfig.numExtractors", "value": "3" }, { "name": "throttlingConfig.numLoaders", "value": "1" }, { "name": "throttlingConfig.recordDirtyData", "value": "true" },
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
112
{ "name": "throttlingConfig.writeToLink", "value": "obs_link_auto" }, { "name": "throttlingConfig.obsBucket", "value": "dirtydata" }, { "name": "throttlingConfig.dirtyDataDirectory", "value": "/user/sqoop/dirtydir" }, { "name": "throttlingConfig.maxErrorRecords", "value": "0" } ], "name": "throttlingConfig" }, { "inputs": [], "name": "jarConfig" }, { "inputs": [ { "name": "schedulerConfig.isSchedulerJob", "value": "true" }, { "name": "schedulerConfig.cycleType", "value": "minute" }, { "name": "schedulerConfig.cycle", "value": "30" }, { "name": "schedulerConfig.startDate", "value": "2018-01-22 14:57:23" } ], "name": "schedulerConfig" }, { "inputs": [], "name": "transformConfig" }, { "inputs": [ { "name": "schedulerConfig.isSchedulerJob", "value": "false" }, { "name": "schedulerConfig.disposableType", "value": "NONE" } ], "name": "schedulerConfig" }, { "inputs": [ { "name": "retryJobConfig.retryJobType", "value": "RETRY_TRIPLE" } ], "name": "retryJobConfig"
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
113
} ] }
Parameter DescriptionParameter Mandatory Type Description
throttlingConfig.numExtractors
No Integer Maximum number of extractionjobs. For example, 20.
throttlingConfig.numLoaders
No Integer This parameter is available onlywhen HBase or Hive serves as thedestination data source.Maximum number of loading jobs.For example, 5.
throttlingConfig.recordDirtyData
No Boolean Whether to write dirty data. Forexample, true.
throttlingConfig.writeToLink
No String Link to which dirty data is written.Currently, dirty data can be writtenonly to OBS or HDFS. Forexample, obslink.
throttlingConfig.obsBucket
No String Name of the OBS bucket to whichdirty data is written. This parameteris valid only when dirty data iswritten to OBS. For example,dirtyData.
throttlingConfig.dirtyDataDirectory
No String Directory to which dirty data iswrittenl To write dirty data to HDFS, set
this parameter to the specifiedHDFS directory.
l To write dirty data to OBS, setthis parameter to the directory inthe OBS bucket. For example, /data/dirtydata/.
throttlingConfig.maxErrorRecords
No String Maximum number of error recordsin a single shard. When the numberof error records of a map exceedsthe upper limit, the taskautomatically ends. The importeddata will not be rolled back.
schedulerConfig.isSchedulerJob
No Boolean Whether to enable a scheduled task.For example, true.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
114
Parameter Mandatory Type Description
schedulerConfig.cycleType
No String Cycle type of a scheduled task. Theoptions are as follows:l minute: minutel hour: hourl day: dayl week: weekl month: month
schedulerConfig.cycle
No Integer Cycle of a scheduled task. IfcycleType is set to minute andcycle is set to 10, the scheduledtask is executed every 10 minutes.
schedulerConfig.runAt
No String Time when a scheduled task istriggered in a cycle. This parameteris valid only when cycleType is setto hour, week, or month.l If cycleType is set to month,
cycle is set to 1, and runAt isset to 15, the scheduled task isexecuted on the 15th day ofeach month. You can set runAtto multiple values and separatethe values with commas (,).For example, if runAt is set to1,2,3,4,5, the scheduled task isexecuted on the first day, secondday, third day, fourth day, andfifth day of each month.
l If cycleType is set to week andrunAt is set tomon,tue,wed,thu,fri, thescheduled task is executed onMonday to Friday.
l If cycleType is set to hour andrunAt is set to 27,57, thescheduled task is executed at the27th and 57th minute in thecycle.
schedulerConfig.startDate
No String Start time of a scheduled task. Forexample, 2018-01-24 19:56:19.
schedulerConfig.stopDate
No String End time of a scheduled task. Forexample, 2018-01-27 23:59:00.If you do not set the end time, thescheduled task is always executedand will never stop.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
115
Parameter Mandatory Type Description
schedulerConfig.disposableType
No Enumeration Whether to delete a job after the jobis executed. The options are asfollows:l NONE: A job will not be
deleted after it is executed.l DELETE_AFTER_SUCCEE
D: A job will be deleted onlyafter it is successfully executed.It is applicable to massive one-time jobs.
l DELETE: A job will be deletedafter it is executed, regardless ofthe execution result.
retryJobConfig.retryJobType
No Enumeration Whether to automatically retry if ajob fails to be executed. Theoptions are as follows:l NONE: Do not retry.l RETRY_TRIPLE: Retry three
times.
4.3.5 Querying a Job
FunctionThis API is used to query all or a specified job.
URIl URI format
GET /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/job/{job_name}?filter={filter}&page_no={page_no}&page_size={page_size}
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
job_name Yes String Job name. When this parameter isset to all, all jobs are returned.
filter No String This parameter is used to enablefuzzy filtration when job_name isset to all.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
116
Parameter Mandatory Type Description
page_no No Integer Page number. Jobs on the specifiedpage will be returned.
page_size No Integer Number of jobs on each page
RequestSample requestGET /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/all
Responsel Sample response
{ "total": 1, "jobs": [{ "name": "oracle2hive", "type": "NORMAL_JOB", "creation-user": "cdm", "creation-date": 1514880878504, "update-date": 1514880878083, "update-user": "cdm", "status": "NEW", "from-link-name": "oraclelink", "from-connector-name": "generic-jdbc-connector", "to-link-name": "hivelink", "to-connector-name": "hive-connector", "from-config-values": { "configs": [{ "inputs": [{ "name": "fromJobConfig.schemaName", "value": "SQOOP" }, { "name": "fromJobConfig.tableName", "value": "SQ_INPUT" }, { "name": "fromJobConfig.columnList", "value": "SQI_ID&SQI_NAME" }], "name": "fromJobConfig", "type": "JOB" }] }, "to-config-values": { "configs": [{ "inputs": [{ "name": "toJobConfig.hive", "value": "hive" }, { "name": "toJobConfig.database", "value": "default" }, { "name": "toJobConfig.table", "value": "sq_input" }, { "name": "toJobConfig.columnList",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
117
"value": "sqi_id&sqi_name" }], "name": "toJobConfig", "type": "JOB" }], }, "driver-config-values": { "configs": [{ "inputs": [{ "name": "throttlingConfig.numExtractors", "value": "1" }] }] } }, }], "page_no": 1, "page_size": 20}
l Response parameter description
Parameter Mandatory Type Description
total No Integer Number of jobs
jobs Yes List Job list. For details, seeDescription of the jobsparameter.
page_no No Integer Page number. Jobs on thespecified page will be returned.
page_size No Integer Number of jobs on each page
simple No Boolean When this parameter is set totrue, compact information willbe returned. In other words,only parameter names andvalues will be returned.Attributes such as size, type,and id will not be returned.
l Description of the jobs parameter
Parameter Mandatory Type Description
name Yes String Job name
type Yes String Job type. The options are asfollows:l NORMAL_JOB: table/file
migrationl BATCH_JOB: entire DB
migration
creation-user Yes String User who created a job
creation-date Yes Timestamp Time when a job is created
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
118
Parameter Mandatory Type Description
update-date Yes Timestamp Time when a job is updated
update-user No String User who updated a job
status No String Status of a job. The options areas follows:l BOOTING: The job is
starting.l RUNNING: The job is
running.l SUCCEEDED: The job is
executed successfully.l FAILED: The job failed.l NEW: The job has not been
executed.
from-link-name
Yes String Source link name
from-connector-name
Yes String Name of the connector of thesource link
to-link-name Yes String Destination link name
to-connector-name
Yes String Name of the connector of thedestination link
from-config-values
Yes Dictionary Source link configuration. Fordetails, see Description offrom-config-values.
to-config-values
Yes Dictionary Destination link configuration.For details, see Description ofto-config-values.
driver-config-values
Yes Dictionary Job parameter configuration.For details, see Description ofdriver-config-values.
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
119
Return Value Description
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3.6 Modifying a Job
Function
This API is used to modify a job.
URIl URI format
PUT /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/job/{job_id}l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
job_id Yes String Job ID. The job name is used asthe unique job ID.
l Parameter description of a message body
The message body parameters for modifying a job are the same as those for creating ajob. For details, see Description of the jobs parameter in Creating a Job.
Request
Sample request
PUT /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/cluster/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/mysql2hive
{ "jobs": [{ "name": "mysql2hive", "from-link-name": "mysqllink", "from-connector-name": "generic-jdbc-connector", "to-link-name": "hivelink", "to-connector-name": "hive-connector", "from-config-values": { "configs": [{ "inputs": [{ "name": "fromJobConfig.schemaName",
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
120
"value": "default" }, { "name": "fromJobConfig.tableName", "value": "test_tbl" }, { "name": "fromJobConfig.partitionColumn", "value": "id" }], "name": "fromJobConfig", "type": "JOB" }] }, "to-config-values": { "configs": [{ "inputs": [{ "name": "toJobConfig.hive", "value": "hive" }, { "name": "toJobConfig.database", "value": "default" }, { "name": "toJobConfig.table", "value": "test_tbl" }], "name": "toJobConfig", "type": "JOB" }] }, "driver-config-values": { "configs": [{ "inputs": [{ "name": "throttlingConfig.numExtractors", "value": "1" }], "name": "throttlingConfig", "type": "JOB" }] } }]}
Responsel Sample response
{ "name": "mysql2hive", "validation-result": [{},{},{}]}
l Parameter Description
Parameter Mandatory Type Description
name Yes String Job name
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
121
Parameter Mandatory Type Description
validation-result Yes List Validation resultl If a job fails to be
modified, the failurecause is returned.
l If a job is successfullymodified, a blank list isreturned.
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3.7 Starting a Job
FunctionThis API is used to start a job.
URIl URI format
PUT /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/job/{job_name}/start
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
job_name Yes String Job name
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
122
Request
Sample requestPUT /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/jdbc2hive/start
Responsel Sample response
{ "submissions": [{ "progress": 1, "job-name": "jdbc2hive", "status": "BOOTING", "creation-date": 1496905778725, "creation-user": "cdm" }]}
l Parameter description
Parameter Mandatory Type Description
submissions Yes List Job running information.For details, seeDescription of thesubmissions parameter.
l Description of the submissions parameter
Parameter Mandatory Type Description
progress Yes Integer Job progress. If a job fails, thevalue is -1. Otherwise, thevalue is 0 to 100.
job-name Yes String Job name
status Yes String Job status. The options are asfollows:l BOOTING: The job is
starting.l FAILURE_ON_SUBMIT
: The job fails to besubmitted.
l RUNNING: The job isrunning.
l SUCCEEDED: The job isexecuted successfully.
l FAILED: The job failed.l UNKNOWN: The job
status is unknown.l NEVER_EXECUTED:
The job has not beenexecuted.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
123
Parameter Mandatory Type Description
creation-date Yes Timestamp Time when a job is created
creation-user Yes String User who created a job
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3.8 Stopping a Job
Function
This API is used to stop a running job.
URIl URI format
PUT /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/job/{job_name}/stop
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
job_name Yes String Job name
Request
Sample request
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
124
PUT /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/jdbc2hive/stop
Responsel Sample response
{ "submissions": [{ "progress": -1, "job-name": "jdbc2hive", "status": "FAILED", "creation-date": 1496905778725, "creation-user": "cdm" }]}
l Parameter description
Parameter Mandatory Type Description
submissions Yes List Job running information.For details, seeDescription of thesubmissions parameter.
l Description of the submissions parameter
Parameter Mandatory Type Description
progress Yes Integer Job progress. If a job fails,the value is -1. Otherwise, thevalue is 0 to 100.
job-name Yes String Job name
status Yes String Job status. The options are asfollows:l BOOTING: The job is
starting.l FAILURE_ON_SUBMI
T: The job fails to besubmitted.
l RUNNING: The job isrunning.
l SUCCEEDED: The jobis executed successfully.
l FAILED: The job failed.l UNKNOWN: The job
status is unknown.l NEVER_EXECUTED:
The job has not beenexecuted.
creation-date Yes Timestamp Time when a job is created
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
125
Parameter Mandatory Type Description
creation-user Yes String User who created a job
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3.9 Querying Job Status
Function
This API is used to query job running status.
URIl URI format
GET /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/job/{job_name}/status
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
job_name Yes String Job name
Request
Sample requestGET /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/jdbc2hive/status
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
126
Responsel Sample job running response
{ "submissions": [{ "progress": 1, "job-name": "jdbc2hive", "status": "BOOTING", "creation-date": 1496905778725, "creation-user": "cdm" }]}
l Sample response of a successful job{ "submissions": [{ "progress": 0, "job-name": "jdbc2hive", "status": "SUCCEEDED", "creation-date": 1496906582997, "creation-user": "cdm", "isStopingIncrement": "", "last-update-date": 1496916580005, "is-execute-auto": false, "last-udpate-user": "cdm", "isDeleteJob": false, "isIncrementing": false, "external-id": "job_local1127970451_0009", "counters": { "org.apache.sqoop.submission.counter.SqoopCounters": { "BYTES_WRITTEN": -1, "TOTAL_FILES": -1, "BYTES_READ": -1, "FILES_WRITTEN": -1, "TOTAL_SIZE": -1, "FILES_READ": -1, "ROWS_WRITTEN": 80, "ROWS_READ": 80 } } }]}
l Parameter description
Parameter Mandatory Type Description
submissions Yes List Job runninginformation. Fordetails, seeDescription of thesubmissionsparameter.
l Description of the submissions parameter
Parameter Mandatory Type Description
job-name Yes String Job name
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
127
Parameter Mandatory Type Description
status Yes String Job status. The options are asfollows:l BOOTING: The job is
starting.l RUNNING: The job is
running.l SUCCEEDED: The job
is executed successfully.l FAILED: The job failed.l NEW: The job has not
been executed.
creation-date Yes Timestamp Time when a job is created
creation-user Yes String User who created a job
isStopingIncrement
Yes Boolean Whether to stop incrementalmigration
last-update-date Yes Timestamp Time when a job is updated
is-execute-auto Yes Boolean Whether to execute a job atthe scheduled time
last-udpate-user Yes String Last user who updates the jobstatus
isDeleteJob Yes Boolean Whether to delete a job afterthe job is executed
isIncrementing Yes Boolean Whether a job is anincremental migration job
external-id Yes String Job ID
counters No Dictionary Job running result statistics.This parameter is availableonly when status isSUCCEEDED. For details,see Description of thecounters parameter.
l Description of the counters parameter
Parameter Mandatory Type Description
org.apache.sqoop.submission.counter.SqoopCounters
Yes List Job running resultstatistics. For details, seeDescription oforg.apache.sqoop.submission.counter.SqoopCounters.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
128
l Description of org.apache.sqoop.submission.counter.SqoopCounters
Parameter Mandatory Type Description
ROWS_WRITTEN No Integer Number of rows that arewritten
ROWS_READ No Integer Number of rows that areread
BYTES_WRITTEN No Integer Number of bytes that arewritten
BYTES_READ No Integer Number of bytes that areread
TOTAL_SIZE No Integer Total number of bytes
FILES_WRITTEN No Integer Number of files that arewritten
FILES_READ No Integer Number of files that areread
FILES_SKIPPED No Integer Number of files that areskipped
TOTAL_FILES No Integer Total number of files
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
129
4.3.10 Querying Job Execution History
FunctionThis API is used to query the historical job execution status.
URIl URI format
GET /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/submissions?jname={job_name}
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
cluster_id Yes String Cluster ID
job_name Yes String Job name
RequestSample requestGET /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/submissions?jname=jdbc2hive
Responsel Sample job running response
{ "submissions": [{ "progress": 1, "job-name": "jdbc2hive", "status": "BOOTING", "creation-date": 1496905778725, "creation-user": "cdm" }]}
l Sample response of a successful job{ "submissions": [{ "progress": 0, "job-name": "jdbc2hive", "status": "SUCCEEDED", "creation-date": 1496906582997, "creation-user": "cdm", "isStopingIncrement": "", "last-update-date": 1496916580005, "is-execute-auto": false, "last-udpate-user": "cdm", "isDeleteJob": false, "isIncrementing": false, "external-id": "job_local1127970451_0009", "counters": { "org.apache.sqoop.submission.counter.SqoopCounters": { "BYTES_WRITTEN": -1, "TOTAL_FILES": -1, "BYTES_READ": -1, "FILES_WRITTEN": -1,
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
130
"TOTAL_SIZE": -1, "FILES_READ": -1, "ROWS_WRITTEN": 80, "ROWS_READ": 80 } } }]}
l Parameter description
Parameter Mandatory Type Description
submissions Yes List Job running information.For details, seeDescription of thesubmissions parameter.
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
4.3.11 Deleting a Job
Function
This API is used to delete a specified job.
URIl URI format
DELETE /cdm/v1.0/{project_id}/clusters/{cluster_id}/cdm/job/{job_name}
l Parameter description
Parameter Mandatory Type Description
project_id Yes String Project ID
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
131
Parameter Mandatory Type Description
cluster_id Yes String Cluster ID
job_name Yes String Job name
RequestSample requestDELETE /cdm/v1.0/1551c7f6c808414d8e9f3c514a170f2e/clusters/6ec9a0a4-76be-4262-8697-e7af1fac7920/cdm/job/jdbc2hive
Responsel Successful sample response
{}l Failed sample response
{ "errCode": "Cdm.0100", "externalMessage": "Job[jdbc2hive] doesn't exist."}
l Parameter description of a failed response
Parameter Mandatory Type Description
errCode Yes String Error code
externalMessage Yes String Error message
Return Valuel Normal
200l Abnormal
Return Value Description
400 Bad Request Request error.
401 Unauthorized Authentication failed.
403 Forbidden No operation permission.
404 Not Found The requested resource is not found.
500 Internal Server Error Internal error. For details about the returnederror code, see Error Code.
503 Service Unavailable Service unavailable.
Cloud Data MigrationAPI Reference 4 API Description
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
132
5 Common Parameters
5.1 Error CodeNo data is returned if an API fails to be called. You can locate the cause of an error accordingto the error code of each API. When the API calling fails, HTTP status code 4xx or 5xx isreturned. The returned message body contains the specific error code and error information. Ifyou fail to locate the cause of an error, contact HUAWEI CLOUD service support and providethe error code so that we can help you solve the problem as soon as possible.
l Sample exception response{ "errCode": "Cdm.0100", "externalMessage": "Job[jdbc2hive] doesn't exist."}
l Parameter description
Parameter Mandatory Type Description
errCode No String Error code
externalMessage No String Error message
l Error code
In the following error message, %s is a variable. In the actual situation, the parametername, table name, job name, and link name are replaced with the actual values.
Error Code Status Code Error Message
Cdm.0000 500 System error.
Cdm.0009 500 %s is not an integer or is beyond the valuerange [0, 2147483647].
Cdm.0010 500 The integer must be within the range of[%s].
Cdm.0011 500 Validator has malformed argument.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
133
Error Code Status Code Error Message
Cdm.0012 500 JDBC driver class is not found.
Cdm.0014 500 Invalid parameter.
Cdm.0015 500 An error occurred during file parse.
Cdm.0016 500 The file to be uploaded cannot be empty.
Cdm.0017 500 MRS Kerberos authentication failed.
Cdm.0018 500 The content of jobs or links is invalid.
Cdm.0019 500 Log %s does not exist.
Cdm.0020 500 The string must contain the followingsubstring: %s
Cdm.0021 500 Failed to connect to the server: %s .
Cdm.0024 500 [%s] must be within the range of [%s].
Cdm.0051 500 Invalid submission engine: %s
Cdm.0052 500 Job %s is running.
Cdm.0053 500 Job %s is not running.
Cdm.0054 500 Job %s does not exist.
Cdm.0055 500 Unsupported job type.
Cdm.0056 500 Failed to submit the job. Cause: %s
Cdm.0057 500 Invalid job execution engine: %s
Cdm.0058 500 Invalid combination of submission andexecution engines.
Cdm.0059 500 Job %s has been disabled. Failed to submitthe job.
Cdm.0060 500 Link %s for this job has been disabled.Failed to submit the job.
Cdm.0061 500 Connector %s does not support thespecified direction. Failed to submit thejob.
Cdm.0062 500 The binary file is applicable only to theSFTP, FTP, HDFS, or OBS connector.
Cdm.0063 500 An error occurred during table creation.Cause: %s
Cdm.0064 500 The data format is incorrect.
Cdm.0065 500 Failed to start the scheduler. Cause: %s
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
134
Error Code Status Code Error Message
Cdm.0066 500 Failed to obtain the sample value. Cause:%s
Cdm.0067 500 Failed to obtain the schema. Cause: %s
Cdm.0100 500 Job [%s] does not exist.
Cdm.0101 500 Link [%s] does not exist.
Cdm.0102 500 Connector [%s] does not exist.
Cdm.0104 500 The job name already exists.
Cdm.0010 500 The integer must be within the range of[%s].
Cdm.0201 500 Failed to obtain the instance.
Cdm.0202 500 The job status is unknown.
Cdm.0204 500 No MRS link is created.
Cdm.0230 500 Failed to load the specified class: %s
Cdm.0231 500 Failed to initialize the specified class: %s
Cdm.0232 500 Failed to write data. Cause: %s
Cdm.0233 500 An exception occurred during dataextraction. Cause: %s
Cdm.0234 500 An exception occurred during data loading.Cause: %s
Cdm.0235 500 All data has been consumed. Cause: %s
Cdm.0236 500 Invalid partitions have been retrieved fromPartitioner.
Cdm.0237 500 Failed to find the JAR file of the connector.
Cdm.0238 500 %s cannot be empty.
Cdm.0239 500 Failed to obtain HDFS. Cause: %s
Cdm.0240 500 Failed to obtain the status of the %s file.
Cdm.0241 500 Failed to obtain the type of the %s file.
Cdm.0242 500 An exception occurs during file check: %s
Cdm.0243 500 Failed to rename %s to %s.
Cdm.0244 500 Failed to create the %s file.
Cdm.0245 500 Failed to delete the %s file.
Cdm.0246 500 Failed to create the %s directory.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
135
Error Code Status Code Error Message
Cdm.0247 500 HBase operation failure. Cause: %s
Cdm.0248 500 Failed to clear data in %s. Cause: %s
Cdm.0249 500 The file name %s is invalid.
Cdm.0250 500 Failed to perform operations in the path: %s
Cdm.0251 500 Failed to load data to HBase. Cause: %s
Cdm.0307 500 Failed to release the link. Cause: %s
Cdm.0315 500 The Link name %s already exists.
Cdm.0316 500 The link that does not exist cannot beupdated.
Cdm.0317 500 Link %s is invalid.
Cdm.0318 500 The job already exists and cannot becreated repeatedly.
Cdm.0319 500 The job that does not exist cannot beupdated.
Cdm.0320 500 Job %s is invalid.
Cdm.0321 500 Link %s has been used.
Cdm.0322 500 Job %s has been used.
Cdm.0323 500 The submission already exists and cannotbe created repeatedly.
Cdm.0327 500 Invalid link or job: %s
Cdm.0411 500 An error occurred when connecting to thefile server.
Cdm.0412 500 An error occurred when disconnecting fromthe file server.
Cdm.0413 500 An error occurred in data transfer to the fileserver.
Cdm.0415 500 An error occurred when downloading filesfrom the file server.
Cdm.0416 500 An error occurred during data extraction.
Cdm.0420 500 The source file or source directory does notexist.
Cdm.0423 500 Duplicate files exist in the destination path.
Cdm.0500 500 The source directory or the [%s] file doesnot exist.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
136
Error Code Status Code Error Message
Cdm.0501 500 Invalid URI [%s].
Cdm.0518 500 Failed to connect to HDFS. Cause: %s
Cdm.0600 500 Failed to connect to the FTP server.
Cdm.0700 500 Failed to connect to the SFTP server.
Cdm.0800 500 Failed to connect to the OBS server.
Cdm.0801 500 OBS bucket [%s] does not exist.
Cdm.0900 500 Table [%s] does not exist.
Cdm.0901 500 Failed to connect to the database server.Cause: %s
Cdm.0902 500 Failed to execute the SQL statement.Cause: %s
Cdm.0903 500 Failed to obtain metadata. Cause: %s
Cdm.0904 500 An error occurred while retrieving datafrom the result. Cause: %s
Cdm.0905 500 No partition column is found.
Cdm.0906 500 No boundary is found in the partitioncolumn.
Cdm.0911 500 The table name or SQL must be specified.
Cdm.0912 500 The table name and SQL cannot bespecified at the same time.
Cdm.0913 500 Schema and SQL cannot be specified at thesame time.
Cdm.0914 500 Partition column is mandatory for query-based import.
Cdm.0915 500 The SQL-based import mode andcolumnList cannot be used at the sametime.
Cdm.0916 500 Last value is mandatory for incrementalread.
Cdm.0917 500 Last value cannot be obtained without fieldcheck.
Cdm.0918 500 If no transfer table is specified,shouldClearStageTable cannot bespecified.
Cdm.0921 500 Type %s is not supported.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
137
Error Code Status Code Error Message
Cdm.0925 500 The partition column contains unsupportedvalues.
Cdm.0926 500 Failed to obtain the schema. Cause: %s
Cdm.0927 500 The transfer table is not empty.
Cdm.0928 500 An error occurred when data is migratedfrom the transfer table to the destinationtable.
Cdm.0931 500 Schema column size [%s] does not matchthe result set column size [%s].
Cdm.0932 500 Failed to obtain the maximum value of thefield.
Cdm.0934 500 Multiple tables of the same name exist indifferent schemas or catalogs.
Cdm.0935 500 No primary key. Specify the partitioncolumn.
Cdm.0936 500 The number of dirty data records reachesthe upper limit.
Cdm.0940 500 Failed to match the exact table name.
Cdm.0941 500 Failed to connect to the server. Cause: %s
Cdm.0950 500 Failed to connect to the database with theexisting authentication information.
Cdm.0960 500 serverlist must be specified.
Cdm.0961 500 Invalid serverlist format.
Cdm.0962 500 The host IP address must be specified.
Cdm.0963 500 The host port must be specified.
Cdm.0964 500 The database must be specified.
Cdm.1000 500 Hive table [%s] does not exist.
Cdm.1010 500 Invalid URI [%s]. The URI must be eithernull or a valid URI.
Cdm.1011 500 Failed to connect to Hive. Cause: %s
Cdm.1100 500 Table [%s] does not exist.
Cdm.1101 500 Failed to obtain the link. Cause: %s
Cdm.1102 500 Failed to create the table. Cause: %s
Cdm.1103 500 No rowkey is found.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
138
Error Code Status Code Error Message
Cdm.1104 500 Failed to open the table. Cause: %s
Cdm.1105 500 Failed to initialize the job. Cause: %s
Cdm.1111 500 The table name is mandatory.
Cdm.1112 500 The import mode is mandatory.
Cdm.1113 500 Whether to clear data before import has notbeen specified.
Cdm.1114 500 The rowkey is empty. Set it in fieldmapping.
Cdm.1115 500 Columns is empty. Set it in field mapping.
Cdm.1116 500 Duplicate column names. Set it in fieldmapping.
Cdm.1117 500 An error occurred when checking whetherthe table exists. Cause: %s
Cdm.1118 500 Table %s does not contain the %s columnfamily.
Cdm.1119 500 The number of column families is %s andthe number of columns is %s.
Cdm.1120 500 The table contains data. Clear the table dataor set the configuration item to specifywhether to clear the table data before theimport.
Cdm.1121 500 Failed to close the link. Cause: %s
Cdm.1400 500 Failed to connect to the NAS server.
Cdm.1401 500 No permission to access the NAS server.
Cdm.1201 500 Failed to connect to the Redis server.Cause: %s
Cdm.1202 500 Failed to connect to the Redis cluster insingle-node mode.
Cdm.1203 500 Failed to extract data from the Redis server.Cause: %s
Cdm.1205 500 The prefix of the Redis value cannot beblank.
Cdm.1206 500 The storage type of the Redis value must bestring or hash.
Cdm.1207 500 When the value of the storage type isstring, valueSeparator must be specified.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
139
Error Code Status Code Error Message
Cdm.1208 500 columnList of Redis must be specified.
Cdm.1209 500 keySeparator of Redis key cannot beempty.
Cdm.1210 500 primaryKeyList of Redis must bespecified.
Cdm.1211 500 primaryKeyList of Redis must exist incolumnList.
Cdm.1212 500 databaseType of Redis must be Originalor DCS.
Cdm.1213 500 serverlist of Redis must be specified.
Cdm.1301 500 Failed to connect to the MongoDB server.Cause: %s
Cdm.1302 500 Failed to extract data from the MongoDBserver. Cause: %s
Cdm.1304 500 The collection of MongoDB servers mustbe specified.
Cdm.1305 500 serverlist of MongoDB must be specified.
Cdm.1306 500 The database name of the MongoDBservice must be specified.
Cdm.1307 500 serverlist of MongoDB must be specified.
Cdm.1400 500 Failed to connect to the NAS server.
Cdm.1401 500 No permission to access the NAS server.
Cdm.1501 500 Failed to connect to the Elasticsearchserver. Cause: %s
Cdm.1502 500 Failed to write data to the Elasticsearchserver. Cause: %s
Cdm.1503 500 Failed to close the Elasticsearch link.Cause: %s
Cdm.1504 500 An error occurred when obtaining theElasticsearch index. Cause: %s
Cdm.1505 500 An error occurred when obtaining theElasticsearch type. Cause: %s
Cdm.1506 500 An error occurred when obtaining theElasticsearch field. Cause: %s
Cdm.1507 500 An error occurred when obtaining theElasticsearch sample data. Cause: %s
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
140
Error Code Status Code Error Message
Cdm.1508 500 The host name or IP address of theElasticsearch server must be specified.
Cdm.1509 500 The port of the Elasticsearch server must bespecified.
Cdm.1510 500 The Elasticsearch index must be specified.
Cdm.1511 500 The Elasticsearch type must be specified.
Cdm.1512 500 columnList of Elasticsearch must bespecified.
Cdm.1513 500 columnList must contain the field typedefinition.
Cdm.1514 500 columnList must contain primaryKey.
Cdm.1515 500 An error occurred when resolving the JSONcharacter string. Cause: %s
Cdm.1516 500 The column name %s is invalid.
Cdm.1517 500 An error occurred when obtaining thenumber of documents.
Cdm.1518 500 The partition fails to be created.
Cdm.1519 500 An error occurred during data extraction.
Cdm.1520 500 Failed to obtain the type. Cause: %s
Cdm.1601 500 Failed to connect to the server.
Cdm.1603 500 Failed to obtain the sample value of the %stopic.
Cdm.1604 500 No data exists in topic %s.
Cdm.1605 500 Invalid brokerList.
5.2 Status CodeTable 5-1 describes the status code.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
141
Table 5-1 Status Code
StatusCode
Code Description
100 Continue The client continues sending the request.This interim response is used to inform the client thatthe initial part of the request has been received andhas not yet been rejected by the server.
101 Switching Protocols Switching protocols. The target protocol must bemore advanced than the source protocol.For example, the current HTTP protocol is switchedto a later version.
201 Created The request for creating a resource has been fulfilled.
202 Accepted The request has been accepted, but the processing hasnot been completed.
203 Non-AuthoritativeInformation
The server successfully processed the request, but isreturning information that may be from anothersource.
204 NoContent The server has successfully processed the request, buthas not returned any content.The status code is returned in response to an HTTPOPTIONS request.
205 Reset Content The server has fulfilled the request, but the requesteris required to reset the content.
206 Partial Content The server has successfully processed a part of theGET request.
300 Multiple Choices There are multiple options for the location of therequested resource. The response contains a list ofresource characteristics and addresses from which theuser or user agent (such as a browser) can choose themost appropriate one.
301 Moved Permanently The requested resource has been assigned a newpermanent URI, and the new URI is contained in theresponse.
302 Found The requested resource resides temporarily under adifferent URI.
303 See Other Retrieve a location.The response to the request can be found under adifferent URI and should be retrieved using a GET orPOST method.
304 Not Modified The requested resource has not been modified. Whenthe server returns this status code, it does not returnany resources.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
142
StatusCode
Code Description
305 Use Proxy The requested resource must be accessed through aproxy.
306 Unused The HTTP status code is no longer used.
400 BadRequest Invalid request.The client should not repeat the request withoutmodifications.
401 Unauthorized The status code is returned after the client providesthe authentication information, indicating that theauthentication information is incorrect or invalid.
402 Payment Required This status code is reserved for future use.
403 Forbidden The server understood the request, but is refusing tofulfill it.The client should not repeat the request withoutmodifications.
404 NotFound The requested resource cannot be found.The client should not repeat the request withoutmodifications.
405 MethodNotAllowed The method specified in the request is not supportedfor the requested resource.The client should not repeat the request withoutmodifications.
406 Not Acceptable The server cannot fulfill the request according to thecontent characteristics of the request.
407 Proxy AuthenticationRequired
This status code is similar to 401, but indicates thatthe client must first authenticate itself with the proxy.
408 Request Time-out The request timed out.The client may repeat the request withoutmodifications at any later time.
409 Conflict The request could not be completed due to a conflictwith the current state of the resource.This status code indicates that the resource that theclient attempts to create already exits, or the requestfails to be processed because of the update of theconflict request.
410 Gone The requested resource is no longer available.The status code indicates that the requested resourcehas been deleted permanently.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
143
StatusCode
Code Description
411 Length Required The server refuses to process the request without adefined Content-Length.
412 Precondition Failed The server does not meet one of the preconditionsthat the requester puts on the request.
413 Request Entity TooLarge
The request is larger than that a server is able toprocess. The server may close the connection toprevent the client from continuing the request. If theserver cannot process the request temporarily, theresponse will contain a Retry-After header field.
414 Request-URI TooLarge
The URI provided was too long for the server toprocess.
415 Unsupported MediaType
The server is unable to process the media format inthe request.
416 Requested range notsatisfiable
The requested range is invalid.
417 Expectation Failed The server fails to meet the requirements of theExpect request-header field.
422 UnprocessableEntity The request is well-formed but is unable to beprocessed due to semantic errors.
429 TooManyRequests The client has sent more requests than its rate limit isallowed within a given amount of time, or the serverhas received more requests than it is able to processwithin a given amount of time. In this case, it isadvisable for the client to re-initiate requests after thetime specified in the Retry-After header of theresponse expires.
500 InternalServerError The server is able to receive the request but it couldnot understand the request.
501 Not Implemented The server does not support the requested function.
502 Bad Gateway The server is acting as a gateway or proxy andreceives an invalid request from a remote server.
503 ServiceUnavailable The requested service is invalid.The client should not repeat the request withoutmodifications.
504 ServerTimeout The request cannot be fulfilled within a given time.This status code is returned to the client only whenthe Timeout parameter is specified in the request.
505 HTTP Version notsupported
The server does not support the HTTP protocolversion used in the request.
Cloud Data MigrationAPI Reference 5 Common Parameters
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
144
A Change History
Release Date What's New
2018-08-03 This is the tenth official release.l Added the cdm.xlarge flavor to Creating a Cluster.l Updated the Start Job by Marker File and Use Quote
Character functions for file migration jobs in the followingsections:– From OBS/OSS– From FTP/SFTP/NAS– To OBS– To FTP/SFTP/NAS
l Added incremental migration in MySQL Binlog mode toFrom a Relational Database.
l Added the data type match function of HBase to To HBase/CloudTable.
2018-07-05 This is the ninth official release.l Added Link to Qiniu Cloud Object Storage (KODO).l Added the linkConfig.runMode parameter to the HDFS and
HBase links. The following sections are updated:– Link to HDFS– Link to HBase
l Updated From Elasticsearch/Cloud Search Service. WhenElasticsearch or Cloud Search Service is the migration source,data can be filtered using filter criteria.
l Updated To OBS. When exported to OBS, the data isfragmented by size.
Cloud Data MigrationAPI Reference A Change History
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
145
Release Date What's New
2018-06-02 This is the eighth official release.l Added From HTTP/HTTPS.l Added the CarbonData format of OBS to the following
sections:– From OBS/OSS– To OBS
l Added the retry function after configuration failure to JobParameter Description.
l Added the pipeline and authentication configurations of theElasticsearch data source to the following sections:– Link to Elasticsearch– To Elasticsearch/Cloud Search Service
2018-05-04 This is the seventh official release.l Added the following sections:
– Stopping a Cluster– Starting a Cluster– Link to CloudTable– To DDS
l Modified the following sections:– Creating a Link– Link to a Relational Database– Link to OBS– Link to OSS on Alibaba Cloud– Link to HDFS– Link to HBase– Source Job Parameters– Destination Job Parameters
l Changed Elasticsearch Service of HUAWEI CLOUD to CloudSearch Service.
l Changed Unlimited Query Service (UQuery) of HUAWEICLOUD to Data Lake Insight (DLI).
Cloud Data MigrationAPI Reference A Change History
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
146
Release Date What's New
2018-04-09 This is the sixth official release.l Added the following sections:
– Link to OSS on Alibaba Cloud– Link to DLI– To DLI
l Modified the following sections:– From OBS/OSS– From HBase/CloudTable– To OBS
2018-01-31 This is the fifth official release.l Deleted the type parameter in the configs field.l Updated the description of several link parameters and job
parameters.l Added a sample JSON file of each connector.l Added a sample JSON file of each type of data source.l Consolidated sections "Cluster Management", "Link
Management", and "Job Management" in section APIDescription.
l Added the GaussDB connector in section Link to aRelational Database.
l Added the following sections:– Before You Start– API Overview– API Use Case– Common Parameters
l Deleted the following sections:– Common Message Headers– AK/SK Authentication– Common Return Values– Common Exception Response
Cloud Data MigrationAPI Reference A Change History
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
147
Release Date What's New
2018-01-11 This is the fourth official release.l Added the following sections:
– Link to HDFS– Link to CloudTable– Link to Kafka
l Deleted the description of the datastore parameter andmodified the description of the instances parameter in sectionCreating a Cluster.
l Deleted the description of the datastore parameter and addedthe description of the version parameter in section Queryingthe Cluster List.
l Deleted the description of the datastore parameter and addedthe description of the version parameter in section QueryingCluster Details.
l Modified the value of thefromJobConfig.disConsumerStrategy parameter in sectionFrom DIS.
l Modified the content in section Creating a Job.l Deleted the following sections:
– Querying Versions– Querying Flavors– Create MRS Data Source– Querying MRS Data Source– Deleting MRS Data Source– Public Configuration Data Structures
2017-11-30 This is the third official release.l Added the NAS, DIS, and Elasticsearch connectors.l Added the NAS and DIS data sources to the job parameter
description of source links.l Added the NAS and Elasticsearch data sources to the job
parameter description of destination links.
2017-10-31 This is the second official release.l Replaced projectId with project_id in this document.l Replaced datastoreId with datastore_id in this document.l Replaced clusterId with cluster_id in this document.l Added and modified the URI format and parameters in several
sections.l Added section "MongoDB Link".l Added section "Common Exception Response".
2017-9-30 This is the first official release.
Cloud Data MigrationAPI Reference A Change History
Issue 10 (2018-08-03) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.
148