Custom Connectors

REST Connectors

REST (Representational State Transfer) is an architectural style that requires web applications to support the HTTP GET, POST, PUT, and DELETE methods and to use a consistent, application-independent interface.Let us understand the following terms related to REST connectors.

Endpoint URL

The endpoint of an API is an unique URL, which represents an object or collection of objects. The endpoint is a reference to a URI that accepts web requests. It is the login endpoint URL to initiate communication with the SaaS provider. To get the endpoint, go through the SaaS provider documentation available on the internet. For example, http://dummy.restapiexample.com/api/v1/ is the REST API endpoint.

Authentication Type

Every back end provides its own Authentication mechanism to provide authorized access to its APIs. You need to get the authentication details from the SaaS provider documentation. For example, for Twitter, go to https://apps.twitter.com and then get the credentials. For Twitter, the authentication is OAuth V1.0a, which you can get from https://apps.twitter.com.

Methods

Methods are tasks that act on a Resource. You must create at least one Method for a Resource after you have created the Resource. You can add a Request Parameter, Request and Response Headers, and a Request and Response body to a Method. The primary or most-commonly-used HTTP verbs (or methods, as they are properly called) are POST, GET, PUT, DELETE, and PATCH. These correspond to create, read/retrieve, update, and delete (or CRUD) operations, respectively. You use the following HTTP methods to map the CRUD operations to HTTP requests.I n a REST request, the resource that you are working with is specified in the URL - Uniform Resource Locator. The URL is a special case of the URI - Uniform Resource Identifier.

• GET - Reads or retrieves a representation of a resource. For example, GET <endpoint>/getEmployeeList/2 retrieves an employee list with an ID of 2.
• POST - Creates a resource. For example, POST <endpointurl>/getEmployeeList creates a new address.
• PUT - Updates an existing resource. For example, PUT <endpointurl>/getEmployeeList/3 modifies the address with an ID of 3.
• DELETE - Deletes a resource identified by an URI. For example, DELETE <endpointurl>/getEmployeeList/4 deletes an address with an ID of 4.
• PATCH - Partially updates a resource identified by an URI.

Headers and Parameters

REST is not a standard in itself but instead makes use of the HTTP standard. HTTP headers allow the client and the server to pass additional information with the request or the response. For example, the Accept and Content-Type HTTP headers can be used to describe the content being sent or requested within an HTTP request. The client may set Accept to application/json if it is requesting a response in JSON or application/xml if it is requesting a response in XML, that is, when sending data, setting the Content-Type to application/xml tells the client that the data being sent in the request is XML.

REST calls (requests) and responses are sent over the HTTP protocol, hence REST requests are in the form of URLs that point to the resource(s) on the server. Required parameters are attached to the end of the URL. For example, in the resource URL http://<name>.com/user/789, user is the resource and 789 is the parameter that is passed to the URL of the resource. You can use any REST client to make REST calls.

REST parameters specify the variable parts of your resources, that is, the data that you are working with. QUERY parameters are the most common type of parameters, which is appended to the path of the URI when submitting a request. For example, https://api.twitter.com/1.1/users/show.json?screen_name=twitterdev is an example of a QUERY parameter URI where screen_name is the name of the parameter and twitterdev is the value of the parameter.

HTTP Status Codes

HTTP Status Codes indicate the status of the HTTP response:
- 1XX - Informational
- 2XX - Success
- 3XX - Redirection
- 4XX - Client error
- 5XX - Server error

Creating REST Connectors

Let us define a REST Resources and Methods and create custom REST connectors. Further, each connector uses an account to connect to the provider’s back end and perform operations. Each connector comes with a predefined set of operations.

Let’s understand how to:

• Create a REST connector
• Configure an account for a REST connector
• Configure an operation for a REST connector

To create a REST connector

1. From the IBM webMethods Integration navigation bar, click Projects.
2. Select a project and then click Connectors > REST > Add Connector.

3. In the Define Connector page, complete the following fields.

   Note

   Required fields are marked with an asterisk on the screen.
   • Name: Type a name for the REST connector. For example, ExternalRESTapi.
   • Description: Type an optional description for the REST connector. The description you enter here will appear on the Connectors page.
   • Authentication Type: Every back end provides its own authentication mechanism. Get the authentication details from the back end documentation and select the supported authentication type from the drop-down list. REST connector supports the following authentication types:
     
     • Credentials
       
     • OAuth V1.0a
       
     • OAuth V2.0
       
     • OAuth V2.0 (JWT Flow)
       
     • AWS S3 Signature
       
     • AWS V2 Signature
       
     • AWS V4 Signature
       
     • OAuth V2.0 (JWT Client Assertion Flow)
       
       Note

       • Currently, IBM webMethods Integration supports the OAuth V2.0 (JWT Client Assertion Flow) authentication method exclusively for Microsoft endpoints. Know more about the OAuth V2.0 (JWT Client Assertion Flow) authentication method.
         
         
       • When you select Authentication Type as Credentials, the system displays a new configurable field, Preemptive Auth, during account creation, with a default value of true. You can change the value of this field as per your specific use case.
         
       
       

   • Default Endpoint URL: Specify the endpoint URL for the connector. It is the login endpoint URL to initiate communication with the SaaS provider. For example, http://dummy.restapiexample.com/api/v1

     Note

     If you are using the OAuth V2.0 (JWT Client Assertion Flow) authentication method, ensure that the default endpoint URL for the REST connector is exclusively for Microsoft endpoints. For example, https://graph.microsoft.com/

   • Connector Icon: Click Browse and select another icon for the REST connector, if necessary.

   

4. Click Next. The Resources page appears.

   

5. Click Add Resource.

   On the Add Resource page, complete the following fields:

   

   • Name: Type the resource name. For example, getEmployeeList.

   • Path: Type the path for the Resource. The Resource path is relative to the endpoint specified. Each REST Resource derives its path from the namespace of the REST Resource. For example, if the REST Resource is named getEmployeeList, the path is /employees.
     You can define dynamic parameters in the resource path by enclosing each parameter within { } brackets. For example, to get the employee data corresponding to a dynamic parameter called employeeID, specify the source path as /employee/{employeeID}. To get the item information from a particular department in a store, specify the resource path as /store/{departmentID}/{itemID}.
     

   • Methods: Every Resource must have a method associated with it. Select an HTTP Method, for example, GET for get actions.

     Note

     • While adding Methods, if the Resource path contains { } brackets, for example, /user/{userID}, you must add a request parameter having the same name, that is, userID, and set the Parameter Type to URI_CONTEXT.
       
     • When a resource’s URL is updated, the associated operations must also be edited and saved to ensure the new endpoint is utilized. Operations that are not updated may continue to use the old URL even after a resource has been modified.

   The next screen that you see will have the specified path and selected HTTP Method. You can optionally provide a short description for this resource and method.

6. On the Get Method page, complete the following fields:

   

   In the Request parameter section, you can set parameters that become part of the outgoing request. Parameters specify the variable parts of your resource. Click Add Parameters to add a parameter to the request. Complete the following fields:

   • Name - Type the parameter name, for example, param1.
   • Value - Type a value for the parameter, for example, 23.

   • Parameter Type - Select the parameter’s type, for example, CFG_PARAM.
     

     The Parameter Type determines how the parameter should be used. If you select an AWS authentication type, then you must add a mandatory request parameter in all the Actions you create. The parameter name must be aws.service and the parameter type must be CFG_PARAM. Type the service name in the endpoint URL as the parameter value. For example, if the endpoint URL is https://<instance>.s3.com/, type the parameter value as s3.
     

     REST services rely on HTTP methods (GET, POST, PUT, DELETE, and PATCH) to make requests to a SaaS provider. Thus the parameters are closely tied to these HTTP methods, as they are sent as part of these HTTP method requests. The parameters are part of the HTTP URI.
     

     CFG_PARAM is an internal configuration parameter.
     

     URI_CONTEXT parameters are passed as the path component of a REST Resource URI, and the parameter names correspond to the URI path variable names specified in the {} annotation. For example, if the URI is https://api.twitter.com/1.1/users/{id}, the Resource path will be /users/{id}, the parameter type will be uriContext, the parameter name will be id, and the value could be the user id, for example, either 1, or 2, or 3.
     

     QUERYSTRING_PARAM parameters are passed as the query component of a REST resource invocation request. For example, if the URI is https://api.twitter.com/1.1/users/show.json?screen_name=twitterdev, the resource path will be /users/show.json, screen_name is the name of the parameter, twitterdev is the value of the parameter, and the parameter type is query.

     FORM_ENCODED_PARAM - Define a parameter of type FORM_ENCODED_PARAM, if you want to send the parameter as part of the Request body. FORM_ENCODED_PARAM allows you to send simple key value parameters embedded in the Request body for POST or PUT requests. This uses the default web form encoding, which is application/x-www-form-urlencoded.

   Note

   For passing parameters of FORM_ENCODED_PARAM type, you will not be able to define the Request body for a Resource, as the generated parameter key value string will be automatically embedded in the Request body.

   • Mandatory - Select this option if you want this parameter to be made mandatory while creating a flow service.

     

   In the request Headers section, allow the client and the server to pass additional information with the request or the response.

   Note

   Do not add an authorization header if you use credentials as the mode of authentication.

7. Click Add Header to add a request HTTP header. In the Add Header panel, complete the following fields:

   

   • Name - Type the Header name.
   • Value - Type a value for the Header.
   • Mandatory - Select this option if you want this Header to be made mandatory while creating a flow service.
     In the Body panel, complete the following fields:

   

   • Content Type: If the documentation of the SaaS provider specifies that the content type of the request body is JSON, select application/json as the content type. If the documentation of the SaaS provider specifies that the content type of the request body is XML, select application/xml as the content type. If the documentation of the SaaS provider specifies that the content type of the request body is binary, select Binary as the content type. These options allow you to control the content in an HTTP request body.
   • Document Type - Select a document type for the request body. If you do not have one, you can create a new document type. Click and the Add new Document type page appears.
     

   

   • Provide a name and description of your document type.
   • Click Load XML to generate a document type from the XML structure or click Load JSON to generate a document type from the JSON structure.
   • Provide the Name and Type of the field in order to define the structure and content of the document type.
     A field can be a String, Document, Document Reference, Object, Boolean, Double, Float, Integer, Long, or Short. If you select Document Reference, select the Document Reference. Fields are used to declare the expected content and structure of signatures, document contents, and pipeline contents. In addition to specifying the name and type of a field, and whether the type is an Array, you can set properties that specify an XML Namespace.
   • Indicate whether the field is required at runtime by selecting the Required option.
   • Select the Content Type you can apply to String, String list, or String table variables.
   • After you have entered the details and constraints for each field, click Save.
     

   

   Note

   Document Types created for a REST connector appears in the Document Types drop-down list for the selected REST connector.

   If the request payload is an array of object, then first create a document type resembling the content of the object, and then select the Array option.

8. Click Next.
   You will be redirected to the GET Response page.

   

   You can optionally add more response statuses by clicking the Add Header option. In the Response Body panel, it is mandatory to have one action defined. Click Add body to add one.

   

   In the Response Body pane, complete the following fields:

   • HTTP Code: Type a single HTTP status code or a code range to indicate the status of the response. Valid values are 100, 101, 102…599 or a range from 100-599.
   • Content Type: If the documentation of the SaaS provider specifies that the content type of the response body is JSON, select application/json as the content type. If the documentation of the SaaS provider specifies that the content type of the response body is XML, select application/xml as the content type. If the documentation of the SaaS provider specifies that the content type of the response body is binary, select Binary as the content type. These options allow you to control the content in an HTTP response body.
   • Error Response: Select this option to handle HTTP errors. If this option is selected when creating a resource, then during the execution of an operation in a Workflow or Flow, if you receive a corresponding Status Code as response from the back end, then an exception will be thrown. For example, let us say, a resource is created with status HTTPs code as 300-500 identified as error response. Then during execution, if we receive any response from 300-500, the operation execution will be considered as failure.

   

   • Document Type: Select a Document Type for the request body . If you do not have one, you can create a new document type. Click and the Add new Document type page appears.
   • Array: If the response payload is an array of object, then first create a document type resembling the content of the object, and then select the Array option.
     

9. Click Next.
   The Add Resource page with all configured details appears.

   

10. Click Save to create the REST connector.

    

    The new REST connector (ExternalRESTapi) appears in the REST Connectors page. Click the drop-down arrow beside the connector name to view the configured accounts. The Used in column displays the Workflows and Flow services where the connector is used.

    Once you have created the REST connector, the next step is to configure an account and an operation for the connector. You can create an account for a connector from the following pages:

    • On the REST Connectors page, point to a connector row, click on the New Account option, and specify the configuration parameters.
    • While creating or editing a Flow service, select an account, choose an action, and then click the Configure Accounts option to create an acount inline.
    • While creating or editing a Workflow, drop a connector to the canvas, click the Settings icon, select an action, and then in the Connect to field, click the + option to specify the configuration parameters for the connector.
      

Understanding OAuth V2.0 (JWT Client Assertion Flow)

OAuth 2.0 (JWT Client Assertion Flow) is an authentication method used for connecting to REST endpoints in a secure way. This method enables you to generate access tokens for backends that utilize OAuth 2.0 by allowing clients to create digitally signed JWT assertions.

Adding custom fields

With the OAuth 2.0 (JWT Client Assertion Flow) authentication method, you can include custom fields in the Add Account window when setting up an account for a REST connector.

1. When you select the OAuth 2.0 (JWT Client Assertion Flow) method from the Authentication Type drop-down list, the Add Field button appears on the screen. Click on this button to configure the REST connector using this authentication method.

   

   The Add button configuration window appears on the screen. This window allows you to define custom fields that are displayed in the Add Account configuration window when setting up an account for the REST connector.

   

2. Fill in values for the required fields as per the following instructions:

   • Select Field Category: Choose a field category from the following options:

     • Header - Header is typically the first part of the token. It serves as a container for metadata that provides information about how the token is secured and should be processed.
       

       • Currently, IBM webMethods Integration supports this authentication method for Microsoft endpoints only. So, to ensure the successful creation of a REST connector, it is essential to include an ‘x5t’ header value. x5t is a base64-encoded SHA-1 thumbprint associated with the X.509 certificate.
       • When you enter ‘x5t’ as the header value in the ‘Name’ field, a custom field appears in the ‘Add Account’ configuration window with a ‘+’ button. You can use this button to select an X.509 certificate from your local folder. After uploading the X.509 certificate, IBM webMethods Integration automatically calculates the ‘x5t’ value, which is then displayed in the field.

     • Claim - Claim is an essential part in JWTs, providing context and additional information about the token.
       Currently, IBM webMethods Integration supports this authentication method for Microsoft endpoints only. So, to ensure the successful creation of a REST connector, it is essential to include an ‘exp’ claim value. exp or expiration time is the time (in minutes) after which the JWT expires.

     • JWT Client Assertion - A JWT client assertion is a specific type of JWT used in OAuth 2.0 authentication. It serves as a means for a client (an application or service) to prove its identity when requesting access to protected resources from an authorization server.
       Currently, IBM webMethods Integration supports this authentication method for Microsoft endpoints only. So, to ensure the successful creation of a REST connector, it is essential to include ‘client_id’ and ‘scope’ assertion values. client_id is an unique identifier associated with the application while scope is the resource identifier (application ID URI) of the resource you want, affixed with the .default suffix. For example, https://graph.microsoft.com/.default.

   • Name: Specify the name for the selected field category.
     The value for this field is case-sensitive. For example, for Microsoft endpoints, it is mandatory to include ‘X5T,’ ‘expiration time,’ ‘client ID,’ and ‘scope.’ The names for these values should be ‘x5t,’ ‘exp,’ ‘client_id,’ and ‘scope,’ respectively.

   • Display Name: Provide a suitable display name for this field. For example, Client ID.

   • Description: Provide a short summary for this field.

   • Select Data Type: Select the data type you want to assign to the value of this field. Available values are String, Integer, anyURI, Boolean.

   • Default Value: Specify the default value you want to set for this field.

   • Allowed Values: Define the values you want to set for this field.

   • Field Properties: Enable the specific property you want to set for this field. Available properties include Array, Required, Encrypted, Hidden.

3. Once you have filled in all the required details, click Add. This action successfully creates and adds the custom field to the list of custom fields in the Define Connector screen. The created custom field then appears on the Add Account configuration window when you are setting up an account for a REST connector.

To configure an account for the REST connector

1. Go to the REST Connectors page and click New Account.

   

   The Add Account configuration window appears.

2. Add an account for the REST connector by filling out the required fields.

   Note

   If you have added custom fields using the OAuth 2.0 (JWT Client Assertion Flow) authentication method, they will be visible in the Add Account configuration window.
   

3. Provide an Account Name.
   All fields will be populated with values. You can add or modify the values if required.

4. Provide a Server URL.
   This is the login endpoint URL you have specified in the Define Connector page while creating the REST connector.

5. Provide a Consumer ID
   Consumer ID is also referred to as the Client ID in OAuth 2.0. This is a client identifier issued to the client to identify itself to the authorization server. For OAuth 2.0, it is the Consumer Key issued by the service provider and used by the consumer to identify itself to the service provider.

6. Provide an Access Token
   Access Token is used for authentication and is issued by the authorization server. For OAuth 2.0, it is a value used by the access token consumer to gain access to the protected resources on behalf of the User, instead of using the User’s Service Provider credentials.

7. Provide a Session Timeout value
   Session timeout value is the maximum number of minutes a session can remain active, in other words, how long you want the server to wait before session timeout (min) terminating a session. The value should be equal to the session timeout value specified at the SaaS provider back end.

8. Provide a Response Timeout
   The number of milliseconds IBM webMethods Integration waits for a response before cancelling its attempt to connect to the back end. In case the response timeout network is slow, or the back-end processing takes longer than usual, increase the response timeout value. It is recommended to specify a value other than 0. If you specify 0, IBM webMethods Integration will wait indefinitely for a response.

9. Provide Retry Count on Response Failure
   The number of times IBM webMethods Integration attempts to connect to the back end to read a response if the initial attempt fails. If an I/O error occurs, it will retry only if you have selected the Retry on Response Failure option.

10. Select Retry on Response Failure
    Select whether IBM webMethods Integration should attempt to resend the request when the response has failed, even though the request was sent successfully. If so, select true to re-establish the connection.

11. Select Trust store Alias
    Select the alias name of the IBM webMethods Integration trust store configuration. The trust store contains trusted certificates used to determine trust for the remote server peer certificates. You can also add a new truststore certificate from this field by clicking the icon.

12. Select Keystore Alias
    Select the alias for the IBM webMethods Integration key store configuration. This is a text identifier for the keystore alias. A keystore file contains the credentials (private key/signed certificate) that a client needs for authentication. You can also add a new keystore certificate from this field by clicking the icon. This option is available only if ‘Credentials’ is selected as the authentication type while creating the connector.

13. Select Client Key Alias
    This is the alias to the private key in the keystore file specified in the Keystore Alias field. The outbound connections use this key to send client credentials to a remote server. To send the client’s identity to a remote server, you must specify values in both the Keystore Alias and the Client Key Alias fields. This option is available only if ‘Credentials’ is selected as the authentication type while creating the connector.

14. Provide Keep Alive Interval
    To handle stale connections, set the Keep Alive Interval in minutes.

15. Provide Idle Timeout
    The idle timeout interval in milliseconds defines the interval for which a connection will be kept alive if it’s idle Timeout not in use. A value > 0 keeps the connection alive for the specified value. The default value of -1 implies that the connection will be kept alive until a request fails due to a connection error.

16. Select a value for Enable SNI
    Server Name Indication (SNI) is an extension to the TLS protocol by which a client indicates which host name it is attempting to connect to at the start of the handshaking process. Enable this option if the SaaS provider offers SNI-based TLS connectivity, and if you want to connect to an SNI enabled SAAS provider to send the host name specified in the Server URL field, as part of the TLS SNI Extension server_name parameter.

17. Provide SNI Server Name
    If you want to explicitly specify a host name to be included as a part of the SNI extension server_name parameter, in case the host name is other than the host name specified in the Server URL field, specify the host name value in the SNI Server Name field.

18. Provide Consumer Secret
    Also referred to as the Client Secret, this is a secret matching to the client identifier. For OAuth v2.0, it is the secret used by the Consumer to establish ownership of the Consumer Key.

19. Provide Instance URL
    This is an optional field and is used to specify a runtime host, if applicable. This may be required in some back ends like Salesforce.

20. Select Refresh Access Token
    Issued with the OAuth v2.0 access token only. A token used by the client to obtain a new access token without having to involve the resource owner.

21. Provide Refresh Token
    A token used by the client to obtain a new access token without involving the resource owner.

22. Provide a Refresh URL
    This is the provider specific URL to refresh an Access Token.

23. Select Refresh URL Request
    This field is available only if authentication type OAuth v2.0 is selected. OAuth 2.0 access tokens typically have a very short lifetime, so when an access token expires, the OAuth profile does not automatically refresh the expired access token. Select the Body Query String and URL Query String options if you want an expired access token to be refreshed automatically. Use the Refresh Token Callback Service option if the back end requires some refresh requests in a custom format, for example, requests which need more parameters than the ones specified by OAuth v2.0, or the back end uses some custom way of organizing parameters, or expects some other HTTP method request (other than POST). If you select this option, you must specify the Flow service name in the Refresh Token Callback Service field.

24. Provide the Refresh Token Callback Service
    This is required when the Refresh URL Request is specified as Refresh Token Callback Service. Use this option to specify a new Flow service that the back end requires when adding an account. Click the icon to add the new Flow service to your account. Once you add a Flow service to your account, a framework of the new Flow service is created. You can view the link of the same Flow service on the REST Connectors page.
    

    

    Click the FlowService_Name link in the REST Connectors page to add any implementation for the Flow service. You will then be redirected to the Flow services page for the implementation.

25. Select Enable Connection Pooling
    Select this option if you want to enable connection pooling for a connection. When you enable connection pooling, IBM webMethods Integration creates the number of connection instances you specified in the connection’s Minimum Pool Size field. Whenever an Integration needs a connection, IBM webMethods Integration provides a connection from the pool. If no connections are available in the pool, and the maximum pool size has not been reached, IBM webMethods Integration creates one or more new connections (according to the number specified in the Pool Increment Size field) and adds them to the connection pool. If the pool is full (as specified in the Maximum Pool Size field), the requesting service will wait for IBM webMethods Integration to obtain a connection till one sec, until a connection becomes available. Periodically, IBM webMethods Integration inspects the pool and removes inactive connections that have exceeded the expiration period of one sec.

26. Provide Minimum Pool Size
    The minimum number of connection objects that remain in the connection pool at all times, if connection pooling is minimum pool size enabled. When the connector creates the pool, it creates this number of connections.

27. Provide Maximum Pool Size
    The maximum number of connection objects that can exist in the connection pool if connection pooling is enabled. When the connection pool has reached its maximum number of connections, the connector will reuse any inactive connections in the pool, or, if all connections are active, it will wait for a connection to become available.

28. Provide Pool Increment Size
    The number of connections by which the pool will be incremented, up to the maximum pool size, if connection pooling is enabled and connections are needed.

29. Provide Block Timeout (msec)
    The number of milliseconds that IBM webMethods Integration will wait to obtain a connection with the SAAS provider before the connection times out and returns an error. For example, you have a pool with maximum pool size of 20. If you receive 30 simultaneous requests for a connection, 10 requests will be waiting for a connection from the pool. If you set the Block Timeout to 5000, the 10 requests will wait for a connection for 5 seconds before they time out and return an error. If the services using the connections require 10 seconds to complete and return connections to the pool, the pending requests will fail and return an error message stating that no connections are available. If you set the Block Timeout value too high, you may encounter problems during error conditions. If a request contains errors that delay the response, other requests will not be sent. This setting should be tuned in conjunction with the maximum pool size to accommodate such bursts in processing. The default value is 1000 msec

30. Provide Expire Timeout (msec)
    The number of milliseconds that an inactive connection can remain in the pool before it is closed and removed from the pool, if connection pooling is enabled. The connection pool will remove inactive connections until the number of connections in the pool is equal to the Initial Pool Size. The inactivity timer for a connection is reset, when the connection is used by the connector. This setting should be tuned in conjunction with the initial pool size to avoid excessive opening and closing of connections during normal processing. The general recommendation is to keep the Expire Timeout value equal to the Session Timeout value. The default value is 1000 msec

31. Select a Session Management value
    The access token is refreshed whenever the session expires. Session expiration is handled according to the setting of the Session Management property in your configuration. To refresh the ‘Access Token’ automatically, set the Session Management to either fixed or auto. The Timeout value should be based on the back-end settings.

    If Session Management is set to none, then access token will not be refreshed.

32. Click Add.
    With this, you have successfully configured the REST connector account. Click the drop-down arrow beside the connector to view the account configured for the connector.

    

    Note

    The accounts created for REST connectors using the Flow Editor do not display the Edit and Delete icons when the same Accounts are displayed on the Connectors page in IBM webMethods Integration. You can use the existing accounts configured using the Flow Editor, but you cannot edit or modify the account details. Ensure that you create a new account for the REST connectors if any changes are required for the existing accounts. Further, the accounts created for REST connectors using the Flow Editor cannot be used in Workflows, that is, pre-existing accounts will not be available in the Connect to drop-down list while creating a Workflow. Ensure that you create new accounts in such cases.

    Once you have configured the account for the REST connector, the next step is to configure an operation for the connector.

To configure an operation for the REST connector

1. Go to the REST Connectors page and click Operations. The Custom Operations page appears.

   

2. On the Custom Operations page, click New Operation.
   The Show i/o signature and Test operation options are available on the Custom Operations screen and appears for all custom connector operations.
   The Show i/o signature option allows you to only view the Input and Output signature of the custom operation. The input and output fields cannot be edited. You can click the input and output fields to view the field properties.
   The Test operation option allows you to test the custom operation. Specify the Account name and the Input data and then click Run to test the operation and view the test results in the Result window. For array inputs, you can add the array items and then add the values. If an operation does not have an input signature, the input fields are not displayed.

   

   On the Connect to account page, select an account that you configured for the connector. Each connector uses an account to connect to the provider’s back end and perform operations. You can also create new account by clicking the + icon.

3. Provide a name and description of the custom operation.

   

4. Click Next.

5. Select the operation.

   

6. Confirm the operation and click Done.

With this you have created a REST connector, configured an account and operation. The connector is now ready for the execution of services.

Configuring Custom Refresh for Access Tokens for Accounts when using OAuth 2.0 Authentication

Summary

OAuth 2.0 access tokens typically have a very limited lifetime. To enable IBM webMethods Integration to refresh a configured access token, you must configure Refresh Access Token to true and Session Management to fixed or auto.

To use the Out-of-Box refresh implementations offered by IBM webMethods Integration, configure Refresh URL Request as “URL Query String” or “Body Query String” along with appropriate values for the fields Consumer ID, Consumer Secret, Refresh Token, Refresh URL and Access Token. This option can be used if the API provider adheres to the standard implementation of the Refresh grant type as described by the OAuth 2.0 Authorization Framework specification.

The following example provides you information on how to implement and configure a custom refresh mechanism for Accounts using IBM webMethods Integration.

Before you begin

• When using the REST connector, if you are using Google API services, then you need to have a working Google account to log into the Google Cloud console to obtain the access token for OAuth 2.0 authorization.
• You need to have access permission for Google APIs.
• When using the endpoint target (Server URL) to initiate communication with the SaaS provider, for example, Twitter, the end point is https://api.twitter.com/1. To get the endpoint, go through the SaaS connector documentation available on the internet.
• When using Custom Connector, every back end provides its own authentication mechanism. Get the authentication details from the backend documentation.
• You need to have a relevant Google connector app in Google Cloud Platform to retrieve its Client ID and Client Secret if using any Google APIs.

Basic Flow

Before you try to configure a refresh token callback for a REST connector, you must have the following configurations (Step 1 - Step 3) completed on the Accounts configuration page:

1. Get the endpoint target (Server URL) for the connection configuration. For example, https://api.twitter.com/1.1.

   

2. Get the Consumer ID and Consumer Secret issued to the client to identify itself to the authorization server.
   Consumer Secret is a secret matching to the client identifier.

   

3. Configure the Session Management parameters to determine at what intervals the access token would be considered for refresh.

   The access token is refreshed during the next immediate execution after the determined interval. Session expiration is handled according to the setting of the Session Management property in your configuration. To refresh the ‘Access Token’ automatically, set the Session Management to either fixed or auto. The Timeout value should be based on the back-end settings.

   Note

   If Session Management is set to none, then access token will not be refreshed.

   

4. Select Refresh URL Request field as Refresh token callback service to configure a custom refresh implementation. This field is available only if the OAuth v2.0 authentication type is selected.

   

   When you select this option, you must specify the Flow service name in the Refresh token callback service field. This is a service which needs to be implemented by a user. The Refresh token callback service option allows you to create a Flow service, which will be executed when the access token has expired or is not valid. You can view the link of the same Flow service on the REST Connectors page.

5. Click the custom callback Flow service name in the REST Connectors page to create a new Flow service

   

   You must add the implementation for refreshing the access token in the Callback Flow service.

   The Flow service adhere to the specification with the input and output parameters shown below:

   These are the inputs which would be provided to the Custom callback Flow service implementation by IBM webMethods Integration. Values will correspond to the current values of these fields in the account configuration

   

   These are the output expected by IBM webMethods Integration. The values returned for these fields will determine the inputs for the subsequent API and refresh execution.

   

   • access_token - The access token issued by the authorization server, applicable for subsequent API and refresh executions.
   • token_type - The type of the token issued. For example, bearer.
   • expires_in - The lifetime in seconds of the access token. For example, the value 3600 denotes that the access token will expire in one hour from the time the response was generated. Relevant for auto session management where expiry interval returned by API provider is given precedence.
   • refresh_token - The refresh token to be used for subsequent refresh execution.
   • scope - (Optional) If identical to the scope requested by the client; else, REQUIRED
   • errorResponse - The details of error in refresh execution.

The newly created Flow service will generate an access token that is mapped to the access token field in the output signature.

Invoking a REST API in a Flow service using a REST Connector

In the earlier section, we created a REST connector and configured the REST connector account. Now let us understand how to invoke a REST API in a Flow service using a REST connector.

1. From the IBM webMethods Integration navigation bar, click Projects. Select a project and then select Flow services. Click the + icon to start creating a new Flow service.

2. Provide a name and description of the new Flow service. For example, ExternalRESTAPIFlow

   

3. On the Flow service, click CONNECTORS.

   

4. Type ExternalRESTapi in the search box, select the connector, and then select Add Custom Operation.

   

5. On the Connect to account page, select the connection name and provide a name and description of the connection.

   

6. Click Next and select the getgetEmployeeList operation.

   

7. Confirm the action to create the custom operation.

   

8. On the Flow service editor, you will see that the account is now configured.

   

9. Click to view the Flow service mapping.

   

10. Save the Flow service and click Run. As the results show, the employee details are fetched.

    

Using Dynamic Accounts in Custom Actions

The Dynamic Account feature enables you to dynamically override certain details of the account you are using to run a custom action created under a custom REST connector.

Enabling dynamic accounts

Perform the following steps to enable dynamic account for a custom action created under a REST connector:

1. Drag-and-drop a custom REST connector on canvas from the Connectors panel, double-click it, and click the + button to create a new custom action.

   

   Note

   If you want to enable dynamic account for an existing custom action, search the relevant action in the Select action drop-down list, and click the Edit icon.
   

   You are redirected to the Add Custom Action screen.

2. Add/select the account you want to use to run the custom action, provide a name for the custom action, toggle-on the Enable Dynamic Account switch, and click Next.

   

   On the Dynamic Account Configuration screen that appears, you can see a list of configuration fields and their default values for the selected account.

3. Select the fields for which you want to specify values at run time and then click Next.

   Note

   All mandatory account configuration fields are selected by default and cannot be deselected.

   

4. Configure the operation for your custom action as explained in the Creating Custom Actions section and then click Next.

   

5. Verify the details of your custom action and then click Done.

   

   You are redirected to the Select action screen.

   

   With this, the dynamic account feature is enabled for your selected custom action.

Using dynamic accounts

Once the dynamic account feature is enabled for a custom action, you can anytime use it as per your requirements.

To use a dynamic account, perform the following steps:

1. Drag-and-drop the relevant Custom REST connector from the Connectors panel on canvas, select the required dynamic account-enabled custom action from the Select action drop-down list, select the account you want to use to run the action, and click Next.

   

   In the action configuration screen that appears, you can observe a $connection block. This block contains all the fields selected at the time of enabling the dynamic connection along with the mandatory fields.

2. Provide values for the relevant fields listed under the $connection block and then click Next.

   The values you provide for these fields override the default values of your selected account.

   Note

   Only encrypted fields (such as, Password, Client Secret) are mandatory under the $connection block. You can optionally keep the rest of the fields under the $connection block blank, in which case, IBM webMethods Integration uses the default field values provided at the time of creating the selected account.

   

3. Test the action to see if it is working as expected and then click Done.

   

   This takes you back to the canvas.

   Now whenever you connect this action to any workflow and run it, IBM webMethods Integration automatically overrides the default values with dynamic values while creating the connection. This connection with the modified values is then used to run the custom action.

SOAP Connectors

Custom SOAP connectors enable you to access third party web services hosted in the cloud or on-premises environment. You can also invoke a SOAP API in a Flow service by using a SOAP Connector.

The following features are supported for SOAP Connectors:

• A SOAP connector implementation follows the WS-I Basic Profile 1.1 specification.
• SOAP connectors can be created by uploading a WSDL file or by using a valid WSDL URL that can be accessed over a network.
• SOAP connectors can be created with WSDLs that are annotated with WS-Security Policy/Policies.
• SOAP connectors with SOAP version 1.1 and 1.2 and Style/Use as Document/Literal and RPC/Literal (RPC/Encoded model is not supported for SOAP version 1.2).

The following SOAP Binding types are supported:

• SOAP over HTTP
• SOAP over HTTPS
• Authentication type: HTTP Basic Token.

SOAP connectors have the following restrictions:

• The WSDL and associated schema(s) must be accessible through a publicly or locally accessible URL.
• Only WSDLs with WS-Security policies are supported. Any other policies, for example, WS-Addressing, WS-Reliable Messaging, and so on, are not supported. If you create SOAP connectors with WSDLs having non-WS-Security Policies, exceptions may appear while executing Flow services.
• Manual addition of WS-Security Policies in a SOAP Connector is not supported. SOAP connectors with WS-Security can be created with only policy-annotated WSDLs, that is, WSDLs that already have WS-Security Policies annotated in them.
• SOAP over JMS is not supported.
• Only Basic Authentication is supported. Other authentication types such as Digest, NTLM, and Kerberos are not supported.
• You will not be able to attach or upload a file while executing a service.

Creating SOAP Connectors

The SOAP connectors enable you to access third party Web Services hosted in the cloud or on-premises. The SOAP connector uses a WSDL to create consumer operations. Let us understand how to create a SOAP connector and configure an account.

To start creating a connector, go to the project in which you want to create a SOAP connector and click the Connectors tab.

You will be redirected to the Connectors page.

To create a new SOAP connector, click SOAP and then click Add Connector.

You will be redirected to the New SOAP Connector page. You can create a SOAP connector by importing from the specified URL, or by importing from a WSDL file. In our example, let us use the Import from URL option to create a new SOAP Connector.

1. Create a new SOAP Connector from the specified URL

2. Click the Import from URL option. You can see the text field appearing in the same window.

3. Specify the URL of the WSDL. The URL should begin with http:// or https://. The URL is used to retrieve the WSDL for the SOAP API. For example, let us use the WSDL URL generated from SOAP API ( Soap_api_test3) in our earlier guide.

4. On filling in the details, click Next.

   

   The Define Connector page appears.

   

   The next step is to define the connector.

5. In the Define Connector page, complete the following fields:

   • Name: Type a name for the SOAP connector, for example, Soap_connector.
   • Description: Type an optional description for the SOAP connector.
   • Connector Icon: Click Browse next to the Connector Icon if you want to select a different icon for your SOAP connector. The icon must be a PNG file and the size cannot exceed 50 KB, else the default image is displayed.
   • User: Enter the user name if authentication is required to access the WSDL URL.
   • Password: Enter the password if authentication is required to access the WSDL URL.
   • Validate Schema with Xerces: Select this option if you want IBM webMethods Integration to also use the Xerces parser to validate the schemas associated with the XML Schema definition.
     

   

   • Click Save.
   Note

   Required fields are marked with an asterisk on the screen.

   The new SOAP connector appears in the SOAP Connectors page. You can edit and remove a SOAP connector from the SOAP Connectors details page. The Used In column displays the Workflow and Flow service where the connector is used.

   You can create a SOAP connector by using the Import from file option in Create SOAP connector page of the Connectors tab. You will be prompted to choose the Primary WSDL file and add imported WSDLs or XSDs or XML to primary WSDL and then define a connector. The ports that are configured here are listed in the Port Binding drop-down field of the Add Account dialog box while configuring a new account. According to the selected Port Binding value the URL field gets updated with the corresponding link. You can also edit the URL to specify a different web service endpoint.

   

   To edit an existing SOAP connector, click the vertical ellipsis icon in the right corner of your connector, and click Edit. The Define Connector page appears. In the Define Connector page, in the Override Existing WSDL ? section, select No, Keep Existing WSDL if you do not want to modify the WSDL URL or the WSDL file. Select Yes, override WSDL if you want to specify a new WSDL URL or upload a new WSDL file in the WSDL source section.

Updating the WSDL may result in addition or removal of Operations or fields in the Input/Output signature of an Operation. This may lead to incorrect mappings if you have used that Operation in an Workflow or Flow service.

To delete an existing SOAP connector, on the SOAP Connectors page, click the vertical ellipsis icon in the right corner of your connector, and click Remove.

The next step is to define the account configuration details.



To configure an account for the SOAP connector, click the New Account option. The Add Account dialog box appears.

6. In the Add Account dialog box, fill in the following fields.

   

   All fields will be populated with values. You can add or modify the below values if required.

   • Account Name: Type a name for the SOAP connector account. Let us say, for example Soap_connector_1I.
   • Port Binding: Select the bind address from the drop-down list, that is, the concrete protocol and data format specification for the web service.
   • URL: This is the URL for the web service. You can edit the URL to specify a different web service endpoint.
   • Response Timeout: The number of milliseconds IBM webMethods Integration waits for a response before canceling its attempt to connect to the back end. In case the network is slow or the back end processing takes longer than usual, increase the response timeout value. It is recommended to specify a value other than 0. If you specify 0, IBM webMethods Integration will wait indefinitely for a response. If you do not specify a timeout, IBM webMethods Integration will consider 5 minutes as the response timeout. If the connection to the host provider ends before IBM webMethods Integration receives a response, the web service operation ends with an exception and a status code of 408.
   • User: User name used to authenticate the consumer at the HTTP or HTTPS transport level on the host server.
   • Password: The password used to authenticate the consumer on the host server.
   • Keystore Alias: Alias to the keystore that contains the private key used to connect to the Web Service host securely. You can also add a new Keystore from this field.
   • Key Alias: Alias to the key in the keystore that contains the private key used to connect to the Web Service host securely. The key must be in the keystore specified in the Keystore Alias field.
   • WS Security User Name: Name to include with the Username Token, if the Web Service’s security policy requires one.
   • WS Security Password: The password to include with the UsernameToken (must be plain text).
   • WS Security Keystore Alias: The alias for the keystore, which contains private keys and certificates associated with those private keys. You can also add a new Keystore from this field.
   • WS Security Key Alias: The text identifier for the private key associated with the Keystore Alias.
   • WS Security Truststore Alias: The alias for the truststore, which contains the trusted root of a certificate or signing authority (CA). You can also add a new Truststore from this field.
   • WS Security Partner Certificate Alias - The file that contains the partner’s self-signed certificate. You can also add a new Partner Certificate from this field.
   • Timestamp Precision: Whether the timestamp placed in the Timestamp element of the security header of an outbound message is precise to seconds or milliseconds. If the precision is set to milliseconds, the timestamp format yyyy-MM-dd’T’HH:mm:ss:SSS’Z’ is used. If the precision is set to seconds, the timestamp format yyyy-MM-dd’T’HH:mm:ss’Z’ is used.
   • Timestamp TTL: The time-to-live value for an outbound message in seconds. This value is used to set the expiry time in the Timestamp element of outbound messages. The timestamp precision value is used only when WS-Security is implemented through a WS-Policy.
   • Timestamp Max Skew: The maximum number of seconds that the Web Services client and host clocks can differ so that the timestamp expiry validation does not fail. The timestamp precision value is used only when WSSecurity is implemented through a WS-Policy. The inbound SOAP message is validated only if the creation timestamp of the message is less than the sum of the timestamp maximum skew value and the current system clock time.

7. Click Add.
   

   With this, you have successfully configured an account (Soap_connector_act1) for the SOAP connector (Soap_connector). Accounts created for SOAP connectors using the Flow Editor do not display the Edit and Delete icons when the same Accounts are displayed on the Connectors page in IBM webMethods Integration. You can use the existing accounts configured using the Flow Editor, but you cannot edit or modify the account details. Ensure that you create a new account for the SOAP connectors if any changes are required for the existing accounts. Further, the accounts created for SOAP connectors using the Flow Editor cannot be used in Workflows, that is, pre-existing accounts will not be available in the Connect to drop-down list while creating a Workflow. Ensure that you create new accounts in such cases.

Invoking a SOAP API in a Flow service using a SOAP Connector

In the earlier section, we created a SOAP connector (Soap_connector) and configured the SOAP connector account (Soap_connector_act1). Now let us understand how to invoke a SOAP API in a Flow service using a SOAP connector.

1. From the IBM webMethods Integration navigation bar, click Projects.
2. Select a project and then select Flow services. Click the + icon to start creating a new Flow service.

3. Provide a name and description of the new Flow service. For example, ExecuteSOAPAPIFlowservices.

   

4. On the Flow service, select CONNECTORS.

   

5. Type Soap_connector in the search box and select the connector.

   

6. Next, in the Type to choose action field, select an operation.

   

   You cannot add operations in a SOAP connector created from a WSDL. Also, you cannot delete operations from a SOAP connector created from a WSDL.

7. Select the Soap_connector_act1 account. You can also create or configure an account inline.

   

8. Click to view the Flow service mapping page.

   

9. Save the Flow service and click Run. As the results show, the weather details are fetched.

   

On-Premises Connectors

Applications uploaded from the on-premises Integration Server are listed in the Connectors > On-Premises Connectors page in IBM webMethods Integration. You will not be able to create accounts or operations for on-premises connectors as those can be uploaded only from webMethods Integration Server.

Click here for information on how to upload an application.

Invoking Flow services for On-Premises Connector

Summary

In this tutorial, we will see how to invoke Flow services for on-premises connectors.

Before you begin

• Check if the application uploaded from the on-premises Integration Server is listed in the On-Premises Connectors page.

• Check if you have the Developer and Admin roles assigned from the Profile icon > User Management > Roles page.

  Note

  When you publish a project that has a Flow service, and if the Flow service has an on-premises connector, the on-premises connector is not pushed to the destination tenant. Ensure that you configure the same on-premises connector on the destination tenant to make the published Flow service work.

Basic Flow

1. Select a project where you want to create the new Flow service.
2. Click Connectors > On-Premises.

3. You will see the list of connectors uploaded from the on-premises system.

   

4. Let us use the AddintApp2 connector as an example for invoking Flow services.
   Click the drop-down arrow beside the connector to view the accounts configured for that connector. The Used in column displays the Workflows and Flow services where the connector is used.

   

5. Click the Flow services tab and on the Flow services page, click the + icon to create a new Flow service.

   

6. Provide a name, for example, OnPremAddIntsFlowService, and an optional description for the new Flow service.

   

7. Select the AddintApp2 connector in the search box.

   

8. Select the myService operation.

   

9. Select the AddintApp111 account.

   

10. On the Flow service editor, click the icon to define the input and output fields.

    

11. Create two Input Fields, value1 and value2.

    

12. Create Output Fields as well.

    

13. Click the mapping icon to map the input and output fields.

    

14. Click the Pipeline Input fields (value1 and value2) and drag it to the service input (num1 and num2). The service output is automatically mapped to value by a dotted line.

    

15. Save the Flow service and run it by clicking the Run icon. Provide the custom field values in the value1 and value2 fields, for example, 10 and 20 respectively.

    

    

16. Run the Flow service and view the execution results.
    With this you have executed the Flow service (OnPremAddIntsFlowService) for an on-premises connector.

Flat File Connectors

You can use the Flat File connector to translate documents into and from flat file formats. To set up the translation, you create the definition and structure of the Flat File connector, which is called a flat file schema. The schema also contains the instructions for parsing or creating the flat file and defines how to identify individual records within a flat file and what data is contained in each of those records.

What is a flat file definition and structure?

The definition and structure of a flat file connector contains the instructions for parsing or creating a flat file. It details the structure of the document, including delimiters, records, and repeated record structures. It also acts as the model against which you can validate an inbound flat file. A flat file structure consists of hierarchical elements that represent each record, field, and subfield in a flat file.

What are the different approaches to create a flat file connector?

You can create a flat file connector using any one of the following approaches:

• Create manually: In this approach, you define the definition and structure of a flat file connector and then manually add the elements or properties.
• Use a sample file: In this approach, you use a sample file to define the definition and structure of a flat file connector. Here, you use the automated wizards to create the structure of the flat file.

What are the high-level steps to create a flat file connector?

Creating a flat file connector manually

IBM webMethods Integration can process flat files in which:

• The records in the flat file are defined using one of the following methods:

  Property	Description
  Delimiter	Each record in the flat file is separated by a delimiter.
  Fixed length	Each record is a fixed number of bytes (for example, mainframe punch or print records).
  Variable length	Each record is preceded by two bytes that indicate the length of the record. Records in the flat file can have different lengths.

• If the flat file contains record identifiers, they must be in the same location in all records in the file. IBM webMethods Integration can then identify fields in these records based on:

  • Fixed position - Each field is defined by 1) the number of bytes from the beginning of the record and 2) the field length. This can be used regardless of whether a field delimiter has been specified.
  • Delimiters - Each field is separated by a delimiter, and you can specify the Nth delimited field in a record to represent the record identifier. This can be used only when a field delimiter (and, if necessary, subfield delimiter) has been specified.

To create a flat file connector manually

1. From the IBM webMethods Integration navigation bar, click Projects.

2. Select a project and then click Connectors > Flat File > Add Connector.
   

   

   The New Flat File Connector page appears.

3. Select Create manually and click Next.

   

4. In the Define Connector page, type a name and description of the flat file connector.

5. Drag and drop or click Browse files and select a connector icon, if necessary.

6. Click Next.

   

7. In the Flat File Definition page, configure the Parser and the Record Identifier.

8. In the Parser section, to configure a delimited record parser, select Delimiter in the Parser area. Use the Delimiter record parser for the Flat File connector when each record is separated by a delimiter. You can type the input or use the custom dropdown input with predefined list of values to select values for Record, Field or Composite, Sub Field, Quoted Release Character & Release Character fields. For a record delimiter, you can specify a character (for example, !) or character representation (for example, \r\n for carriage return).

   Specify the following fields:

   Delimiter	Record type	Description
   Record	Character	Character that separates records in a flat file document.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the record delimiter for this document is located. For example, if you specify 3 as the character position, you have indicated that the record delimiter appears in the fourth character position from the beginning of the document.

   Delimiter	Field or composite type	Description
   Field or Composite	Character	Character that separates fields in a flat file document.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located. For example, if you specify 4 as the character position, you have indicated that the field delimiter appears in the fifth character position from the beginning of the document.

   Delimiter	Subfield type	Description
   Subfield	Character	Character that separates subfields in a flat file document. The default is a period “.”.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the subfield delimiter for this document is located. For example, if you specify 5 as the character position, you have indicated that the subfield delimiter appears in the sixth character position from the beginning of the document.

   Delimiter	Quoted Release Character type	Description
   Quoted Release Character	Character	Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. For example, your field delimiter is (,) and your release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the Quoted Release Character, the Quoted Release Character will be used.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the quoted release character for this document is located. For example, if you specify 5 as the character position, you have indicated that the quoted release character appears in the sixth character position from the beginning of the document.

   Delimiter	Release character type	Description
   Release Character	Character	Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located. For example, if you specify 5 as the character position, you have indicated that the field delimiter appears in the sixth character position from the beginning of the document.

   

   Next, set the record identifier.

9. Use a Fixed Length record parser type when each record is of a fixed length (for example, mainframe punch or print records). This parser splits a file into records of the same pre-specified length. To configure a fixed length record parser, in the Parser Type area, select Fixed Length.

   Specify the following fields:

   Fixed Length	Description
   Record Length	In the Record Length field, enter the length, in characters, of each record in the flat file. Record length cannot be empty.

   Fixed Length	Field or composite type	Description
   Field or Composite	Character	Character that separates fields or composites in a flat file document.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located. For example, if you specify 4 as the character position, you have indicated that the field delimiter appears in the fifth character position from the beginning of the document.

   Fixed Length	Subfield type	Description
   Subfield	Character	Character that separates subfields in a flat file document.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the subfield delimiter for this document is located. For example, if you specify 5 as the character position, you have indicated that the subfield delimiter appears in the sixth character position from the beginning of the document.

   Fixed length	Quoted release character type	Description
   Quoted Release Character	Character	Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. For example, your field delimiter is (,) and your release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane.When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the and the Quoted Release Character, the Quoted Release Character will be used.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the quoted release character for this document is located. For example, if you specify 5 as the character position, you have indicated that the quoted release character appears in the sixth character position from the beginning of the document.

   Fixed Length	Release Character type	Description
   Release Character	Character	Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
   	–OR–
   	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located. For example, if you specify 5 as the character position, you have indicated that the field delimiter appears in the sixth character position from the beginning of the document.

   

   Next, set the record identifier.

10. The Variable Length record parser type expects each record to be preceded by two bytes that indicate the length of the record. Each record may be a different length.

    To configure a variable length record parser, in the Parser area, select Variable length, and specify the following fields:

    Variable Length	Field or Composite type	Description
    Field or Composite	Character	Character that separates fields or composites in a flat file document.
    	–OR–
    	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located. For example, if you specify 4 as the character position, you have indicated that the field delimiter appears in the fifth character position from the beginning of the document.

    Variable length	Subfield type	Description
    Subfield	Character	Character that separates subfields in a flat file document.
    	–OR–
    	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the subfield delimiter for this document is located. For example, if you specify 5 as the character position, you have indicated that the subfield delimiter appears in the sixth character position from the beginning of the document.

    Variable Length	Quoted Release Character type	Description
    Quoted Release Character	Character	Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. For example, your field delimiter is (,) and your release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the and the Quoted Release Character, the Quoted Release Character will be used.
    	–OR–
    	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the quoted release character for this document is located. For example, if you specify 5 as the character position, you have indicated that the quoted release character appears in the sixth character position from the beginning of the document.

    Variable Length	Release Character type	Description
    Release Character	Character	Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
    	–OR–
    	Character Position	Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located. For example, if you specify 5 as the character position, you have indicated that the field delimiter appears in the sixth character position from the beginning of the document.

    

    Next, set the record identifier.

11. Specify a Record Identifier

    When parsing a file, IBM webMethods Integration looks at a record and extracts an identifier out of the data and uses that identifier to connect the record definition with a particular record in the flat file. The name of the record definition must match the value obtained by the record identifier.

    To set the record identifier for a definition, if the flat file contains a record identifier, select Yes in the Record identifier area, and set the record identifier to one of the following values.

    Note

    The Record Identifier Position has default value selected as Start at field position.

    Record Identifier	Value	Description
    	Start at field position	Identifies the field in the record (counting from zero) that contains the identifier. Start at field Position identifiers use the value of the specified field as the record identifier. These identifiers count from zero (0). For example, if 2 is specified, the third field is used as the record identifier.
    	Start at character position	Identifies the character position in the record (counting from zero) where the record identifier is located. Start at character position record identifiers compare the value that occurs in the record, at the specified offset, to all the record names defined in the flat file structure. Note that the start at character position identifier cannot distinguish between all types of record names. For example, if you name records “Rec1” and “Rec,” some instances of “Rec1” may be identified as “Rec,” because “Rec1” begins with “Rec.”

    Select No if the flat file does not contain a record identifier.

    

12. Click Next to define the Flat File Structure. Use the Flat File Structure page to add records to the flat file structure and define the hierarchical relationships between them.

    The connector receiving the flat file uses the structure to read the flat file. This structural information identifies the parent-child relationships between different records in the flat file. By nesting record elements in the flat file structure (adding record elements to a record), you can represent the hierarchical structure of the data in the flat file.

    

    Enable the Edit Mode in the Data fields panel on the Structure page to:

    • add a new element using the icon and select Record Definition from the Element type field to specify a name for the record definition. For more information see the Usage Notes under step 11 of creating-a-flat-file-connector-from-a-sample-file.
    • quickly navigate within the schema using the right, left, up and down arrows,
    • perform edit actions for example cut, copy, paste and delete fields to modify the fields of the element in the property panel within the file schema, and
    • utilize the keyboard shortcuts for modifying the flat file schema fields.

    

    

    Note

    The Show Data feature on the Data fields panel is available while creating the flat file using the import file function initially and not when you create the flat file manually or edit an existing flat file.

13. To add a Composite Definition after you have added a record definition, specify the following:

    Extractor type	Description
    Nth Field	Field number in the record that contains the composite you want to extract. This pulls the subfield data from the composite. If you leave this property empty, the composite will not be extracted.

    To add a Field Definition after you have added a record definition, specify the following:

    Extractor type	Description
    Fixed Position	Counting from zero (0), indicates a fixed number of bytes to be extracted from a record.
    	Position
    Nth field	Counting from zero (0), indicates the field that you want to extract from the record.
    	Extractor
    	Type a value to indicate the position of the field that you want to extract from the record. This value cannot be null and must be an integer greater than or equal to zero (0). For example, if you type 1, the second field will be extracted.

    This option is available only if you specified a field delimiter when configuring the definition and structure of the Flat File connector. This extractor returns the field as a key–value pair. The key is the name of the field. The value is the String value of the field.

    Note

    Ensure that the Record identifier position and Extractor type field values are the same for Field Definitions, else you might encounter errors while running the connector. For example, if you have selected the Record identifier position as Start at character position, then the Extractor type must be set to Fixed position. If you have selected the Record identifier position as Start at field Position, then the Extractor type must be set to Nth position.

14. Click Next, the Summary page displays the Overview tab which shows the details under the Description and the File Type and the Structure tab, which displays both the file schema and sample output.

    

15. Click Save, to create the flat file connector. You will be redirected to Flat File Connectors page.

    

    You can Edit, or Remove a flat file connector from the Flat File Connector page.

Creating a flat file connector from a sample file

When you create a flat file connector from a sample file, ensure that the format of the sample file is *.txt.

To create a flat file connector from a sample file

1. From the IBM webMethods Integration navigation bar, click Projects.

2. Select a project and then click Connectors > Flat File > Add Connector.

   

   The New Flat File Connector page appears.

3. Select Import from file, and then Drag and drop or click Browse Files to upload the sample file from your computer.

   

4. Click Next.

5. In the Define Connector page, type a name and description of the flat file connector.

6. Drag and drop or click Browse files and select a connector icon, if necessary.

7. In the Preview pane, IBM webMethods Integration updates the sample file contents based on your selections.

   Note

   The Show Preview will be collapsed by default on the Description page. Click to view the file content.

   

8. Click Next.

9. In the Flat File Definition page, configure the Parser and the Record Identifier. Select the values from the dropdown menu or provide the custom input for the Delimiter, Fixed Length & Variable Length parsers. If the custom input is not in the valid format an error is seen.

   In the Parser panel, select one of the following to indicate how the data in the sample file is formatted:

   Record Parser	Description
   Delimiter	Use this parser when each record is separated by a delimiter.
   Fixed Length	Use a fixed length record parser when each record is of a fixed length (for example, mainframe punch or print records). This parser splits a file into records of the same pre-specified length.
   Variable Length	This parser expects each record to be preceded by two bytes that indicate the length of the record. Each record may be of different length.

   Select Use first row as field labels to make the first row as header.

   Note

   Use first row as field labels checkbox is disabled until record length is selected. This option is not available in manual flat file creation or editing existing flat file.

   

   If you select Delimiter as the parser type, specify the parser properties in the record parser panel. IBM webMethods Integration updates the Preview contents based on your selections. Use the following table to specify the delimiters used in the sample file.

   Property	Description
   Record	Character that separates records in a flat file document. If a new line character (\n) exists at the end of each record and the Record delimiter is not ending with \n, then IBM webMethods Integration cannot render the records properly under the Preview pane.
   Field or Composite	Character that separates fields in a flat file document.
   Subfield	Character that separates subfields in a flat file document.
   Quoted Release Character	Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. For example, your field delimiter is (,) and your quoted release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the Quoted Release Character, the Quoted Release Character will be used.
   Release Character	Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.

   If you select Fixed Length as the parser type, specify the parser properties in the record parser panel. IBM webMethods Integration updates the Preview contents based on your selections. Specify the position of each field in Field separators. The Field separators must be comma separated integer values, for example, 5, 10, 15. You can set a value for Record length to adjust the record so that it appears correctly under the Preview pane. Record length supports only positive integer values between one and total number of characters in the file. and cannot be empty.

   If you selected Variable Length as the parser type, specify the parser properties in the Record Parser panel. IBM webMethods Integration updates the Preview contents based on your selections. Use the following table to specify the delimiters used in the sample file.

   Property	Description
   Field	Character that separates fields in a flat file document.
   Subfield	Character that separates subfields in a flat file document.
   Quoted release character	Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. For example, your field delimiter is (,) and your quoted release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertToValues service to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertToString service to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertToString service, if you have specified both the Release Character and the Quoted Release Character, the Quoted Release Character will be used.
   Release character	Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertToValues service to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertToString service to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.

10. Specify a Record Identifier

    When parsing a file, IBM webMethods Integration looks at a record and extracts an identifier out of the data and uses that identifier to connect the record definition with a particular record in the flat file. The name of the record definition must match the value obtained by the record identifier.

    To set the record identifier for a definition, if the flat file contains a record identifier, select Yes in the Record identifier area, and then set the record identifier to one of the following values.

    Note

    The Record Identifier Position has default value selected as Start at field position.

    Record identifier	Value	Description
    	Start at field position	Identifies the field in the record (counting from zero) that contains the identifier. Start at field position identifiers use the value of the specified field as the record identifier. These identifiers count from zero (0). For example, if 2 is specified, the third field is used as the record identifier.
    	Start at character position	Identifies the character position in the record (counting from zero) where the record identifier is located. Start at character position record identifiers compare the value that occurs in the record, at the specified offset, to all the record names defined in the flat file structure. Note that the Start at character position identifier cannot distinguish between all types of record names. For example, if you name records “Rec1” and “Rec,” some instances of “Rec1” may be identified as “Rec,” because “Rec1” begins with “Rec.”

    Select No if the flat file does not contain a record identifier.

    

11. Click Next to view the Flat File Structure.

    

    You can add records to the flat file structure and define the hierarchical relationships between them. The application receiving the flat file uses the structure to read the flat file. This structural information identifies the parent-child relationships between different records in the flat file. By nesting record elements in the flat file structure (adding record elements to a record), you can represent the hierarchical structure of the data in the flat file.

    Enable the Edit Mode and use the Data fields panel on the Structure page to:

    • add a new element using the icon and select Record Definition from the Element type field to specify a name for the record definition,
    • quickly navigate within the schema using the right, left, up and down arrows,
    • perform edit actions for example cut, copy, paste and delete fields and
    • utilize keyboard shortcuts for modifying the flat file schema fields.

    

    You can also modify the fields of the element in the property panel within the file schema once the Edit Mode is on.

    

    Usage Notes: The option within the Data Fields panel functions in two ways:
    - If the selected element is a Field, it adds a sibling Field element to the end of the list.
    - If the selected element is any parent Element, the + option adds a child element to that parent.
    - The right arrow functionality is applicable only when a parent element is present below the selected element.
    - For schema elements, only the add and paste options are enabled. The paste option adds the copied or cut element as a child element.

    Select the Show Data to preview the sample data if you have created the flat file using the import feature.

    The Show Data feature is available only while creating the flat file using the import file function initially and not when you create the flat file manually or edit an existing flat file.

    

    Use the Expand or Collapse All option to view the schema properties for the flat file.

    

    

12. To add a Composite Definition after you have added a record definition, click the Element Type field and select Composite Definition. Specify a name for the record definition and click Save.

    Extractor type	Description
    Nth Field-Extractor	Field number in the record that contains the composite you want to extract. This pulls the subfield data from the composite. If you leave this property empty, the composite will not be extracted.

13. To add a Field Definition after you have added a record definition, click the icon and select Field Definition in the Element type field.

    Extractor Type	Description
    Fixed Position	Counting from zero (0), indicates a fixed number of bytes to be extracted from a record.
    	Extractor
    	Position
    Nth Field	Counting from zero (0), indicates the field that you want to extract from the record.
    	Extractor

    This option is available only if you specified a field delimiter when configuring the definition and structure of the flat file connector. This extractor returns the field as a key–value pair. The key is the name of the field. The value is the String value of the field.

14. Click Next, the Summary page appears where the Overview tab provides information such as the File name, Description, and Connector icon. Additionally, it shows File Type details, including Parser type and Parser fields values. The Structure tab on the same page shows both the file schema and sample data details for the output.

    

15. Click Save to create the flat file connector. You will be redirected to Flat File Connectors page.

    

    You can Edit, or Remove a Flat File connector from the Flat File Connector page.

Flat File Predefined Operations

The following predefined Flat File operations are available:

convertFlatFileToDocument

Converts a flattened flat file data into a structured data, which will conform to the document type associated with the flat file connector.

Input Variables

Output Variables

convertDocumentToFlatFile

Converts a structured data conforming to the given document type structure associated with the connector, to the flattened flat file data.

Input Variables

Output Variables

Example - Invoking Flow services for Flat File Connector

After creating the Flat File connector, let us see how to create a Flow service and invoke one of the following predefined operation:

• convertFlatFileToDocument - Converts the flat file to a document (inbound)
• convertDocumentToFlatFile - Converts a document to a flat file (outbound)

Before you begin

• Log in to your tenant and enable Flow services.
• Check if you have the Developer and Admin roles assigned from the Settings > Roles page.
• In IBM webMethods Integration, ensure you have the Flat File connector with defined connector details. For example, Delimited_flatfile_connector.

Basic Flow

1. Select the project where you want to create the new Flow service. You can also create a new project.

2. Click the Flow services tab and on the Flow services page, click the icon.

   

3. Provide a name, for example, Flatfile_flowservice, and an optional description for the new Flow service.

   

4. Type Delimited_flatfile_connector in the search box, select it, and then select the operation, convertFlatFileToDocument.

   

   

5. Click the mapping icon to map the input and output fields.

   

6. As shown below, click the pipeline input fields (ffDataString) and enter values. The service output (getObjectOutput) is automatically mapped to the pipeline output by a dotted line.

   

   

7. Save the Flow service and then run it.

   

   The data is now converted into a structured data, which will conform to the document type associated with the flat file connector.

Usage Notes

When the convertDocumentToFlatFile operation executes, the field that is defined to start after the end of the fixed length record will not be included in the output data if the following conditions are met:

• The definition and structure of the Flat File connector uses a fixed length record delimiter.
• The definition and structure of the Flat File connector contains a fixed position field that begins beyond the defined length of the fixed length record.
• The input to the convertDocumentToFlatFile operation contains a value for the fixed position field that begins beyond the defined length of the fixed length record.