Custom Connectors

webMethods.io Integration allows you to add the following custom connectors to create your own connectors and operations:

  • REST Connectors
  • SOAP Connectors
  • On-Premises Connectors
  • Flat File Connectors

REST Connectors

REST (Representational State Transfer) is an architectural style that requires web applications to support the HTTP GET, POST, PUT, and DELETE methods and to use a consistent, application-independent interface.Let us understand the following terms related to REST connectors.

Endpoint URL

The endpoint of an API is an unique URL, which represents an object or collection of objects. The endpoint is a reference to a URI that accepts web requests. It is the login endpoint URL to initiate communication with the SaaS provider. To get the endpoint, go through the SaaS provider documentation available on the internet. For example, http://dummy.restapiexample.com/api/v1/ is the REST API endpoint.

Authentication Type

Every back end provides its own Authentication mechanism to provide authorized access to its APIs. You need to get the authentication details from the SaaS provider documentation. For example, for Twitter, go to https://apps.twitter.com and then get the credentials. For Twitter, the authentication is OAuth V1.0a, which you can get from https://apps.twitter.com.

Methods

Methods are tasks that act on a Resource. You must create at least one Method for a Resource after you have created the Resource. You can add a Request Parameter, Request and Response Headers, and a Request and Response body to a Method. The primary or most-commonly-used HTTP verbs (or methods, as they are properly called) are POST, GET, PUT, DELETE, and PATCH. These correspond to create, read/retrieve, update, and delete (or CRUD) operations, respectively. You use the following HTTP methods to map the CRUD operations to HTTP requests.I n a REST request, the resource that you are working with is specified in the URL - Uniform Resource Locator. The URL is a special case of the URI - Uniform Resource Identifier.

Headers and Parameters

REST is not a standard in itself but instead makes use of the HTTP standard. HTTP headers allow the client and the server to pass additional information with the request or the response. For example, the Accept and Content-Type HTTP headers can be used to describe the content being sent or requested within an HTTP request. The client may set Accept to application/json if it is requesting a response in JSON or application/xml if it is requesting a response in XML, that is, when sending data, setting the Content-Type to application/xml tells the client that the data being sent in the request is XML.

REST calls (requests) and responses are sent over the HTTP protocol, hence REST requests are in the form of URLs that point to the resource(s) on the server. Required parameters are attached to the end of the URL. For example, in the resource URL http://<name>.com/user/789, user is the resource and 789 is the parameter that is passed to the URL of the resource. You can use any REST client to make REST calls.

REST parameters specify the variable parts of your resources, that is, the data that you are working with. QUERY parameters are the most common type of parameters, which is appended to the path of the URI when submitting a request. For example, https://api.twitter.com/1.1/users/show.json?screen_name=twitterdev is an example of a QUERY parameter URI where screen_name is the name of the parameter and twitterdev is the value of the parameter.

HTTP Status Codes

HTTP Status Codes indicate the status of the HTTP response:
- 1XX - Informational
- 2XX - Success
- 3XX - Redirection
- 4XX - Client error
- 5XX - Server error

Creating REST Connectors

Let us define a REST Resources and Methods and create custom REST connectors. Further, each connector uses an account to connect to the provider’s back end and perform operations. Each connector comes with a predefined set of operations.

Let’s understand how to:

To create a REST connector

  1. From the webMethods.io Integration navigation bar, click Projects.
  2. Select a project and then click Connectors > REST > Add Connector.
  3. In the Define Connector page, complete the following fields.

    Note: Required fields are marked with an asterisk on the screen.

    • Name: Type a name for the REST connector. For example, ExternalRESTapi.
    • Description: Type an optional description for the REST connector. The description you enter here will appear on the Connectors page.
    • Authentication Type: Every back end provides its own authentication mechanism. Get the authentication details from the back end documentation and select the supported authentication type from the drop-down list. REST connector supports the following authentication types:
      • Credentials
      • OAuth V1.0a
      • OAuth V2.0
      • OAuth V2.0 (JWT Flow)
      • AWS S3 Signature
      • AWS V2 Signature
      • AWS V4 Signature
      • OAuth V2.0 (JWT Client Assertion Flow)
        Note: Currently, webMethods.io Integration supports the OAuth V2.0 (JWT Client Assertion Flow) authentication method exclusively for Microsoft endpoints. Know more about the OAuth V2.0 (JWT Client Assertion Flow) authentication method.
    • Default Endpoint URL: Specify the endpoint URL for the connector. It is the login endpoint URL to initiate communication with the SaaS provider. For example, http://dummy.restapiexample.com/api/v1
      Note: If you are using the OAuth V2.0 (JWT Client Assertion Flow) authentication method, ensure that the default endpoint URL for the REST connector is exclusively for Microsoft endpoints. For example, https://graph.microsoft.com/
    • Connector Icon: Click Browse and select another icon for the REST connector, if necessary.

  4. Click Next. The Resources page appears.

  5. Click Add Resource.

    On the Add Resource page, complete the following fields:

    • Name: Type the resource name. For example, getEmployeeList.
    • Path: Type the path for the Resource. The Resource path is relative to the endpoint specified. Each REST Resource derives its path from the namespace of the REST Resource. For example, if the REST Resource is named getEmployeeList, the path is /employees.
      You can define dynamic parameters in the resource path by enclosing each parameter within { } brackets. For example, to get the employee data corresponding to a dynamic parameter called employeeID, specify the source path as /employee/{employeeID}. To get the item information from a particular department in a store, specify the resource path as /store/{departmentID}/{itemID}.

      Notes:

      • While adding Methods, if the Resource path contains { } brackets, for example, /user/{userID}, you must add a request parameter having the same name, that is, userID, and set the Parameter Type to URI_CONTEXT.
      • When a resource’s URL is updated, the associated operations must also be edited and saved to ensure the new endpoint is utilized. Operations that are not updated may continue to use the old URL even after a resource has been modified.
    • Methods: Every Resource must have a method associated with it. Select an HTTP Method, for example, GET for get actions.

    The next screen that you see will have the specified path and selected HTTP Method. You can optionally provide a short description for this resource and method.

  6. On the Get Method page, complete the following fields:

    In the Request parameter section, you can set parameters that become part of the outgoing request. Parameters specify the variable parts of your resource. Click Add Parameters to add a parameter to the request. Complete the following fields:

    • Name - Type the parameter name, for example, param1.
    • Value - Type a value for the parameter, for example, 23.
    • Parameter Type - Select the parameter’s type, for example, CFG_PARAM.

      The Parameter Type determines how the parameter should be used. If you select an AWS authentication type, then you must add a mandatory request parameter in all the Actions you create. The parameter name must be aws.service and the parameter type must be CFG_PARAM. Type the service name in the endpoint URL as the parameter value. For example, if the endpoint URL is https://<instance>.s3.com/, type the parameter value as s3.

      REST services rely on HTTP methods (GET, POST, PUT, DELETE, and PATCH) to make requests to a SaaS provider. Thus the parameters are closely tied to these HTTP methods, as they are sent as part of these HTTP method requests. The parameters are part of the HTTP URI.

      CFG_PARAM is an internal configuration parameter.

      URI_CONTEXT parameters are passed as the path component of a REST Resource URI, and the parameter names correspond to the URI path variable names specified in the {} annotation. For example, if the URI is https://api.twitter.com/1.1/users/{id}, the Resource path will be /users/{id}, the parameter type will be uriContext, the parameter name will be id, and the value could be the user id, for example, either 1, or 2, or 3.

      QUERYSTRING_PARAM parameters are passed as the query component of a REST resource invocation request. For example, if the URI is https://api.twitter.com/1.1/users/show.json?screen_name=twitterdev, the resource path will be /users/show.json, screen_name is the name of the parameter, twitterdev is the value of the parameter, and the parameter type is query.

      FORM_ENCODED_PARAM - Define a parameter of type FORM_ENCODED_PARAM, if you want to send the parameter as part of the Request body. FORM_ENCODED_PARAM allows you to send simple key value parameters embedded in the Request body for POST or PUT requests. This uses the default web form encoding, which is application/x-www-form-urlencoded.

      Note: For passing parameters of FORM_ENCODED_PARAM type, you will not be able to define the Request body for a Resource, as the generated parameter key value string will be automatically embedded in the Request body.

    • Mandatory - Select this option if you want this parameter to be made mandatory while creating a flow service.

      In the request Headers section, allow the client and the server to pass additional information with the request or the response.

      Note: Do not add an authorization header if you use credentials as the mode of authentication.

  7. Click Add Header to add a request HTTP header. In the Add Header panel, complete the following fields:

    • Name - Type the Header name.
    • Value - Type a value for the Header.
    • Mandatory - Select this option if you want this Header to be made mandatory while creating a flow service.
      In the Body panel, complete the following fields:

    • Content Type: If the documentation of the SaaS provider specifies that the content type of the request body is JSON, select application/json as the content type. If the documentation of the SaaS provider specifies that the content type of the request body is XML, select application/xml as the content type. If the documentation of the SaaS provider specifies that the content type of the request body is binary, select Binary as the content type. These options allow you to control the content in an HTTP request body.
    • Document Type - Select a document type for the request body. If you do not have one, you can create a new document type. Click and the Add new Document type page appears.

    • Provide a name and description of your document type.
    • Click Load XML to generate a document type from the XML structure or click Load JSON to generate a document type from the JSON structure.
    • Provide the Name and Type of the field in order to define the structure and content of the document type.
      A field can be a String, Document, Document Reference, Object, Boolean, Double, Float, Integer, Long, or Short. If you select Document Reference, select the Document Reference. Fields are used to declare the expected content and structure of signatures, document contents, and pipeline contents. In addition to specifying the name and type of a field, and whether the type is an Array, you can set properties that specify an XML Namespace.
    • Indicate whether the field is required at runtime by selecting the Required option.
    • Select the Content Type you can apply to String, String list, or String table variables.
    • After you have entered the details and constraints for each field, click Save.

    Note: Document Types created for a REST connector appears in the Document Types drop-down list for the selected REST connector.

    If the request payload is an array of object, then first create a document type resembling the content of the object, and then select the Array option.

  8. Click Next.
    You will be redirected to the GET Response page.

    You can optionally add more response statuses by clicking the Add Header option. In the Response Body panel, it is mandatory to have one action defined. Click Add body to add one.

    In the Response Body pane, complete the following fields:

    • HTTP Code: Type a single HTTP status code or a code range to indicate the status of the response. Valid values are 100, 101, 102…599 or a range from 100-599.
    • Content Type: If the documentation of the SaaS provider specifies that the content type of the response body is JSON, select application/json as the content type. If the documentation of the SaaS provider specifies that the content type of the response body is XML, select application/xml as the content type. If the documentation of the SaaS provider specifies that the content type of the response body is binary, select Binary as the content type. These options allow you to control the content in an HTTP response body.
    • Error Response: Select this option to handle HTTP errors. If this option is selected when creating a resource, then during the execution of an operation in a Workflow or Flow, if you receive a corresponding Status Code as response from the back end, then an exception will be thrown. For example, let us say, a resource is created with status HTTPs code as 300-500 identified as error response. Then during execution, if we receive any response from 300-500, the operation execution will be considered as failure.

    • Document Type: Select a Document Type for the request body . If you do not have one, you can create a new document type. Click and the Add new Document type page appears.
    • Array: If the response payload is an array of object, then first create a document type resembling the content of the object, and then select the Array option.
  9. Click Next.
    The Add Resource page with all configured details appears.

  10. Click Save to create the REST connector.

    The new REST connector (ExternalRESTapi) appears in the REST Connectors page. Click the drop-down arrow beside the connector name to view the configured accounts. The Used in column displays the Workflows and Flow services where the connector is used.

    Once you have created the REST connector, the next step is to configure an account and an operation for the connector. You can create an account for a connector from the following pages:

    • On the REST Connectors page, point to a connector row, click on the New Account option, and specify the configuration parameters.
    • While creating or editing a Flow service, select an account, choose an action, and then click the Configure Accounts option to create an acount inline.
    • While creating or editing a Workflow, drop a connector to the canvas, click the Settings icon, select an action, and then in the Connect to field, click the + option to specify the configuration parameters for the connector.
Understanding OAuth V2.0 (JWT Client Assertion Flow)

OAuth 2.0 (JWT Client Assertion Flow) is an authentication method used for connecting to REST endpoints in a secure way. This method enables you to generate access tokens for backends that utilize OAuth 2.0 by allowing clients to create digitally signed JWT assertions.

Adding custom fields

With the OAuth 2.0 (JWT Client Assertion Flow) authentication method, you can include custom fields in the Add Account window when setting up an account for a REST connector.

  1. When you select the OAuth 2.0 (JWT Client Assertion Flow) method from the Authentication Type drop-down list, the Add Field button appears on the screen. Click on this button to configure the REST connector using this authentication method.

    The Add button configuration window appears on the screen. This window allows you to define custom fields that are displayed in the Add Account configuration window when setting up an account for the REST connector.

  2. Fill in values for the required fields as per the following instructions:

    • Select Field Category: Choose a field category from the following options:
      • Header - Header is typically the first part of the token. It serves as a container for metadata that provides information about how the token is secured and should be processed.
        Notes:
        • Currently, webMethods.io Integration supports this authentication method for Microsoft endpoints only. So, to ensure the successful creation of a REST connector, it is essential to include an ‘x5t’ header value. x5t is a base64-encoded SHA-1 thumbprint associated with the X.509 certificate.
        • When you enter ‘x5t’ as the header value in the ‘Name’ field, a custom field appears in the ‘Add Account’ configuration window with a ‘+’ button. You can use this button to select an X.509 certificate from your local folder. After uploading the X.509 certificate, webMethods.io Integration automatically calculates the ‘x5t’ value, which is then displayed in the field.
      • Claim - Claim is an essential part in JWTs, providing context and additional information about the token.
        Note: Currently, webMethods.io Integration supports this authentication method for Microsoft endpoints only. So, to ensure the successful creation of a REST connector, it is essential to include an ‘exp’ claim value. exp or expiration time is the time (in minutes) after which the JWT expires.
      • JWT Client Assertion - A JWT client assertion is a specific type of JWT used in OAuth 2.0 authentication. It serves as a means for a client (an application or service) to prove its identity when requesting access to protected resources from an authorization server.
        Note: Currently, webMethods.io Integration supports this authentication method for Microsoft endpoints only. So, to ensure the successful creation of a REST connector, it is essential to include ‘client_id’ and ‘scope’ assertion values. client_id is an unique identifier associated with the application while scope is the resource identifier (application ID URI) of the resource you want, affixed with the .default suffix. For example, https://graph.microsoft.com/.default.
    • Name: Specify the name for the selected field category.
      Note: The value for this field is case-sensitive. For example, for Microsoft endpoints, it is mandatory to include ‘X5T,’ ‘expiration time,’ ‘client ID,’ and ‘scope.’ The names for these values should be ‘x5t,’ ‘exp,’ ‘client_id,’ and ‘scope,’ respectively.
    • Display Name: Provide a suitable display name for this field. For example, Client ID.
    • Description: Provide a short summary for this field.
    • Select Data Type: Select the data type you want to assign to the value of this field. Available values are String, Integer, anyURI, Boolean.
    • Default Value: Specify the default value you want to set for this field.
    • Allowed Values: Define the values you want to set for this field.
    • Field Properties: Enable the specific property you want to set for this field. Available properties include Array, Required, Encrypted, Hidden.
  3. Once you have filled in all the required details, click Add. This action successfully creates and adds the custom field to the list of custom fields in the Define Connector screen. The created custom field then appears on the Add Account configuration window when you are setting up an account for a REST connector.

Notes:

  • Currently, you can create custom fields with the same names as predefined fields on the ‘Add Account’ configuration window. However, this can result in duplicate entries with the same name appearing on the ‘Add Account’ configuration window. For instance, creating a custom field with the same name as an existing predefined field, such as ‘Issuer,’ will lead to multiple entries with identical names on the ‘Add Account’ configuration window.
  • If you enter x5t as the header value in the Name field, a custom field appears in the Add Account configuration window with a ‘+’ button. You can use this button to select an X.509 certificate from your local folder. After uploading the X.509 certificate, webMethods.io Integration automatically calculates the x5t value, and it is displayed in the field. However, when configuring this custom field, even if you enter default and allowed values, they do not appear on the account configuration window.
  • When configuring a custom field, if you specify a default value but do not include that value in the list of allowed values, currently, no warning or error messages are displayed to notify this inconsistency.
  • You can add multiple custom fields using the OAuth V2 JWT Client Assertion Flow authentication method for a REST connector.
  • Once you create a custom field, it cannot be edited or deleted.

To configure an account for the REST connector

  1. Go to the REST Connectors page and click New Account.

    The Add Account configuration window appears.

  2. Add an account for the REST connector by filling out the required fields.

    Note: If you have added custom fields using the OAuth 2.0 (JWT Client Assertion Flow) authentication method, they will be visible in the Add Account configuration window.

  3. Provide an Account Name.
    Note: All fields will be populated with values. You can add or modify the values if required.

  4. Provide a Server URL.
    This is the login endpoint URL you have specified in the Define Connector page while creating the REST connector.

  5. Provide a Consumer ID
    Consumer ID is also referred to as the Client ID in OAuth 2.0. This is a client identifier issued to the client to identify itself to the authorization server. For OAuth 2.0, it is the Consumer Key issued by the service provider and used by the consumer to identify itself to the service provider.

  6. Provide an Access Token
    Access Token is used for authentication and is issued by the authorization server. For OAuth 2.0, it is a value used by the access token consumer to gain access to the protected resources on behalf of the User, instead of using the User’s Service Provider credentials.

  7. Provide a Session Timeout value
    Session timeout value is the maximum number of minutes a session can remain active, in other words, how long you want the server to wait before session timeout (min) terminating a session. The value should be equal to the session timeout value specified at the SaaS provider back end.

  8. Provide a Response Timeout
    The number of milliseconds webMethods.io Integration waits for a response before cancelling its attempt to connect to the back end. In case the response timeout network is slow, or the back-end processing takes longer than usual, increase the response timeout value. It is recommended to specify a value other than 0. If you specify 0, webMethods.io Integration will wait indefinitely for a response.

  9. Provide Retry Count on Response Failure
    The number of times webMethods.io Integration attempts to connect to the back end to read a response if the initial attempt fails. If an I/O error occurs, it will retry only if you have selected the Retry on Response Failure option.

  10. Select Retry on Response Failure
    Select whether webMethods.io Integration should attempt to resend the request when the response has failed, even though the request was sent successfully. If so, select true to re-establish the connection.

  11. Select Trust store Alias
    Select the alias name of the webMethods.io Integration trust store configuration. The trust store contains trusted certificates used to determine trust for the remote server peer certificates. You can also add a new truststore certificate from this field by clicking the icon.

  12. Select Keystore Alias
    Select the alias for the webMethods.io Integration key store configuration. This is a text identifier for the keystore alias. A keystore file contains the credentials (private key/signed certificate) that a client needs for authentication. You can also add a new keystore certificate from this field by clicking the icon. This option is available only if ‘Credentials’ is selected as the authentication type while creating the connector.

  13. Select Client Key Alias
    This is the alias to the private key in the keystore file specified in the Keystore Alias field. The outbound connections use this key to send client credentials to a remote server. To send the client’s identity to a remote server, you must specify values in both the Keystore Alias and the Client Key Alias fields. This option is available only if ‘Credentials’ is selected as the authentication type while creating the connector.

  14. Provide Keep Alive Interval
    To handle stale connections, set the Keep Alive Interval in minutes.

  15. Provide Idle Timeout
    The idle timeout interval in milliseconds defines the interval for which a connection will be kept alive if it’s idle Timeout not in use. A value > 0 keeps the connection alive for the specified value. The default value of -1 implies that the connection will be kept alive until a request fails due to a connection error.

  16. Select a value for Enable SNI
    Server Name Indication (SNI) is an extension to the TLS protocol by which a client indicates which host name it is attempting to connect to at the start of the handshaking process. Enable this option if the SaaS provider offers SNI-based TLS connectivity, and if you want to connect to an SNI enabled SAAS provider to send the host name specified in the Server URL field, as part of the TLS SNI Extension server_name parameter.

  17. Provide SNI Server Name
    If you want to explicitly specify a host name to be included as a part of the SNI extension server_name parameter, in case the host name is other than the host name specified in the Server URL field, specify the host name value in the SNI Server Name field.

  18. Provide Consumer Secret
    Also referred to as the Client Secret, this is a secret matching to the client identifier. For OAuth v2.0, it is the secret used by the Consumer to establish ownership of the Consumer Key.

  19. Provide Instance URL
    This is an optional field and is used to specify a runtime host, if applicable. This may be required in some back ends like Salesforce.

  20. Select Refresh Access Token
    Issued with the OAuth v2.0 access token only. A token used by the client to obtain a new access token without having to involve the resource owner.

  21. Provide Refresh Token
    A token used by the client to obtain a new access token without involving the resource owner.

  22. Provide a Refresh URL
    This is the provider specific URL to refresh an Access Token.

  23. Select Refresh URL Request
    This field is available only if authentication type OAuth v2.0 is selected. OAuth 2.0 access tokens typically have a very short lifetime, so when an access token expires, the OAuth profile does not automatically refresh the expired access token. Select the Body Query String and URL Query String options if you want an expired access token to be refreshed automatically. Use the Refresh Token Callback Service option if the back end requires some refresh requests in a custom format, for example, requests which need more parameters than the ones specified by OAuth v2.0, or the back end uses some custom way of organizing parameters, or expects some other HTTP method request (other than POST). If you select this option, you must specify the Flow service name in the Refresh Token Callback Service field.

  24. Provide the Refresh Token Callback Service
    This is required when the Refresh URL Request is specified as Refresh Token Callback Service. Use this option to specify a new Flow service that the back end requires when adding an account. Click the icon to add the new Flow service to your account. Once you add a Flow service to your account, a framework of the new Flow service is created. You can view the link of the same Flow service on the REST Connectors page.

    Click the FlowService_Name link in the REST Connectors page to add any implementation for the Flow service. You will then be redirected to the Flow services page for the implementation.

  25. Select Enable Connection Pooling
    Select this option if you want to enable connection pooling for a connection. When you enable connection pooling, webMethods.io Integration creates the number of connection instances you specified in the connection’s Minimum Pool Size field. Whenever an Integration needs a connection, webMethods.io Integration provides a connection from the pool. If no connections are available in the pool, and the maximum pool size has not been reached, webMethods.io Integration creates one or more new connections (according to the number specified in the Pool Increment Size field) and adds them to the connection pool. If the pool is full (as specified in the Maximum Pool Size field), the requesting service will wait for webMethods.io Integration to obtain a connection till one sec, until a connection becomes available. Periodically, webMethods.io Integration inspects the pool and removes inactive connections that have exceeded the expiration period of one sec.

  26. Provide Minimum Pool Size
    The minimum number of connection objects that remain in the connection pool at all times, if connection pooling is minimum pool size enabled. When the connector creates the pool, it creates this number of connections.

  27. Provide Maximum Pool Size
    The maximum number of connection objects that can exist in the connection pool if connection pooling is enabled. When the connection pool has reached its maximum number of connections, the connector will reuse any inactive connections in the pool, or, if all connections are active, it will wait for a connection to become available.

  28. Provide Pool Increment Size
    The number of connections by which the pool will be incremented, up to the maximum pool size, if connection pooling is enabled and connections are needed.

  29. Provide Block Timeout (msec)
    The number of milliseconds that webMethods.io Integration will wait to obtain a connection with the SAAS provider before the connection times out and returns an error. For example, you have a pool with maximum pool size of 20. If you receive 30 simultaneous requests for a connection, 10 requests will be waiting for a connection from the pool. If you set the Block Timeout to 5000, the 10 requests will wait for a connection for 5 seconds before they time out and return an error. If the services using the connections require 10 seconds to complete and return connections to the pool, the pending requests will fail and return an error message stating that no connections are available. If you set the Block Timeout value too high, you may encounter problems during error conditions. If a request contains errors that delay the response, other requests will not be sent. This setting should be tuned in conjunction with the maximum pool size to accommodate such bursts in processing. The default value is 1000 msec

  30. Provide Expire Timeout (msec)
    The number of milliseconds that an inactive connection can remain in the pool before it is closed and removed from the pool, if connection pooling is enabled. The connection pool will remove inactive connections until the number of connections in the pool is equal to the Initial Pool Size. The inactivity timer for a connection is reset, when the connection is used by the connector. This setting should be tuned in conjunction with the initial pool size to avoid excessive opening and closing of connections during normal processing. The general recommendation is to keep the Expire Timeout value equal to the Session Timeout value. The default value is 1000 msec

  31. Select a Session Management value
    The access token is refreshed whenever the session expires. Session expiration is handled according to the setting of the Session Management property in your configuration. To refresh the ‘Access Token’ automatically, set the Session Management to either fixed or auto. The Timeout value should be based on the back-end settings.

    Note: If Session Management is set to none, then access token will not be refreshed.

  32. Click Add.
    With this, you have successfully configured the REST connector account. Click the drop-down arrow beside the connector to view the account configured for the connector.

    Note: The accounts created for REST connectors using the Flow Editor do not display the Edit and Delete icons when the same Accounts are displayed on the Connectors page in webMethods.io Integration. You can use the existing accounts configured using the Flow Editor, but you cannot edit or modify the account details. Ensure that you create a new account for the REST connectors if any changes are required for the existing accounts. Further, the accounts created for REST connectors using the Flow Editor cannot be used in Workflows, that is, pre-existing accounts will not be available in the Connect to drop-down list while creating a Workflow. Ensure that you create new accounts in such cases.

    Once you have configured the account for the REST connector, the next step is to configure an operation for the connector.

To configure an operation for the REST connector

  1. Go to the REST Connectors page and click Operations. The Custom Operations page appears.

  2. On the Custom Operations page, click New Operation.
    The Show i/o signature and Test operation options are available on the Custom Operations screen and appears for all custom connector operations.
    The Show i/o signature option allows you to only view the Input and Output signature of the custom operation. The input and output fields cannot be edited. You can click the input and output fields to view the field properties.
    The Test operation option allows you to test the custom operation. Specify the Account name and the Input data and then click Run to test the operation and view the test results in the Result window. For array inputs, you can add the array items and then add the values. If an operation does not have an input signature, the input fields are not displayed.

    On the Connect to account page, select an account that you configured for the connector. Each connector uses an account to connect to the provider’s back end and perform operations. You can also create new account by clicking the + icon.

  3. Provide a name and description of the custom operation.

  4. Click Next.

  5. Select the operation.

  6. Confirm the operation and click Done.

With this you have created a REST connector, configured an account and operation. The connector is now ready for the execution of services.

Configuring Custom Refresh for Access Tokens for Accounts when using OAuth 2.0 Authentication

Summary

OAuth 2.0 access tokens typically have a very limited lifetime. To enable webMethods.io Integration to refresh a configured access token, you must configure Refresh Access Token to true and Session Management to fixed or auto.

To use the Out-of-Box refresh implementations offered by webMethods.io Integration, configure Refresh URL Request as “URL Query String” or “Body Query String” along with appropriate values for the fields Consumer ID, Consumer Secret, Refresh Token, Refresh URL and Access Token. This option can be used if the API provider adheres to the standard implementation of the Refresh grant type as described by the OAuth 2.0 Authorization Framework specification.

The following example provides you information on how to implement and configure a custom refresh mechanism for Accounts using webMethods.io Integration.

Before you begin

Basic Flow

Before you try to configure a refresh token callback for a REST connector, you must have the following configurations (Step 1 - Step 3) completed on the Accounts configuration page:

  1. Get the endpoint target (Server URL) for the connection configuration. For example, https://api.twitter.com/1.1.

  2. Get the Consumer ID and Consumer Secret issued to the client to identify itself to the authorization server.
    Consumer Secret is a secret matching to the client identifier.

  3. Configure the Session Management parameters to determine at what intervals the access token would be considered for refresh.

    The access token is refreshed during the next immediate execution after the determined interval. Session expiration is handled according to the setting of the Session Management property in your configuration. To refresh the ‘Access Token’ automatically, set the Session Management to either fixed or auto. The Timeout value should be based on the back-end settings.

    Note: If Session Management is set to none, then access token will not be refreshed.

  4. Select Refresh URL Request field as Refresh token callback service to configure a custom refresh implementation. This field is available only if the OAuth v2.0 authentication type is selected.

    When you select this option, you must specify the Flow service name in the Refresh token callback service field. This is a service which needs to be implemented by a user. The Refresh token callback service option allows you to create a Flow service, which will be executed when the access token has expired or is not valid. You can view the link of the same Flow service on the REST Connectors page.

  5. Click the custom callback Flow service name in the REST Connectors page to create a new Flow service

    You must add the implementation for refreshing the access token in the Callback Flow service.

    The Flow service adhere to the specification with the input and output parameters shown below:

    These are the inputs which would be provided to the Custom callback Flow service implementation by webMethods.io Integration. Values will correspond to the current values of these fields in the account configuration

    These are the output expected by webMethods.io Integration. The values returned for these fields will determine the inputs for the subsequent API and refresh execution.

    • access_token - The access token issued by the authorization server, applicable for subsequent API and refresh executions.
    • token_type - The type of the token issued. For example, bearer.
    • expires_in - The lifetime in seconds of the access token. For example, the value 3600 denotes that the access token will expire in one hour from the time the response was generated. Relevant for auto session management where expiry interval returned by API provider is given precedence.
    • refresh_token - The refresh token to be used for subsequent refresh execution.
    • scope - (Optional) If identical to the scope requested by the client; else, REQUIRED
    • errorResponse - The details of error in refresh execution.

The newly created Flow service will generate an access token that is mapped to the access token field in the output signature.

Invoking a REST API in a Flow service using a REST Connector

In the earlier section, we created a REST connector and configured the REST connector account. Now let us understand how to invoke a REST API in a Flow service using a REST connector.

  1. From the webMethods.io Integration navigation bar, click Projects. Select a project and then select Flow services. Click the + icon to start creating a new Flow service.

  2. Provide a name and description of the new Flow service. For example, ExternalRESTAPIFlow

  3. On the Flow service, click CONNECTORS.

  4. Type ExternalRESTapi in the search box, select the connector, and then select Add Custom Operation.

  5. On the Connect to account page, select the connection name and provide a name and description of the connection.

  6. Click Next and select the getgetEmployeeList operation.

  7. Confirm the action to create the custom operation.

  8. On the Flow service editor, you will see that the account is now configured.

  9. Click to view the Flow service mapping.

  10. Save the Flow service and click Run. As the results show, the employee details are fetched.

Using Dynamic Accounts in Custom Actions

The Dynamic Account feature enables you to dynamically override certain details of the account you are using to run a custom action created under a custom REST connector.

Enabling dynamic accounts

Perform the following steps to enable dynamic account for a custom action created under a REST connector:

  1. Drag-and-drop a custom REST connector on canvas from the Connectors panel, double-click it, and click the + button to create a new custom action.

    Note: If you want to enable dynamic account for an existing custom action, search the relevant action in the Select action drop-down list, and click the Edit icon.

    You are redirected to the Add Custom Action screen.

  2. Add/select the account you want to use to run the custom action, provide a name for the custom action, toggle-on the Enable Dynamic Account switch, and click Next.

    On the Dynamic Account Configuration screen that appears, you can see a list of configuration fields and their default values for the selected account.

  3. Select the fields for which you want to specify values at run time and then click Next.

    Note: All mandatory account configuration fields are selected by default and cannot be deselected.

  4. Configure the operation for your custom action as explained in the Creating Custom Actions section and then click Next.

  5. Verify the details of your custom action and then click Done.

    You are redirected to the Select action screen.

    With this, the dynamic account feature is enabled for your selected custom action.

Using dynamic accounts

Once the dynamic account feature is enabled for a custom action, you can anytime use it as per your requirements.

To use a dynamic account, perform the following steps:

  1. Drag-and-drop the relevant Custom REST connector from the Connectors panel on canvas, select the required dynamic account-enabled custom action from the Select action drop-down list, select the account you want to use to run the action, and click Next.

    In the action configuration screen that appears, you can observe a $connection block. This block contains all the fields selected at the time of enabling the dynamic connection along with the mandatory fields.

  2. Provide values for the relevant fields listed under the $connection block and then click Next.

    The values you provide for these fields override the default values of your selected account.

    Note: Only encrypted fields (such as, Password, Client Secret) are mandatory under the $connection block. You can optionally keep the rest of the fields under the $connection block blank, in which case, webMethods.io Integration uses the default field values provided at the time of creating the selected account.

  3. Test the action to see if it is working as expected and then click Done.

    This takes you back to the canvas.

    Now whenever you connect this action to any workflow and run it, webMethods.io Integration automatically overrides the default values with dynamic values while creating the connection. This connection with the modified values is then used to run the custom action.

SOAP Connectors

Custom SOAP connectors enable you to access third party web services hosted in the cloud or on-premises environment. You can also invoke a SOAP API in a Flow service by using a SOAP Connector.

The following features are supported for SOAP Connectors:

The following SOAP Binding types are supported:

SOAP connectors have the following restrictions:

Creating SOAP Connectors

The SOAP connectors enable you to access third party Web Services hosted in the cloud or on-premises. The SOAP connector uses a WSDL to create consumer operations. Let us understand how to create a SOAP connector and configure an account.

Note: You cannot add or delete operations to a SOAP connector created from a WSDL.

To start creating a connector, go to the project in which you want to create a SOAP connector and click the Connectors tab.

You will be redirected to the Connectors page.

To create a new SOAP connector, click SOAP and then click Add Connector.

You will be redirected to the New SOAP Connector page. You can create a SOAP connector by importing from the specified URL, or by importing from a WSDL file. In our example, let us use the Import from URL option to create a new SOAP Connector.

  1. Create a new SOAP Connector from the specified URL

  2. Click the Import from URL option. You can see the text field appearing in the same window.

  3. Specify the URL of the WSDL. The URL should begin with http:// or https://. The URL is used to retrieve the WSDL for the SOAP API. For example, let us use the WSDL URL generated from SOAP API ( Soap_api_test3) in our earlier guide.

  4. On filling in the details, click Next.

    The Define Connector page appears.

    The next step is to define the connector.

  5. In the Define Connector page, complete the following fields:

    • Name: Type a name for the SOAP connector, for example, Soap_connector.
    • Description: Type an optional description for the SOAP connector.
    • Connector Icon: Click Browse next to the Connector Icon if you want to select a different icon for your SOAP connector. The icon must be a PNG file and the size cannot exceed 50 KB, else the default image is displayed.
    • User: Enter the user name if authentication is required to access the WSDL URL.
    • Password: Enter the password if authentication is required to access the WSDL URL.
    • Validate Schema with Xerces: Select this option if you want webMethods.io Integration to also use the Xerces parser to validate the schemas associated with the XML Schema definition.

    • Click Save.

    Note: Required fields are marked with an asterisk on the screen.

    The new SOAP connector appears in the SOAP Connectors page. You can edit and remove a SOAP connector from the SOAP Connectors details page. The Used In column displays the Workflow and Flow service where the connector is used.

    Note: You can create a SOAP connector by using the Import from file option in Create SOAP connector page of the Connectors tab. You will be prompted to choose the Primary WSDL file and add imported WSDLs or XSDs or XML to primary WSDL and then define a connector. The ports that are configured here are listed in the Port Binding drop-down field of the Add Account dialog box while configuring a new account. According to the selected Port Binding value the URL field gets updated with the corresponding link. You can also edit the URL to specify a different web service endpoint.

    To edit an existing SOAP connector, click the vertical ellipsis icon in the right corner of your connector, and click Edit. The Define Connector page appears. In the Define Connector page, in the Override Existing WSDL ? section, select No, Keep Existing WSDL if you do not want to modify the WSDL URL or the WSDL file. Select Yes, override WSDL if you want to specify a new WSDL URL or upload a new WSDL file in the WSDL source section.

  6. Note: Updating the WSDL may result in addition or removal of Operations or fields in the Input/Output signature of an Operation. This may lead to incorrect mappings if you have used that Operation in an Workflow or Flow service.

    To delete an existing SOAP connector, on the SOAP Connectors page, click the vertical ellipsis icon in the right corner of your connector, and click Remove.

    The next step is to define the account configuration details.

    To configure an account for the SOAP connector, click the New Account option. The Add Account dialog box appears.

  7. In the Add Account dialog box, fill in the following fields.

    Note: All fields will be populated with values. You can add or modify the below values if required.

    • Account Name: Type a name for the SOAP connector account. Let us say, for example Soap_connector_1I.
    • Port Binding: Select the bind address from the drop-down list, that is, the concrete protocol and data format specification for the web service.
    • URL: This is the URL for the web service. You can edit the URL to specify a different web service endpoint.
    • Response Timeout: The number of milliseconds webMethods.io Integration waits for a response before canceling its attempt to connect to the back end. In case the network is slow or the back end processing takes longer than usual, increase the response timeout value. It is recommended to specify a value other than 0. If you specify 0, webMethods.io Integration will wait indefinitely for a response. If you do not specify a timeout, webMethods.io Integration will consider 5 minutes as the response timeout. If the connection to the host provider ends before webMethods.io Integration receives a response, the web service operation ends with an exception and a status code of 408.
    • User: User name used to authenticate the consumer at the HTTP or HTTPS transport level on the host server.
    • Password: The password used to authenticate the consumer on the host server.
    • Keystore Alias: Alias to the keystore that contains the private key used to connect to the Web Service host securely. You can also add a new Keystore from this field.
    • Key Alias: Alias to the key in the keystore that contains the private key used to connect to the Web Service host securely. The key must be in the keystore specified in the Keystore Alias field.
    • WS Security User Name: Name to include with the Username Token, if the Web Service’s security policy requires one.
    • WS Security Password: The password to include with the UsernameToken (must be plain text).
    • WS Security Keystore Alias: The alias for the keystore, which contains private keys and certificates associated with those private keys. You can also add a new Keystore from this field.
    • WS Security Key Alias: The text identifier for the private key associated with the Keystore Alias.
    • WS Security Truststore Alias: The alias for the truststore, which contains the trusted root of a certificate or signing authority (CA). You can also add a new Truststore from this field.
    • WS Security Partner Certificate Alias - The file that contains the partner’s self-signed certificate. You can also add a new Partner Certificate from this field.
    • Timestamp Precision: Whether the timestamp placed in the Timestamp element of the security header of an outbound message is precise to seconds or milliseconds. If the precision is set to milliseconds, the timestamp format yyyy-MM-dd’T’HH:mm:ss:SSS’Z’ is used. If the precision is set to seconds, the timestamp format yyyy-MM-dd’T’HH:mm:ss’Z’ is used.
    • Timestamp TTL: The time-to-live value for an outbound message in seconds. This value is used to set the expiry time in the Timestamp element of outbound messages. The timestamp precision value is used only when WS-Security is implemented through a WS-Policy.
    • Timestamp Max Skew: The maximum number of seconds that the Web Services client and host clocks can differ so that the timestamp expiry validation does not fail. The timestamp precision value is used only when WSSecurity is implemented through a WS-Policy. The inbound SOAP message is validated only if the creation timestamp of the message is less than the sum of the timestamp maximum skew value and the current system clock time.
  8. Click Add.

    With this, you have successfully configured an account (Soap_connector_act1) for the SOAP connector (Soap_connector). Note: Accounts created for SOAP connectors using the Flow Editor do not display the Edit and Delete icons when the same Accounts are displayed on the Connectors page in webMethods.io Integration. You can use the existing accounts configured using the Flow Editor, but you cannot edit or modify the account details. Ensure that you create a new account for the SOAP connectors if any changes are required for the existing accounts. Further, the accounts created for SOAP connectors using the Flow Editor cannot be used in Workflows, that is, pre-existing accounts will not be available in the Connect to drop-down list while creating a Workflow. Ensure that you create new accounts in such cases.

Invoking a SOAP API in a Flow service using a SOAP Connector

In the earlier section, we created a SOAP connector (Soap_connector) and configured the SOAP connector account (Soap_connector_act1). Now let us understand how to invoke a SOAP API in a Flow service using a SOAP connector.

  1. From the webMethods.io Integration navigation bar, click Projects.
  2. Select a project and then select Flow services. Click the + icon to start creating a new Flow service.
  3. Provide a name and description of the new Flow service. For example, ExecuteSOAPAPIFlowservices.

  4. On the Flow service, select CONNECTORS.

  5. Type Soap_connector in the search box and select the connector.

  6. Next, in the Type to choose action field, select an operation.

    Note: You cannot add operations in a SOAP connector created from a WSDL. Also, you cannot delete operations from a SOAP connector created from a WSDL.

  7. Select the Soap_connector_act1 account. You can also create or configure an account inline.

  8. Click to view the Flow service mapping page.

  9. Save the Flow service and click Run. As the results show, the weather details are fetched.

On-Premises Connectors

Applications uploaded from the on-premises Integration Server are listed in the Connectors > On-Premises Connectors page in webMethods.io Integration. You will not be able to create accounts or operations for on-premises connectors as those can be uploaded only from webMethods Integration Server.

Click here for information on how to upload an application.

Invoking Flow services for On-Premises Connector

Summary

In this tutorial, we will see how to invoke Flow services for on-premises connectors.

Before you begin

Basic Flow

  1. Select a project where you want to create the new Flow service.
  2. Click Connectors > On-Premises.
  3. You will see the list of connectors uploaded from the on-premises system.

  4. Let us use the AddintApp2 connector as an example for invoking Flow services.
    Click the drop-down arrow beside the connector to view the accounts configured for that connector. The Used in column displays the Workflows and Flow services where the connector is used.

  5. Click the Flow services tab and on the Flow services page, click the + icon to create a new Flow service.

  6. Provide a name, for example, OnPremAddIntsFlowService, and an optional description for the new Flow service.

  7. Select the AddintApp2 connector in the search box.

  8. Select the myService operation.

  9. Select the AddintApp111 account.

  10. On the Flow service editor, click the icon to define the input and output fields.

  11. Create two Input Fields, value1 and value2.

  12. Create Output Fields as well.

  13. Click the mapping icon to map the input and output fields.

  14. Click the Pipeline Input fields (value1 and value2) and drag it to the service input (num1 and num2). The service output is automatically mapped to value by a dotted line.

  15. Save the Flow service and run it by clicking the Run icon. Provide the custom field values in the value1 and value2 fields, for example, 10 and 20 respectively.

  16. Run the Flow service and view the execution results.
    With this you have executed the Flow service (OnPremAddIntsFlowService) for an on-premises connector.

Flat File Connectors

You can use the Flat File connector to translate documents into and from flat file formats. To set up the translation, you create the definition and structure of the Flat File connector, which is called a flat file schema. The schema also contains the instructions for parsing or creating the flat file and defines how to identify individual records within a flat file and what data is contained in each of those records.

What is a flat file definition and structure?

The definition and structure of a flat file connector contains the instructions for parsing or creating a flat file. It details the structure of the document, including delimiters, records, and repeated record structures. It also acts as the model against which you can validate an inbound flat file. A flat file structure consists of hierarchical elements that represent each record, field, and subfield in a flat file.

What are the different approaches to create a flat file connector?

You can create a flat file connector using any one of the following approaches:

What are the high-level steps to create a flat file connector?

Step 1 Define the connector details. In this stage, you define the connector details and the approach. You can create the flat file connector either manually or by using a sample file.
Step 2 Define the record parser and specify a record identifier. In this stage, you associate a record parser with the flat file connector that will process flat files inbound to webMethods.io Integration. You also specify how you want the record to be identified after it is parsed.
Step 3 Define the flat file structure. If you are creating the flat file connector manually, in this stage, you specify the hierarchical structure of the flat file by creating and nesting the record definitions.

Creating a flat file connector manually

webMethods.io Integration can process flat files in which:

To create a flat file connector manually

  1. From the webMethods.io Integration navigation bar, click Projects.

  2. Select a project and then click Connectors > Flat File > Add Connector.

    The New Flat File Connector page appears.

  3. Select Create manually and click Next.

  4. In the Define Connector page, type a name and description of the flat file connector.

  5. Drag and drop or click Browse files and select a connector icon, if necessary.

  6. Click Next.

  7. In the Flat File Definition page, configure the Parser and the Record Identifier.

  8. In the Parser section, to configure a delimited record parser, select Delimiter in the Parser area. Use the Delimiter record parser for the Flat File connector when each record is separated by a delimiter. You can type the input or use the custom dropdown input with predefined list of values to select values for Record, Field or Composite, Sub Field, Quoted Release Character & Release Character fields. For a record delimiter, you can specify a character (for example, !) or character representation (for example, \r\n for carriage return).

    Specify the following fields:

    Delimiter Record type Description
    Record Character Character that separates records in a flat file document.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the record delimiter for this document is located.
    Note: For example, if you specify 3 as the character position, you have indicated that the record delimiter appears in the fourth character position from the beginning of the document.
    Delimiter Field or composite type Description
    Field or Composite Character Character that separates fields in a flat file document.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located.
    Note: For example, if you specify 4 as the character position, you have indicated that the field delimiter appears in the fifth character position from the beginning of the document.
    Delimiter Subfield type Description
    Subfield Character Character that separates subfields in a flat file document. The default is a period “.”.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the subfield delimiter for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the subfield delimiter appears in the sixth character position from the beginning of the document.
    Delimiter Quoted Release Character type Description
    Quoted Release Character Character Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters.
    Note: For example, your field delimiter is (,) and your release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the Quoted Release Character, the Quoted Release Character will be used.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the quoted release character for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the quoted release character appears in the sixth character position from the beginning of the document.
    Delimiter Release character type Description
    Release Character Character Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter.
    Note: For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the field delimiter appears in the sixth character position from the beginning of the document.

    Next, set the record identifier.

  9. Use a Fixed Length record parser type when each record is of a fixed length (for example, mainframe punch or print records). This parser splits a file into records of the same pre-specified length. To configure a fixed length record parser, in the Parser Type area, select Fixed Length.

    Specify the following fields:

    Fixed Length Description
    Record Length In the Record Length field, enter the length, in characters, of each record in the flat file. Record length cannot be empty.
    Fixed Length Field or composite type Description
    Field or Composite Character Character that separates fields or composites in a flat file document.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located.
    Note: For example, if you specify 4 as the character position, you have indicated that the field delimiter appears in the fifth character position from the beginning of the document.
    Fixed Length Subfield type Description
    Subfield Character Character that separates subfields in a flat file document.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the subfield delimiter for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the subfield delimiter appears in the sixth character position from the beginning of the document.
    Fixed length Quoted release character type Description
    Quoted Release Character Character Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters.
    Note: For example, your field delimiter is (,) and your release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane.When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the and the Quoted Release Character, the Quoted Release Character will be used.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the quoted release character for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the quoted release character appears in the sixth character position from the beginning of the document.
    Fixed Length Release Character type Description
    Release Character Character Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter.
    Note: For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the field delimiter appears in the sixth character position from the beginning of the document.

    Next, set the record identifier.

  10. The Variable Length record parser type expects each record to be preceded by two bytes that indicate the length of the record. Each record may be a different length.

    To configure a variable length record parser, in the Parser area, select Variable length, and specify the following fields:

    Variable Length Field or Composite type Description
    Field or Composite Character Character that separates fields or composites in a flat file document.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located.
    Note: For example, if you specify 4 as the character position, you have indicated that the field delimiter appears in the fifth character position from the beginning of the document.
    Variable length Subfield type Description
    Subfield Character Character that separates subfields in a flat file document.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the subfield delimiter for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the subfield delimiter appears in the sixth character position from the beginning of the document.
    Variable Length Quoted Release Character type Description
    Quoted Release Character Character Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. Note: For example, your field delimiter is (,) and your release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the and the Quoted Release Character, the Quoted Release Character will be used.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the quoted release character for this document is located. Note: For example, if you specify 5 as the character position, you have indicated that the quoted release character appears in the sixth character position from the beginning of the document.
    Variable Length Release Character type Description
    Release Character Character Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter.
    Note: For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
    –OR–  
    Character Position Starting from the beginning of the document and counting from zero (0), the character position at which the field delimiter for this document is located.
    Note: For example, if you specify 5 as the character position, you have indicated that the field delimiter appears in the sixth character position from the beginning of the document.

    Next, set the record identifier.

  11. Specifying a Record Identifier

    When parsing a file, webMethods.io Integration looks at a record and extracts an identifier out of the data and uses that identifier to connect the record definition with a particular record in the flat file. The name of the record definition must match the value obtained by the record identifier.

    To set the record identifier for a definition, if the flat file contains a record identifier, select Yes in the Record identifier area, and set the record identifier to one of the following values.(Note: The Record Identifier Position has default value selected as Start at field Position).

    Record Identifier Value Description
    Start at field Position Identifies the field in the record (counting from zero) that contains the identifier. Start at field Position identifiers use the value of the specified field as the record identifier. These identifiers count from zero (0).
    Note: For example, if 2 is specified, the third field is used as the record identifier.
    Start at character position Identifies the character position in the record (counting from zero) where the record identifier is located. Start at character position record identifiers compare the value that occurs in the record, at the specified offset, to all the record names defined in the flat file structure. Note that the start at character position identifier cannot distinguish between all types of record names.
    Note: For example, if you name records “Rec1” and “Rec,” some instances of “Rec1” may be identified as “Rec,” because “Rec1” begins with “Rec.”

    Select No if the flat file does not contain a record identifier.

  12. Click Next to define the Flat File Structure. Use the Flat File Structure page to add records to the flat file structure and define the hierarchical relationships between them.

    The connector receiving the flat file uses the structure to read the flat file. This structural information identifies the parent-child relationships between different records in the flat file. By nesting record elements in the flat file structure (adding record elements to a record), you can represent the hierarchical structure of the data in the flat file.

    To add a record, click the icon icon and specify a name for the record definition. Select Record Definition from the Element type field to specify a record definition.

  13. To add a Composite Definition after you have added a record definition, specify the following:

    Extractor type Description
    Nth Field Field number in the record that contains the composite you want to extract. This pulls the subfield data from the composite. If you leave this property empty, the composite will not be extracted.

    To add a Field Definition after you have added a record definition, specify the following:

    Extractor type Description
    Fixed Position Counting from zero (0), indicates a fixed number of bytes to be extracted from a record.
      Position
    Nth field Counting from zero (0), indicates the field that you want to extract from the record.
      Extractor
      Type a value to indicate the position of the field that you want to extract from the record. This value cannot be null and must be an integer greater than or equal to zero (0). Note: For example, if you type 1, the second field will be extracted.

    This option is available only if you specified a field delimiter when configuring the definition and structure of the Flat File connector. This extractor returns the field as a key–value pair. The key is the name of the field. The value is the String value of the field.

    Note: Ensure that the Record identifier position and Extractor type field values are the same for Field Definitions, else you might encounter errors while running the connector. For example, if you have selected the Record identifier position as:

    • Start at character position, then the Extractor type must be set to Fixed position.

    • Start at field Position, then the Extractor type must be set to Nth position.

  14. Click Next, the Summary page displays the Description (File name, Description, and Connector icon) and the File Type (Parser type and Parser fields value) details.

  15. Click Save, to create the flat file connector. You will be redirected to Flat File Connectors page.

    You can Edit, or Remove a Flat File connector from the Flat File Connector page.

Creating a flat file connector from a sample file

When you create a flat file connector from a sample file, ensure that the format of the sample file is *.txt.

To create a flat file connector from a sample file

  1. From the webMethods.io Integration navigation bar, click Projects.

  2. Select a project and then click Connectors > Flat File > Add Connector.

    The New Flat File Connector page appears.

  3. Select Import from file, and then Drag and drop or click Browse Files to upload the sample file from your computer.

  4. Click Next.

  5. In the Define Connector page, type a name and description of the flat file connector.

  6. Drag and drop or click Browse files and select a connector icon, if necessary.

  7. In the Preview pane, webMethods.io Integration updates the sample file contents based on your selections. (Note: The “Show Preview” will be collapsed by default on the Description page. Click to view the file content.

  8. Click Next.

  9. In the Flat File Definition page, configure the Parser and the Record Identifier. Use the custom dropdown input with predefined list of values or type the input for the Delimiter, Fixed Length & Variable Length parsers.

  10. Select Use first row as field labels to make the first row as header.(Note: Use first row as field labels checkbox is disabled until record length is selected. This option is not available in manual flat file creation or editing existing flat file.)

  11. In the Parser panel, select one of the following to indicate how the data in the sample file is formatted:

    Record Parser Description
    Delimiter Use this parser when each record is separated by a delimiter.
    Fixed Length Use a fixed length record parser when each record is of a fixed length (for example, mainframe punch or print records). This parser splits a file into records of the same pre-specified length.
    Variable Length This parser expects each record to be preceded by two bytes that indicate the length of the record. Each record may be of different length.

    If you select Delimiter as the parser type, specify the parser properties in the record parser panel. webMethods.io Integration updates the Preview contents based on your selections. Use the following table to specify the delimiters used in the sample file.

    Property Description
    Record Character that separates records in a flat file document. Note: If a new line character (\n) exists at the end of each record and the Record delimiter is not ending with \n, then webMethods.io Integration cannot render the records properly under the Preview pane.
    Field or Composite Character that separates fields in a flat file document.
    Subfield Character that separates subfields in a flat file document.
    Quoted Release Character Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. Note: For example, your field delimiter is (,) and your quoted release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertFlatFileToDocument operation to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertDocumentToFlatFile operation to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertDocumentToFlatFile operation, if you have specified both the Release Character and the Quoted Release Character, the Quoted Release Character will be used.
    Release Character Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. Note: For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertFlatFileToDocument operation to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertDocumentToFlatFile operation to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.

    If you select Fixed Length as the parser type, specify the parser properties in the record parser panel. webMethods.io Integration updates the Preview contents based on your selections. Specify the position of each field in Field separators. The Field separators must be comma separated integer values, for example, 5, 10, 15. You can set a value for Record length to adjust the record so that it appears correctly under the Preview pane. Record length supports only positive integer values between one and total number of characters in the file. and cannot be empty.

    If you selected Variable Length as the parser type, specify the parser properties in the Record Parser panel. webMethods.io Integration updates the Preview contents based on your selections. Use the following table to specify the delimiters used in the sample file.

    Property Description
    Field Character that separates fields in a flat file document.
    Subfield Character that separates subfields in a flat file document.
    Quoted release character Character used to enable a section of text within a field to be represented as its literal value. Any delimiter characters that appear within this section will not be treated as delimiters. For example, your field delimiter is (,) and your quoted release character is “. When you want to use (,) within a field as text, you must prefix it with your quoted release character. When using the convertToValues service to create the strings Doe, John and Doe, Jane, the record would appear as “Doe, John”,“Doe, Jane”. When using the convertToString service to create “Doe, John”,“Doe, Jane”, the value of the record would be Doe, John and Doe, Jane. When using the convertToString service, if you have specified both the Release Character and the Quoted Release Character, the Quoted Release Character will be used.
    Release character Character used to enable a delimiter to be used for its intended, original meaning. The character following the release character will not be treated as a delimiter. For example, your field delimiter is + and your release character is \. When using + within a field as text, you must prefix it with your release character. When using the convertToValues service to create the strings a+b+c and d+e+f, the record would appear as a\+b\+c+d\+e\+f. When using the convertToString service to create a\+b\+c+d\+e\+f, the value of the record would be a+b+c and d+e+f.
  12. Specifying a Record Identifier

    When parsing a file, webMethods.io Integration looks at a record and extracts an identifier out of the data and uses that identifier to connect the record definition with a particular record in the flat file. The name of the record definition must match the value obtained by the record identifier.

    To set the record identifier for a definition, if the flat file contains a record identifier, select Yes in the Record identifier area, and then set the record identifier to one of the following values. (Note: The Record Identifier Position has default value selected as Start at field Position).

    Record identifier Value Description
    Start at field Position Identifies the field in the record (counting from zero) that contains the identifier. Start at field Position identifiers use the value of the specified field as the record identifier. These identifiers count from zero (0). Note: For example, if 2 is specified, the third field is used as the record identifier.
    Start at Character position Identifies the character position in the record (counting from zero) where the record identifier is located. Start at Character position record identifiers compare the value that occurs in the record, at the specified offset, to all the record names defined in the flat file structure. Note that the Start at Character position identifier cannot distinguish between all types of record names. Note: For example, if you name records “Rec1” and “Rec,” some instances of “Rec1” may be identified as “Rec,” because “Rec1” begins with “Rec.”
  13. Select No if the flat file does not contain a record identifier.

  14. Click Next to view the Flat File Structure.

    You can add records to the flat file structure and define the hierarchical relationships between them. The application receiving the flat file uses the structure to read the flat file. This structural information identifies the parent-child relationships between different records in the flat file. By nesting record elements in the flat file structure (adding record elements to a record), you can represent the hierarchical structure of the data in the flat file.

    To add a record, click the Add new element icon and select Record Definition from the Element type field to Specify a name for the record definition.

  15. To add a Composite Definition after you have added a record definition, click the Element Type field and select Composite Definition. Specify a name for the record definition and click Save.

    Extractor type Description
    Nth Field-Extractor Field number in the record that contains the composite you want to extract. This pulls the subfield data from the composite. If you leave this property empty, the composite will not be extracted.
  16. To add a Field Definition after you have added a record definition, click the icon icon and select Field Definition in the Element type field.

    Extractor Type Description
    Fixed Position Counting from zero (0), indicates a fixed number of bytes to be extracted from a record.
      Extractor
      Position
    Nth Field Counting from zero (0), indicates the field that you want to extract from the record.
      Extractor

    This option is available only if you specified a field delimiter when configuring the definition and structure of the flat file connector. This extractor returns the field as a key–value pair. The key is the name of the field. The value is the String value of the field.

  17. Click Next, the Summary page displays the Description (File name, Description, and Connector icon) and the File Type (Parser type and Parser fields value) details.

  18. Click Save to create the flat file connector. . You will be redirected to Flat File Connectors page.

    You can Edit, or Remove a Flat File connector from the Flat File Connector page.

Flat File Predefined Operations

The following predefined Flat File operations are available:

convertFlatFileToDocument

Converts a flattened flat file data into a structured data, which will conform to the document type associated with the flat file connector.

Input Variables
ffDataString String The flat file input with type of String.
ffData Object The flat file input with type of InputStream or ByteArray. If both ffData and ffDataString are parsed, ffDataString takes precedence.
ffIterator Object Optional. An object that encapsulates and keeps track of the input data during processing. It is used only when the iterate variable has been set to true.
encoding String Optional. The encoding of the InputStream passed in to ffData. The default encoding is UTF–8.
delimiters Document Optional. A document object that contains the segment terminator and the field and subfield separators. If the delimiter is null, it will be located using the information defined in the definition and structure of the Flat File connector. To specify a delimiter, you can specify:
- One character or character representation (for example, *, \n for line terminator, \t for tab)
- The space character
 Variable Description
  record String Character used to separate records. If you want to specify the two–character carriage return line feed (CRLF) characters, specify \r\n.
   field String Character used to separate fields.
   subfield String Character used to separate subfields.
   release String Character used to ignore a record, field, or subfield delimiter in a field. If a release character occurs in a field or subfield before the delimiter, it will be prefixed with the release.
   quotedRelease String Character to use to ignore a record, field, or subfield delimiter in a field. If a quoted release character occurs in a field or subfield before the delimiter, it will be prefixed with quotedRelease before being written to the output string. The string is pre- and appended with the quoted release character. For example, if * is a delimiter, the field value is a*b, and the quoted release character is “, the string appears as “a*b”.
iterate String Optional. Whether you want to process the input all at one time.
Valid values are true and false.
batchsize String Optional. Whether you want to process the specified number of records. The input value of this field is a positive integer. It is applicable only when the iterate field value is set to true. If you do not specify this field value, default value is set to 1.
createIfNull String Optional. Whether to create the document object if all the fields are null.
skipWhiteSpace String Optional. Whether white space at the beginning of records will be ignored.
keepResults String Optional. Whether to return the parsed data.
validate String Optional. Whether to return error messages that describe how ffData differs from the definition and structure of the Flat File connector.
returnErrors String Optional. Whether to return the validation errors.
maxErrors String Optional. The maximum number of errors that can be returned from one record. When the flat file parser encounters more than the maximum number of errors within a record, the parser will stop parsing and return the parsed data and errors processed up until that point.
flags String Optional. Flags that you can set to govern convertFlatFileToDocument options.
   Variable Description
   addRecordCount String Whether you want the operation to add an additional field (@record–count) to each parsed record in the resulting document object.
  detailedErrors String Whether you want detailed conditional validation error information.
  skipToFirstRecord String Whether you want the operation to wait until it finds the first valid record before reporting invalid records as errors.
  trimWhitespace String Whether you want the operation to delete any blank spaces at the beginning of fields, at the end of fields, or both.
  resultAsArray String Whether you want the operation to return the document object that represents the input flat file data as a document reference that can be mapped to the document types generated.
Output Variables
_dt Document. The name of the output document is flatfile_application_name_dt, where flatfile_application_name is the name of the flat file application. For example, if the connector name is test, then the name of the output document will be test_dt. The structure of the output document will be similar to what you have defined in your Flat File connector.
isValid String. Whether flat file contains validation errors.
errors String Optional. An array containing the validation errors, if any, that were found in ffData or ffDataString.
hasMore String Optional. Indicates when the iteration is terminated. Use this variable as condition to break the batching loop.
This is applicable only when the iterate input is set to true.
Valid values are:
  • true. A few records are pending for processing. Iteration is not complete.
  • false. All the records are processed and the iteration ends.

convertDocumentToFlatFile

Converts a structured data conforming to the given document type structure associated with the connector, to the flattened flat file data.

Input Variables
<flatfile_application_name>_dt Document Structure conforming to the Document type created for the corresponding application of this operation.
spacePad String Optional. How to position the records in the flat file.
signalError String Whether to create errors in the output.
noEmptyTrailing Fields String Whether trailing empty fields are to be removed from the output. Used only with records that have delimited fields.
noEmptyTrailing SubFields String Whether trailing empty subfields are to be removed from the output. Used only with records that have delimited fields. If no value is specified for the noEmptyTrailingSubFields parameter,webMethods.io Integration uses the value set for the noEmptyTrailingFields parameter.
delimiters Document Optional. The separator characters used to construct the output string. To specify a delimiter, you can specify: - One character or character representation (for example, *, \n for line terminator, \t for tab)
  Value Description
  record String Character to use to separate records. If you want to specify the two–character carriage return line feed (CRLF) characters, specify \r\n.
  field String Character to use to separate fields.
  subfield String Character to use to separate subfields.
  release String Character to use to ignore a record, field, or subfield delimiter in a field. If a release character occurs in a field or subfield before the delimiter, it will be prefixed with release before being written to the output string.
  quotedRelease String Character to use to ignore a record, field, or subfield delimiter in a field. If a quoted release character occurs in a field or subfield before the delimiter, it will be prefixed with quotedRelease before being written to the output string. The string is pre- and appended with the quoted release character. For example, if * is a delimiter, the field value is a*b, and the quoted release character is “, the string appears as “a*b”.
  FormatInfo Document Any values mapped to the FormatInfo variable will be passed unmodified to all format operations invoked by convertDocumentToFlatFile and convertFlatFileToDocument.
outputFileName String Optional. If you want the output returned in a file instead of in the string output variable, provide the name of the file you want created as a result of this operation.
Encoding String The type of encoding used to write data to the output file. The default encoding is UTF–8.
sortInput String Optional. Whether you want the operation to sort the input records to match the definition and structure of the Flat File application.
returnAsBytes Returns the document as a string or as a byte array instead of a string.
Output Variables
string String Data that represents the flat file document.
bytes Object If the input variable returnAsBytes is true, returns the output as a byte array encoded using the specified encoding. The string value is not returned.
errorArray Object String array containing messages pertaining to errors that occurred during conversion.

Example - Invoking Flow services for Flat File Connector

After creating the Flat File connector, let us see how to create a Flow service and invoke one of the following predefined operation:

Before you begin

Basic Flow

  1. Select the project where you want to create the new Flow service. You can also create a new project.
  2. Click the Flow services tab and on the Flow services page, click the icon icon.

  3. Provide a name, for example, Flatfile_flowservice, and an optional description for the new Flow service.

  4. Type Delimited_flatfile_connector in the search box, select it, and then select the operation, convertFlatFileToDocument.

  5. Click the mapping icon to map the input and output fields.

  6. As shown below, click the pipeline input fields (ffDataString) and enter values. The service output (getObjectOutput) is automatically mapped to the pipeline output by a dotted line.

  7. Save the Flow service and then run it.

    The data is now converted into a structured data, which will conform to the document type associated with the flat file connector.

Usage Notes

When the convertDocumentToFlatFile operation executes, the field that is defined to start after the end of the fixed length record will not be included in the output data if the following conditions are met: