Using End-to-End Monitoring
This section describes the Dashboard, Alert and Environment Groups and Stages features of End-to-End Monitoring.
This section describes the Dashboard, Alert and Environment Groups and Stages features of End-to-End Monitoring.
The dashboard for End-to-End Monitoring gives you a collective view of all the business transactions carried out within your cloud platform. The default Dashboard view includes the transaction widgets, Transactions list and Groups list.
The following widgets are available on the dashboard:
NETWORK VIEW – This is a graphical representation of all the transactions across all the supported nodes. It shows the transactions flowing through each runtime as well as the status of each runtime for the selected time range. This widget also provides you a summary of all the transactions processed based on the following parameters:
Total
Failed
Success
An example of the NETWORK VIEW widget is as follows:
TOP 5 GROUPS BY ERROR RATE – This is a graphical representation which shows the top 5 groups based on the error rate. The values are displayed in percentage. The All Transactions group is excluded from this widget.
An example of this widget is as follows:
TOP 5 RULES VIOLATED – This is a graphical representation which shows the top 5 rules that are violated the maximum number of times. The X-axis of this graph shows a count of the number of times these rules are violated. The result shown on this widget corresponds to the selected duration. For example, if you have set the duration as Last 30 minutes and a rule is violated 1 hour before, then that rule violation will not appear on this widget. An example of this widget is as follows:
HELP TOPICS – This provides you quick access to the help content for some of the commonly used topics.
All groups – This lists all the transaction groups you create using the application and also the default All Transactions group. At the group level, you can see the following parameters for the selected time range:
Note: You can customize the parameters to view by clicking
next to Create Group.
Name – Name of the group.
Total transactions – The total number of transactions for each group.
Avg. time (ms) – Average time of execution for all the transactions within a group.
Error rate (%) – The rate of error in percentage for all the transactions within a group.
Success rate (%) – The rate of success in percentage for all the transactions within a group.
Environment – Environments for all transactions within a group.
Stage – Stages for all transactions within a group.
Environment group – Environment groups created for all transactions within a group.
Transaction ID - Custom transaction ID set by you for better traceability and usability.
You now have the ability to specify a custom transaction ID as part of your FlowServices. This persists as part of your transaction trace stored in End-to-End Monitoring. This feature allows you to reference or search a transaction based on your own identifier such as order ID or shipping reference. The identifier does not necessarily have to be unique to the transaction. The identifier can be used to find all distinct transactions related to a common asset such as a customer or order.
Note: You can use the custom identifier to filter for related transactions from the updated End-to-End Monitoring dashboard using the search field.
If you have not defined a custom transaction ID, the Business flow map shows undefined under Transaction ID.
A single transaction can have multiple identifiers. For instance, developers can choose to associate different transaction IDs with different services belonging to the same transaction like when a batch file splits out into separate records. In this case anyone of the identifiers can be used to filter and these are shown in the user interface as a comma separated list.
If your transaction belongs to a particular Software AG cloud product node, click on the node in the Business flow map of the transaction details page to view the custom transaction ID.
In end-to-end transactions involving multiple Software AG cloud products, transaction ID for a particular product’s transaction can be viewed by clicking on the product node in the Business flow map of the transaction details page.
Custom transaction IDs can be set programmatically with the service pub.flow:setCustomContextID
from:
Viewing the transactions list now provides Transaction ID option in Show Columns.
Filtering the business transactions now lets you filter using Transaction ID too.
Creating a group of transactions allows you to categorize the transactions based on your requirements.
In the Dashboard page, select Groups tab and click Create Group.
In the Create group dialog box, provide the following details:
Parameter | Description |
---|---|
Name | Type a name for this group. Note: The group name cannot be edited after you save this form. |
Environment | Provide the environment where this group will be created. |
Stage | Provide the stage under which this group will be created. For webMethods.io Integration Cloud tenants this can be either of the following options:
However, in case of Software AG cloud tenants, the stage is automatically mapped to the environment. Hence, the stage option is not available during group creation in Software AG cloud tenants. |
Status | Group the transactions on the basis of their status. Note: Click Select All to select all the listed options. It can be a combination of:
|
Starts with | Group the transactions on the basis of the application from where the business flow starts. You can choose one or more products when creating the group. For example, you can have a group where you have one transaction starting from webMethods.io API and another transaction starting from webMethods.io Integration. Note: Click Select All to select all the listed options. You can select from the following:
Note: Currently, End-to-End Monitoring support for B2B is very basic. You may not get complete trace for all types of B2B transactions. For example, transactions invoked by the AS2 protocol may not provide you the complete trace results. |
Duration (ms) | Group the transactions on the basis of their execution time in milliseconds. You can select from the following:
|
Product(s) involved | Group the transactions on the basis of the product(s) involved in a business transaction. Use the drop-down list to select one or more products. The group includes all the transactions for the selected product. If you select more than one product, it will list the transactions for the standalone products along with the transactions that involve both the products. For example, if you select webMethods.io API and webMethods.io Integration, the group will include the following: Note: Click Select All to select all the listed options.
|
Transaction name(s) | Group the transactions on the basis of their names. The transaction name relates to an asset that produces transactions as defined by the chosen product types. For example, webMethods.io Integration generates transactions based on the name of services being run, whereas API Gateway generates transactions based on the name of the associated API. You can choose more than one transaction name and it can either be a complete match or a partial match. Note:
|
Error Message(s) | Group the transactions on the basis of an error message. Click + to add more than one entry. |
Click Save.
On the Dashboard page, select the group for which you want to view the transactions. On the Transactions list page, you will see all the transactions for the selected time frame and the set filter(s).
Click on the top-right corner of the screen to open the Show Columns dialog box.
Select the parameters for which you want to view the transaction details and click Save. The transactions list shows the following parameters:
Parameter | Description |
---|---|
Status | Status of the transaction. Currently, we have the following states:
|
Name | Name of the transaction. This name is based on the starting point of the transaction. For example, if the transaction starts from webMethods.io API, then this name would be the name of the API. |
Duration (ms) | Total time taken by the transaction to complete. |
Starts with | The application from which the transaction starts. For example, webMethods.io Integration. |
Started on | Start time for the transaction. |
Error message | Message with which the transaction failed. |
Trace ID | This is a unique identifier for the transaction. |
Transaction ID | This is a custom transaction ID set by you for better traceability and usability. |
This is an example of the transaction list view:
You can also filter the transactions from this page. For more information, see Filtering the Business Transactions.
On the Dashboard page, select in the row for the transaction that you want to edit.
Note: Editing a transaction group impacts all the rules associated with it.
Make the necessary changes and click Update.
On the Dashboard page, select in the row for the transaction that you want to delete.
Note: Deleting a transaction group deletes all the rules and the rule violations associated with it.
Click Delete. On successful deletion, you will see a confirmation message on the screen.
You can filter the business transactions within any group to view a custom set of transactions.
Note: Filters that were set during group creation are already applied when you open the group.
To filter the transactions
On the Transactions list page, click as shown in the following example:
In the Filter dialog box, you can choose from the following filters:
Note: Filters set during group creation are greyed out and not available for selection.
Value | Description |
---|---|
Status | Select the transaction status. You can select from the following and click Save:
|
Name | Type the name of the business transaction and click Save. To add more than one transaction name, click +. |
Duration | Select from the following operators using the drop-down list and provide a valid time in milliseconds:
|
Transaction starts with | Filter the transactions on the basis of the application from where the business flow starts. Select the checkbox next to the product. You can select more than one product. |
Error message | Filter the transactions based on the error message with which it failed. Type the error message and click Save. For example, server not found. You can filter based on more than one error message by clicking+next to the error message box. |
Product(s) involved | Filter the transactions based on the products that are involved with this transaction. Select the checkbox next to the product and click Save. You can select more than one product. |
Trace ID | Filter the transactions based on this unique identifier associated with a transaction. Enter the trace ID and click Save. You can filter based on more than one Trace ID by clicking + next to the trace ID box. |
Transaction ID | Filter the transactions based on the custom transaction identifier set by you. Enter the transaction ID and click Save. |
The applied filters show up on the top of the page as shown in the following example:
You can close a particular filter by selecting the filter and clicking X adjacent to the filter name.
After you apply the filters and retrieve the list of transactions, you can further sort the list by clicking the column headings. Sort in ascending or descending order by toggling the column header.
Note: When you add a custom transaction ID to your end-to-end business transaction, you can filter based on the custom transaction ID from the End-to-End Monitoring user interface. This is useful when you need to tag a specific transaction, beyond the filtering capability offered by service name, trace ID, status or duration.
The Business Transaction Details page provides the following information about the transaction:
Parameter | Description |
---|---|
Name | Transaction name. |
Trace ID | Distinct identifier for the transaction generated internally by the application. |
Transaction ID | Transaction identifier set by you. |
Status | Status of the transaction. |
Start period | Time at which the transaction started. |
End period | Time at which the transaction ended. |
Duration (ms) | Total time taken by the transaction to execute. This includes the Total transit time and the Processing time of the transaction. Total transit time is the time from which the first product in the cross-product transaction receives the request from the client until it sends back the response to the client, excluding the processing time taken by each product. This is dependent on network latency and other factors such as Software AG products on different tenants, in a cross-product transaction. |
This is an example of a business transactions details page:
Duration is the total time taken by a transaction from the time a client initiates a request to the time a response is received by the client. Let us consider the following scenario where the duration for a transaction to complete is 200ms. In the following image, 200ms is the time difference between INT22 and INT11. The time taken by the transaction within the API component which is 50ms is already taken into account as part of this 200ms value. End-to-End Monitoring records the duration only from the time a client request enters the first component of the cloud platform (INT11) and the time the response exits the last component of the cloud platform (INT22).
Note: The application does not note the time the client sends the request (T1) and the time the client receives the response (T2), as this happens outside the Software AG environment.
This provides a logical representation of the business flow showing the path taken by the transaction through the various cloud components. Also, the processing time for the transaction within each component is visible.
A legend is available to identify the status of a transaction. An example of a business flow is as follows:
To view the component details for a transaction
On the Application dashboard, select the transaction group for which you want to view the details.
On the Business Transaction Details page, click the component from the business flow map for which you want to view the details. The details open in a separate pane and includes the following information:
Field | Description |
---|---|
Tenant name | Provides the tenant of the transaction. |
Status | Provides the status of the transaction. Status can be:
|
Transaction ID | Transaction identifier set by you. |
Cause of failure | Provides the reason for the error to occur. If a transaction is successful, then this parameter is not visible. |
Processing time | Time spent by the transaction within this component. |
Stage | Phase at which the failure has occurred during the execution of a transaction. Stage represents an activity in the life cycle of the transaction. |
Environment group | The mapped environment group where the failure has occured during the execution of the transaction. |
More details | Click this URL to open the details of the component highlighted in the Business Flow Map. Currently, this link is available for B2B and WebMethods Integration Cloud nodes only. Clicking this link redirects you to an instance of the respective component in a new tab. For example, if you click on an Integration Cloud component, you are redirected to the instance of webMethods Integration Cloud in a new tab. |
This is an example of component details pane:
This is an example of the More details page of a workflow:
The Alert page lists all the Rule violations and the Rule list. From the Alert page, you can create rules. These rules are a set of conditions. When these conditions are met by a group of transactions, the rule violation occurs. You can configure a rule such that it triggers an alert in the form of an on-screen notification or email or both. The rules that you create are applicable to all the transactions of a group with which it is associated. For information on creating a rule, see Creating a Rule.
Notification Alerts – Notifications for End-to-End Monitoring are currently shown on the top-right corner of your application screen. For example, when there are two new rule violations, you will see .
Email Alerts – When you create a rule, you have an option to be notified through an email. Whenever a group violates the conditions defined in a rule, you will receive system generated emails with the details of the violation.
Rule Violations – This includes a list of all the rules that are violated by the transactions in a group. For more information, see Working with Rule Violations.
Rule List – This includes the list of all the rules that you create in the application. It also lists all the pre-defined rules available with the application. For more information on pre-defined rules, see Default Rules.
To view the alerts
Click at the top of the screen to open the Rule violations pane. Here, you will see a list of all the new rule violations along with the older ones. If there are new rule violations, you will see a number with
. For example, two new rule violations are indicated by
at the top of the screen.
In the Rule violations pane, click View all to go to the Rule violations section of the Alert page.
In the Rule violations tab, you will see all the rules violations listed in a table format.
Note: The rules are evaluated by the system for every 10 minutes interval.
The application triggers alerts when a group of transactions violates the conditions defined in a rule associated with it.
To create a rule
On the Alert page, click Rule list.
Select Create rule.
Note: The application provides you a set of default rules. For more information, see Default Rules.
In the Create rule dialog box, provide the following details:
In the General information section, provide the details for the following and click Next.
Value | Description |
---|---|
Name | Provide a name for the rule. Note: You cannot change the rule name once you save the rule. |
Description | Provide a meaningful description for the rule. |
Environment | Select the environment that the rule is applicable to, from the drop-down list. You can select only one environment at a time. |
Stage | Provide the stage to which this rule is applicable. For webMethods.io Integration Cloud tenants this can be either of the following options:
However, in case of Software AG cloud tenants, the stage is automatically mapped to the environment. Hence, the stage option is not available during rule creation in Software AG cloud tenants. |
Group | Select the group from the drop-down list. This group list will only contain the groups associated with the tenant selected in the Environment drop-down. Note: Currently, you can associate a rule only to a single group. However, a group can be associated with multiple rules. |
This is an example of the General information section:
In the Rule & Action section, provide the details for the following and click Next.
Value | Description |
---|---|
KPI | This is the key performance index. Select from the following options:
|
Time range | The time range on the basis of which this rule should trigger an alert when violated. Select from the following options:
|
Operator | Select the operator on the basis of which this rule should trigger an alert when violated. Select from the following options:
Note: Value could be a number, percentage, or time and changes on the basis of the KPI you select. |
Value | This field varies on the basis of your selection in the KPI field. The value field changes on the basis of the KPI you select:
|
In the Action section, select your notification preference and click Next. You can select either or both of the following options:
Value | Description |
---|---|
Show app notification | Select this option to get on-screen app notifications. |
Send email | Select this option to subscribe for email alerts whenever a group violates a rule. Provide the recipient email address. You can enter more than one email address separated by a comma. |
This is an example of the Rule & Action section:
The application provides you with a set of pre-defined rules. By default, these rules are disabled. These pre-defined rules are as follows:
Note: Other than the Name, you can modify all the other values as per your requirements.
Name | Description | Group | KPI | Time Range | Operator | Value | Show app notification | Send email |
---|---|---|---|---|---|---|---|---|
Average response time is out of compliance | The average response time for all transactions in last 1 hour is greater than 1000 ms. | All Transactions | Average response time | In last 1 hour | > | 100 (ms) | True | False |
Error count is out of compliance | The error count for all transactions in last 1 hour is more than 100. | All Transactions | Error count | In last 1 hour | > | 10 | True | False |
Error rate is out of compliance | The error rate for all transactions in last 1 hour exceeds 10%. | All Transactions | Error rate | In last 1 hour | > | 10% | True | False |
Note: Each of these default rules will be available for every environment in your tenant. For instance, if you have four environments then the Rule list will show 12 default rules.
The Rule violations page provides details of all the violations. Use this page to search for specific violations and to view all the violations within the various groups for a specified duration. The following table provides you details of all the actions possible in the Rule violations page:
Action | Description |
---|---|
Show all | Use this drop-down list to filter all the violations on the basis of duration or status. The values include:
|
![]() |
This is the refresh button. Click this to refresh the list of rule violations. It shows the time when the list was last refreshed. |
![]() |
This is the column view settings button. Click this to customize the columns you want to view for the rule violations. |
![]() |
Select this drop-down list to filter the rule violations on the basis of group name. You also get a search box when you click this drop-down list. Use this search box to type a group name that you want to search. To search the rule violations across all the available groups, click Select All. Note: You can select more than one group. |
Search by name | Type a rule name in this search box to filter all the rule violations on the basis of rule name. |
To sort the rule violations, click on the header row of any column. You can sort in both ascending and descending order.
The application lists all the rules that you create in the Rule list page. The following table provides you details of all the actions possible in the Rule list page:
Action | Description |
---|---|
Create rule | Use this action button to create a new rule. For more information, see Creating a Rule. |
![]() |
To edit a rule, click this action button on the row of the corresponding rule. Note: You cannot modify the rule name. |
![]() |
To delete a rule, click this action button on the row of the corresponding rule. |
![]() |
Select this toggle button in the Status column to enable or disable a rule. ![]() ![]() Note: When you use this button, you will see corresponding system messages on screen to indicate if a rule is enabled or disabled successfully. |
![]() |
This is the column view settings button. Click this to customize the columns you want to view for the listed rules. In the Show Columns dialog, select the columns you want to view and click Save. |
![]() |
Select this drop-down list to filter the rules on the basis of group name. You also get a search box when you click this drop-down list. Use this search box to type a group name that you want to search. Note: You can select more than one group. |
Search by name | Type a rule name in this search box to filter the rules on the basis of rule name. |
To sort the rules, click on the header row of any column. You can sort in both ascending and descending order.
For a detailed tutorial click Step-by-Step Tutorial
Get a granular view of the transaction data by filtering the transactions based on the environment group and the related stages.
Customers implement their solutions with Software AG Cloud products spanning multiple environments (one product installed in each environment) as well as each environment having stages like Development, Test and Production. This feature helps in providing a more granular view of the transaction data by allowing you to filter the transactions based on the environment group and the related stages. Using the environment group, you can view the transactions that span across more than one environment. Using the stages filter, you can further filter the transaction data based on the different stages the environments of the selected environment group are mapped with.
For example, let us assume there are two environments T1 and T2. A transaction starts from T1 and moves to T2. If the tenant is logged into T1, then the transaction details can be traced as the transaction starts from T1. However, if the tenant is logged into T2, then the transaction details are unavailable. To solve this, we have the Environment groups using which you can get the transaction details.
When you access End-to-End Monitoring from the App Switcher, you will see the following screen providing you the details on how to create environment groups. Select Next to start the environment creation process. This Environment group creation flow window will appear until you disable it using the Don’t show this message again checkbox.
You can skip this step and create the Environment groups at a later stage, by clicking the Later option.
Note: If you are a non-Software AG Cloud user, contact your Application Administrator to create the Environment groups.
If you selected Later on the onboarding screen, you can also create environment groups by following these steps:
Select End-to-End Monitoring. Go to User Profile > Administrator > Manage environment groups.
On the Manage environment groups page, click Create.
In the Create environment group window, provide an environment group name.
Note: You cannot change this name once you create the environment group.
In Filter by stage, select the stages based on which you want to filter the environments. You can select more than one stage. Stages may include the following:
The application auto-populates the environments list based on your selection of Stages.
Select all the environments that you want to include in your group and click Save.
Note: After you select an environment, it will no longer be available for selection when you create another environment group. You can map an environment to a single environment group only.
On successful creation of the environment group, the application shows you a confirmation message and you will see the newly created environment group in the Environment group listing table.
Go to User Profile > Administrator > Manage environment groups.
On the Manage environment groups page, click in the row of the environment group that you want to edit.
Make the required edits.
Note: You cannot edit the environment group name.
After you make the changes, click Save.
Go to User Profile > Administrator > Manage environment groups.
In the Manage environment groups section, click in the row of the environment group that you want to delete.
Note: The deletion of an environment group is a permanent action.
On successful deletion, the application shows you a confirmation message and the environment group listing is updated.
Note:
Environment groups are currently specific to the region of the logged-in environment. Any listing shows environment groups from that specific region only.
After you add an environment to an environment group, it is no longer available for addition to any other environment group. You will receive the below prompt when attempting to create an environment group when all environments have already been mapped.
If you are a webMethods Integration Cloud (WMIC) tenant, the environment creation process is different from the above mentioned steps. Non-Software AG Cloud tenants will need to contact Software AG Cloud Operations with the following questionnaire to assist in the creation of the environment groups.
What is the Account Name?
The account name links the various environment groups together and is usually the customer name or the master account name in Software AG Cloud.
What are the environment groups, and the environments within each group?
Specify a list of environment groups and the list of environments within each environment group and the stages (Development, Test, Pre-Live, Live) each environment belongs to. The list of environments within an environment group are used to filter the information when a given environment group is selected as part of the stages filter. All the environment groups will be linked to the same account name. Environment is also referred to as tenants in webMethods Integration Cloud (WMIC) terminology.
Note:
- A webMethods Integration Cloud (WMIC) tenant can have multiple stages. However, an API Gateway(APIGW) tenant can have only one stage.
- If information regarding environments and environment groups needs to be updated, contact Software AG Global support with information in the above questionnaire.
- Only four pre-defined stages are available for Non-Software AG Cloud tenants:
- Development
- Test
- Pre-Live
- Live
This table explains some valid and invalid configurations. You can use the same table template when you contact Software AG global support. Consider all these configurations for a given Account ID UPS:
Tenant Name | Environment Group | Tenant Type | Stage | Valid/Reason for Invalidation |
---|---|---|---|---|
Tenant1 | Logistics | WMIC |
|
Valid |
Tenant2 | Logistics | APIGW |
|
Valid |
Tenant3 | Sales | WMIC |
|
Valid |
Tenant4 | Sales | APIGW |
|
Valid |
Tenant5 | Logistics | APIGW |
|
Invalid APIGW tenants can be mapped to a single stage only. |
Tenant6 | Logistics | WMIC |
|
Invalid Production stage for WMIC is invalid. Valid options are:
|
Tenant4 | Logistics | APIGW |
|
Invalid Tenant4 is already mapped to the Sales group, hence it is unavailable for the other groups. |
The main dashboard page of End-to-End Monitoring contains a listing of all groups. There are new column entries available on this table as a part of the environment groups and stages feature. To view these entries, click . You can now add the following additional columns for table sorting purposes:
On Dashboard page All groups table, click .
You will notice the additional column entries:
On Alert page Rule violations table, click .
You will notice the same additional column entries as Dashboard page above.
On Alert page Rule list table, click .
You will notice the same additional column entries as Dashboard page above.
After environments have been grouped as per your need, this feature provides filters on the Mast head to conveniently refine information listed on the Dashboard or Alert pages as per the Environment group or Stage in view or Both.
Dashboard page All groups table listing filtered as per Logistic Environment group on the masthead:
Dashboard page All groups table listing filtered as per Development Stage in view on the masthead:
Alert page Rule list table listing filtered as per Sales Environment group on the masthead:
For a detailed tutorial click Step-by-Step Tutorial - Stages Feature
Consider a scenario where the Environment groups in your account are mapped in the following manner:
Environment group | Environment (Stage) |
---|---|
Fin1 | devuhm02 tenant in Development stage |
Logistics | devuhm123 tenant in Development stage prodenv1 tenant in Production stage |
Marketing | envstage1 tenant in PreStage stage |
Now, on the masthead when you select All Environment group and All Stage in view:
You will notice that all dashboard page widgets and All groups listing automatically update information as per the masthead selections.
Next, create a group test2 which is linked to environment devuhm02 and save the group. Change the masthead selections to Fin1 Environment group:
You will now notice that the four widgets reflect new data and the new listing only shows test2 apart from All transactions.
You will notice similar filtering capabilities on Alert page as well:
Next, create a group test1 which is linked to environment prodenv1 and save the group. Change the masthead selections to Logistics Environment group:
You will now notice that the four widgets reflect new data and the new listing only shows test1 apart from All transactions.
Corresponding changes in Alert page:
Consider a scenario where you have an account with groups prior to account mapping. None of these groups are mapped to an environment group or stage. Let us take the below journey of migrating these groups to an environment groups and stages mapped account.
Let us start with creating two groups, Old_Success and Old_fail with the below selected criteria:
You will notice at this stage that the masthead is empty and does not provide any options related to environment groups or stages:
Now let us create environment group mapping with the below criteria:
Environment group | Environment (Stage) |
---|---|
Fin1 | devuhm02 tenant in Development stage |
Logistics | devuhm123 tenant in Development stage prodenv1 tenant in Production stage |
Marketing | envstage1 tenant in PreStage stage |
Let us now select edit button against Old_Success and Old_fail groups and select Development stages for both groups:
The masthead filters are now successfully able to filter the Old_Success and Old_fail groups. Hence, these groups have been successfully migrated to the new mapping.
A hybrid integration consists of a transaction that spans multiple platforms and products including both cloud and on-premises setups. Software AG’s End-to-End Monitoring allows you to track your integrations from the cloud through to your on-premises installation to give you a complete view of your integration landscape regardless of the topology of your IT infrastructure.
A hybrid integration use case is one where the transaction starts in a Software AG cloud product webMethods.io Integration and this cloud product calls a service on an on-premises Software AG product Integration Server to complete the transaction.
Hybrid monitoring is the capability that monitors the end-to-end hybrid transaction and displays it in the End-To-End Monitoring user interface.
The End-To-End Monitoring node on the Software AG installer allows you to setup the on-premises monitoring agents that enable the end-to-end transaction monitoring of hybrid integration.
Note:
For this release hybrid monitoring is supported for transactions from Integration Server and Microservices Runtime. In the future we will add support for all our products and even your own products, through an open tracing API.
Install Integration Server or Microservices Runtime, if not done already. You will need to have hybrid transaction configured for Integration Server 10.7 and above versions, and Microservices Runtime 10.7 and above versions.
Configure webMethods Cloud tenant connection in the Integration Server Administration UI. Prior to version 10.11, this configuration supported a connection to just a single webMethods.io Integration tenant. Versions 10.11 and above webMethods Integration Server, provides the capability to connect to multiple webMethods.io Integration tenants from the same Integration Server installation supporting hybrid integration transactions from multiple tenants. This functionality is also available on version 10.7 with advanced fixes. Use one of the two configuration options provided below, based on if this fix release is available at the time of using this functionality.
In the Integration Server Administration UI, navigate to webMethods Cloud >> Tenant Connections.
Configure the User Name, Password and webMethods Cloud URL.
In webMethods Integration Server, navigate to webMethods Cloud >> Tenant Connections. Select the pre-existing tenant connection named default, provide the tenant details and Save. You can use this default connection to create your account.
If a custom tenant connection is being used, create a new tenant connection with an alias name and tenant details.
Following hybrid monitoring configurations are required in Integration Server and Microservices Runtime.
Make the following selections during Software AG Installer setup. If you have Integration Server pre-installed, only select End-to-End Monitoring.
If you are on release version 10.11, select End-to-End Monitoring Core 10.11 under Infrastructure.
Using Software AG update manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core, Integration Server and Microservices runtime plug-in fix.
Post fix installation, copy the plugin jar file:
<INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
<INSTALL-DIR>\IntegrationServer\lib\jars\
Copy log4j properties file:
<INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
<INSTALL-DIR>\IntegrationServer\instances\default\
Add the following lines at the end of file:
<INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf
Microsoft Windows
wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\
Linux
wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/
Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf
, edit the property Dlog4j.configurationFile
and append e2ecustomlog4j2.properties
to it.
Microsoft Windows:
Existing
wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties"
Appended
wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"
Linux:
Existing
wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties"
Appended
wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"
If multiple tenant connection is supported in webMethods Integration Server, and a custom tenant connection is being used then:
<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config
agent.onprem_multitenant_alias = ${SW_AGENT_ONPREM_MULTITENANT_ALIAS:default}
default
with tenant connection name.Note:
- When a custom tenant connection is not used, End-to-End Monitoring uses default tenant connection value for
agent.onprem_multitenant_alias
inagent.config
. You need to save default tenant connection details in webMethods Integration Server when there is no custom connection.- Hybrid data transfer takes place within tenants of one given region only. When multiple tenant connections are created for different tenants belonging to the same region, specifying one of the tenant connection names for
agent.onprem_multitenant_alias
inagent.config
is sufficient to trace hybrid transactions of other tenants belonging to the same region.
Make the following selections during Software AG Installer setup. If you have Microservices Runtime pre-installed, only select End-to-End Monitoring.
Using Software AG update manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core, Integration Server and Microservices runtime plug-in fix.
Post fix installation, copy the plugin jar file:
<INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
<INSTALL-DIR>\IntegrationServer\lib\jars\
Copy log4j properties file:
<INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
<INSTALL-DIR>\IntegrationServer\
Add the following lines at the end of file:
Microsoft Windows
<INSTALL-DIR>\IntegrationServer\bin\setenv.bat
set JAVA_UHM_OPTS=-javaagent:"../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/" -Xbootclasspath/a:"../E2EMonitoring/agent/uhm-apm-agent.jar"
set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHM_OPTS%
set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% -Dlog4j.configurationFile="./e2ecustomlog4j2.properties"
Linux
<INSTALL-DIR>\IntegrationServer\bin\setenv.sh
JAVA_UHM_OPTS="-javaagent:../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/ -Xbootclasspath/a:../E2EMonitoring/agent/uhm-apm-agent.jar"
JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} ${JAVA_UHM_OPTS}"
JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} -Dlog4j.configurationFile=./e2ecustomlog4j2.properties"
There are scenarios when multiple runtimes like multiple Integration Server cluster nodes or Integration Server installations are pointing to the same end-to-end agent jar file. In such situations, you need to make hybrid monitoring configuration changes in each runtime. This is applicable to both Integration server and Microservices Runtime.
You now need to choose the on-premise services that you wish to trace in End-to-End Monitoring. This is done by changing the auditing setting of each service. You can choose to trace both top level services or private sub-level services depending on how much information you want to trace. However, take caution not to trace too much as this will consume your cloud storage and effectively reduce the number of historical transactions that you can view. It can also potentially impact the performance of your integrations and increase network traffic. Note that you do not need to trace every sub-service to ensure that the transaction is monitored*, any errors are propagated up the chain until they are caught by an audited service.
Note:
- * Excludes asynchronous services that are invoked using messaging or sub-threads. In which case you need to have these services audited. They are considered top level services in their own right.
- This setup piggybacks the on-premises service auditing feature.
In Software AG Designer, for the on-premises Integration Server services associated with hybrid transaction including nested services, enable auditing with the following values:
Enable auditing: Always
Log on: Error, Success and start
Here is an example of a hybrid transaction in the End-to-End Monitoring user interface:
Enabling auditing in numerous services of Integration Server can be tedious and time consuming when you use the Software AG Designer. You can now also decide what services to audit using the extended settings of your webMethods runtime. This feature compliments the auditing options that can be set for each service individually. The advantage is that you can have different logging levels between different environments without having to modify service settings. This significantly simplifies your service auditing.
In Integration Server settings, select Extended and click Edit Extended Settings.
The following new keys are now available for auditing purposes. Use these keys to set audit levels.
Include service names | Exclude service names |
---|---|
watt.server.audit.service.include |
watt.server.audit.service.exclude |
Note:
Any change in these values is refreshed within ten minutes.
To include service names with exact match, use the following syntax. After you set this extended property in Integration Server, these services are included for tracing. This list is captured during the first invocation of the service.
Syntax | Description |
---|---|
watt.server.audit.service.include=svc1,svc2 |
svc1 and svc2 are service names in this example where svc1 and svc2 are the fully qualified service names including the folder namespace. |
To include service names starting with a certain text, use the following syntax. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service. You can also include partial names to specify services sharing the same name-space.
Syntax | Description |
---|---|
watt.server.audit.service.include=svc1*,svc2* |
All service names starting with svc1 and svc2 are traced, where svc1 and svc2 could be jc.helloworld:* and acme.ordermgmt:pub:* respectively. |
When you are tracing many services, using a file makes it easier. You can also choose to specify a file if you have a long list of services to specify. To include service names through a file, use the following syntax. Save your file with a .txt extension. List your service names in include.txt
.
Place each service name on a separate line in the file. Use #
to comment out. Save the file in the relative or absolute path as per the table below. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.
Syntax | File location |
---|---|
watt.server.audit.service.include=$file{./include.txt} |
<install-dir>/IntegrationServer/instances/default |
watt.server.audit.service.include=$file{c:/home/include.txt} |
c:/home/include.txt |
Certain keywords are provided to simplify auditing. The keyword $toplevel
is provided to simply trace all services that are called from an external client. Child services will only be invoked in this case if they are flagged individually for audit. This property can also be used in a comma separated list with all of the aforementioned features. To include all top level services, use the following syntax. When you include top level services, it is recommended to use $wmservices
in the exclude list. This reduces the transaction logs overhead of all background webMethods.io services in the e2etrace.log
file.
Syntax | Description |
---|---|
watt.server.audit.service.include=$toplevel |
All top level services are traced. |
Note: We recommend that you use the keyword
$wmservices
with your exclude list if you are using this property, in order to avoid too much tracing.
To exclude service names, use the following syntax. After you set this extended property in Integration Server, these services are excluded from tracing. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.
Syntax | Description |
---|---|
watt.server.audit.service.exclude=svc3,svc4 |
svc3 and svc4 are service names in this example. |
When you are tracing many services, using a file makes it easier. To exclude service names using a file, use the following syntax in the given location. Save your file with a .txt extension. List your service names in exclude.txt
.
Place each service name on a separate line in the file. Use #
to comment out. Save the file in the relative or absolute path as per the table below. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.
Syntax | File location |
---|---|
watt.server.audit.service.exclude=$file{./exclude.txt} |
<install-dir>/IntegrationServer/instances/default |
watt.server.audit.service.exclude=$file{c:/home/exclude.txt} |
c:/home/exclude.txt |
To exclude service names through the agent.config
file, use the following syntax. When you set this property, all service names in the agent.config
file and agent.exclude_serv_list
property are excluded from tracing.
Syntax | Description |
---|---|
watt.server.audit.service.exclude=$wmservices |
Excludes services through the use of agent.config . |
Example
agent.exclude_serv_list=${SW_AGENT_EXCLUDE_SERV_LIST:wm.,com.,pub.} |
wmservices represents any webMethods services. These service names start with wm. or pub. or com. . However, you can update this list with namespaces of your choice. |
These key properties can be combined. Following are some simple combination examples.
Combination example | Description |
---|---|
watt.server.audit.service.include=$toplevel,svc1,svc2,$file{./include.txt} |
All top level services are traced, along with svc1 and svc2, and services listed in the file include.txt . |
watt.server.audit.service.exclude=svc1,svc2,$wmservices,${file{./exclude.txt} |
svc1 and svc2 are excluded from tracing. Services listed in exclude.txt are also excluded from tracing. Services listed in agent.config file and agent.exclude_serv_list property are also excluded from tracing. |
Important:
- What is eventually traced is always an effective difference of all services listed in the following two keys :
watt.server.audit.service.include
andwatt.server.audit.service.exclude
.- However, the include property precludes the exclude property. The values in the include list will always take precedence even if a match is found in the exclude list. These properties are complimentary to the service audit properties and do not replace them, unless you have changed the agent property
agent.global_incl_serv_flag
.
Following are the configurable properties in the End-to-End Monitoring agent available at <INSTALL-DIR>\E2EMonitoring\agent\config
Property | Description |
---|---|
agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000} |
This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes for testing purposes. |
collector.establish_cloud_communication=${SW_AGENT_COLLECTOR_ESTABLISH_CLOUD_COMMUNICATION:true} |
This property defines connectivity with the cloud collector. Default value is true, which means data is sent to the cloud collector. |
agent.onprem_multitenant_alias |
Hybrid data transfer takes place within tenants of one given region only. When multiple tenant connections are created for different tenants belonging to the same region, specifying one of the tenant connection names for agent.onprem_multitenant_alias in agent.config is sufficient to trace hybrid transactions of other tenants belonging to the same region. However, this is required only if you have multiple tenants in the same region and you wish to route the trace using a specific tenant. Also note that, End-to-End Monitoring can only trace to one tenant but is able to determine the original tenant for each trace. |
agent.global_incl_serv_flag=${SW_AGENT_GLOBAL_INCL_SERV_FLAG:GLOBAL_OR_LOCAL} |
This property allows you to select between Software AG Designer audit property or Integration Server extended settings WATT properties based tracing. Default Value is GLOBAL_OR_LOCAL . Possible values are GLOBAL_OR_LOCAL , GLOBAL , LOCAL .
|
agent.exclude_serv_list=${SW_AGENT_EXCLUDE_SERV_LIST:wm.,com.,pub.} |
This property is considered only if WATT exclude is set with $wmservices . $wmservices represents any webMethods services. These service names start with wm. or pub. or com. , and are excluded from tracing. |
agent.watt_check_interval=${SW_AGENT_WATT_CHECK_INTERVAL:600000} |
Any changes in WATT property or the content in include or exclude files is refreshed in cache at this interval. Default value is 600000ms=10minutes. |
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:uhm.e2em-hybrid-az-us.webmethods.io:443} |
This property is commented out by default. Uncomment this property for hybrid transactions. SW_AGENT_COLLECTOR_BACKEND_SERVICES Update the property value based on tenant type and region. This is applicable to both AWS and Azure tenants. Note: From release 10.12 onwards, you will need to update this property for both AWS and Azure tenants, in order to enable hybrid monitoring. |
SW_AGENT_COLLECTOR_BACKEND_SERVICES
values are based on the Software AG Cloud Regions :
Azure End-Point | Value |
---|---|
US2 East Azure | uhm.e2em-hybrid-az-us.webmethods.io:443 |
AU1 Australia East Azure | uhm.e2em-hybrid-az-au.webmethods.io:443 |
EU3 West Azure | uhm.e2em-hybrid-az-eu.webmethods.io:443 |
AWS End-Point | Value |
---|---|
US1 Oregon AWS | uhm.e2em-hybrid-aw-us.webmethods.io:443 |
EU2 Frankfurt AWS | uhm.e2em-hybrid-aw-eu.webmethods.io:443 |
AU2 Sydney AWS | uhm.e2em-hybrid-aw-au.webmethods.io:443 |
Note:
Hybrid Monitoring data transfer from on-premises Integration Server or Microservices Runtime to cloud takes place through the tenant endpoint configured under webMethods Cloud Tenant Connections on port 11800.
<InstallDir>/IntegrationServer/instances/default/logs
<InstallDir>/IntegrationServer/logs
e2etraces.log
Go to the file e2ecustomlog4j2.properties
at the following location:
Integration Server
<INSTALL-DIR>\IntegrationServer\instances\default
Microservices Runtime
<INSTALL-DIR>\IntegrationServer
Apply DEBUG logging level to the following properties:
appender.FILE_OUT.filter.threshold.level = DEBUG
appender.FILE_OUT1.filter.threshold.level = DEBUG
appender.FILE_OUT2.filter.threshold.level = DEBUG
logger.20.level = DEBUG
logger.21.level = DEBUG
logger.22.level = DEBUG
Here is a sample of the on-premises transaction trace:
2021-08-06T19:49:50,328 DEBUG [e2eAgentAppender] -
[ENTRY]SpanInfo [traceId=[26655816.49441.45841902585488], segmentId:26655969.276.162825958992328, spanId:0, serviceName=forTesting:javaAsyncService1, startTime=1628259589923, duration=4ms, component=ONPREMISEIS, tenant=VMWODEPLAT01:5555, status=Success, requestID=5745b59f-089a-4b13-9038-f82188d68a32, parentLandscape=WMIO]
For a detailed tutorial, see Step-by-Step Tutorial - Hybrid Monitoring
End-to-End Monitoring supports hybrid monitoring involving on-premises API gateway version 10.11 and above.
Note:
On-premises Integration Server and API Gateway plugins cannot be installed in the same installation. This is a known limitation.
Using Software AG Installer install on-premises API Gateway, End-to-End Monitoring Core and API Gateway Plugin.
Using Software AG update manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core and API Gateway plug-in fix.
webMethods Integration Server supports multiple tenant connections with 10.11 and above versions. This functionality is also available on version 10.7 with advanced fixes. Configure your tenant connection as per either of the below 3 arrangements:
webMethods Integration Server without multiple tenant connection support
In the Integration Server Administration UI, navigate to webMethods Cloud >> Tenant Connections.
webMethods Integration Server with multiple tenant connection support
Default tenant connection - Select the pre-existing tenant connection named default, provide the tenant details and Save. You can use this default connection to create your account.
New custom tenant conection - If you need to use a custom tenant connection, create a new tenant connection with a different alias name. This custom tenant connection is to be used while creating your webMethods cloud account.
Copy file: uhm-onpremise-is-http-plugin.jar
<INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-http-plugin.jar
<INSTALL-DIR>\IntegrationServer\lib\jars\
Copy file: uhm-api-onpremise-plugin.jar
<INSTALL-DIR>\E2EMonitoring\agent\plugins\ uhm-api-onpremise-plugin.jar
<INSTALL-DIR>\IntegrationServer\instances\default\packages\WmAPIGateway\ code\jars
Copy file: e2ecustomlog4j2.properties
<INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
<INSTALL-DIR>\IntegrationServer\instances\default\
Add the following lines at the end of file:
<INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf
Microsoft Windows
wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\
Linux
wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/
Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf
, edit the property Dlog4j.configurationFile
and append e2ecustomlog4j2.properties
to it.
Microsoft Windows:
Existing
wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties"
Appended
wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"
Linux:
Existing
wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties"
Appended
wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"
Open the file: <INSTALL-DIR>\E2EMonitoring\agent\config\agent.config
, and edit below properties:
agent.service_name=${SW_AGENT_NAME: ONPREMISEIS}
with agent.service_name=${SW_AGENT_NAME:ONPREMISEAPI}
agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:default}
with agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:<TenantName}
Property | Description |
---|---|
agent.service_name=${SW_AGENT_NAME:ONPREMISEIS} |
Indicates service name that monitors product calls. Set SW_AGENT_NAME to ONPREMISEAPI , to monitor on-premise API Gateway calls. |
agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:default} |
Indicates cloud tenant from which transactions to on-premise API gateway takes place. Tenant name is the value for SW_AGENT_ONPPREM_TENANT_ID . |
agent.onprem_multitenant_alias = ${SW_AGENT_ONPREM_MULTITENANT_ALIAS:default} |
Indicates name of the custom tenant connection used in Integration server. Custom tenant connection name created in the Integration Server is the value for the SW_AGENT_ONPREM_MULTITENANT_ALIAS . This property is commented out by default, un-comment it if custom tenant connection is used in the Integration Server. |
agent.apigw_tag=${SW_AGENT_APIGW_TAG:<custom tag>} |
End-to-End Monitoring, by default, monitors all the on-premises webMethods API Gateway calls with tag e2em. If you need to monitor an API containing custom tags , add this property in agent.config with custom tag value. To enable tracing, add the e2em tags to the webMethods API Gateway API, in case you do not have them already. |
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES: uhm.e2em-hybrid-az-us.webmethods.io:443} |
This property is commented out by default. Uncomment this property for hybrid transactions. SW_AGENT_COLLECTOR_BACKEND_SERVICES Update the property value based on tenant type and region. This is applicable to both AWS and Azure tenants. Note: From release 10.12 onwards, you will need to update this property for both AWS and Azure tenants, in order to enable hybrid monitoring. |
SW_AGENT_COLLECTOR_BACKEND_SERVICES
values are based on the Software AG Cloud Regions :
Azure End-Point | Value |
---|---|
US2 East Azure | uhm.e2em-hybrid-az-us.webmethods.io:443 |
AU1 Australia East Azure | uhm.e2em-hybrid-az-au.webmethods.io:443 |
EU3 West Azure | uhm.e2em-hybrid-az-eu.webmethods.io:443 |
AWS End-Point | Value |
---|---|
US1 Oregon AWS | uhm.e2em-hybrid-aw-us.webmethods.io:443 |
EU2 Frankfurt AWS | uhm.e2em-hybrid-aw-eu.webmethods.io:443 |
AU2 Sydney AWS | uhm.e2em-hybrid-aw-au.webmethods.io:443 |
You will have these property changes in agent.config
take effect only after restarting the Integration Server.
Note:
- To see segment traces of on-premises API gateway transaction in logs set all log levels to DEBUG in
<INSTALL-DIR>\IntegrationServer\instances\default\e2ecustomlog4j2.properties
- End-to-End Monitoring can trace transactions belonging to single tenant, even if end-to-end transactions involve multiple API gateway tenants at a time.
- Hybrid monitoring works with on-premises Integration Server as well as on-premises API gateway. However, End-to-End Monitoring does not support usage of both products together as part of Hybrid integration.
- End-to-End Monitoring, by default, monitors all the on-premises webMethods API Gateway calls with tag e2em.