Using End-to-End Monitoring

Dashboard

The dashboard for End-to-End Monitoring gives you a collective view of all the business transactions carried out within your cloud platform. The default Dashboard view includes the transaction widgets, Transactions list and Groups list.

– An alert icon to notify you of any new transaction notifications.

– User help that includes the help content for this application.

– Use the link provided here to log off from the application.

Time range – Below is the drop-down list for time range selection. By default, it is set to half an hour prior from the current system time. For example, if current system time is 3:30 PM, then the time range shows as 3 PM to 3 30 PM. Click next to the time range to select a specific duration.

Transaction widgets

The following widgets are available on the dashboard:

• NETWORK VIEW – This is a graphical representation of all the transactions across all the supported nodes. It shows the transactions flowing through each runtime as well as the status of each runtime for the selected time range. This widget also provides you a summary of all the transactions processed based on the following parameters:

  • Total

  • Failed

  • Success

  An example of the NETWORK VIEW widget is as follows:

  

• TOP 5 GROUPS BY ERROR RATE – This is a graphical representation which shows the top 5 groups based on the error rate. The values are displayed in percentage. The All Transactions group is excluded from this widget.

  An example of this widget is as follows:

  

• TOP 5 RULES VIOLATED – This is a graphical representation which shows the top 5 rules that are violated the maximum number of times. The X-axis of this graph shows a count of the number of times these rules are violated. The result shown on this widget corresponds to the selected duration. For example, if you have set the duration as Last 30 minutes and a rule is violated 1 hour before, then that rule violation will not appear on this widget. An example of this widget is as follows:

  

• HELP TOPICS – This provides you quick access to the help content for some of the commonly used topics.

• All groups – This lists all the transaction groups you create using the application and also the default All Transactions group. At the group level, you can see the following parameters for the selected time range:

  Note: You can customize the parameters to view by clicking next to Create Group.

  • Name – Name of the group.

  • Total transactions – The total number of transactions for each group.

  • Avg. time (ms) – Average time of execution for all the transactions within a group.

  • Error rate (%) – The rate of error in percentage for all the transactions within a group.

  • Success rate (%) – The rate of success in percentage for all the transactions within a group.

  • Environment – Environments for all transactions within a group.

  • Stage – Stages for all transactions within a group.

  • Environment group – Environment groups created for all transactions within a group.

  • Transaction ID - Custom transaction ID set by you for better traceability and usability.

Custom transaction tracing for enhanced filtering

You now have the ability to specify a custom transaction ID as part of your FlowServices. This persists as part of your transaction trace stored in End-to-End Monitoring. This feature allows you to reference or search a transaction based on your own identifier such as order ID or shipping reference. The identifier does not necessarily have to be unique to the transaction. The identifier can be used to find all distinct transactions related to a common asset such as a customer or order.

Note: You can use the custom identifier to filter for related transactions from the updated End-to-End Monitoring dashboard using the search field.

Finer details on custom transaction IDs

• If you have not defined a custom transaction ID, the Business flow map shows undefined under Transaction ID.

  

• A single transaction can have multiple identifiers. For instance, developers can choose to associate different transaction IDs with different services belonging to the same transaction like when a batch file splits out into separate records. In this case anyone of the identifiers can be used to filter and these are shown in the user interface as a comma separated list.

  

• If your transaction belongs to a particular Software AG cloud product node, click on the node in the Business flow map of the transaction details page to view the custom transaction ID.

• In end-to-end transactions involving multiple Software AG cloud products, transaction ID for a particular product’s transaction can be viewed by clicking on the product node in the Business flow map of the transaction details page.

• Custom transaction IDs can be set programmatically with the service pub.flow:setCustomContextID from:

  • webMethods.io Integration FlowServices.
  • On premises Integration Server FlowServices.
    
    

• Viewing the transactions list now provides Transaction ID option in Show Columns.

  

• Filtering the business transactions now lets you filter using Transaction ID too.

  

Creating a Group of Transactions

Creating a group of transactions allows you to categorize the transactions based on your requirements.

1. In the Dashboard page, select Groups tab and click Create Group.

2. In the Create group dialog box, provide the following details:

   Parameter	Description
   Name	Type a name for this group. Note: The group name cannot be edited after you save this form.
   Environment	Provide the environment where this group will be created.
   Stage	Provide the stage under which this group will be created. For webMethods.io Integration Cloud tenants this can be either of the following options: Development Test Pre-Live Live However, in case of Software AG cloud tenants, the stage is automatically mapped to the environment. Hence, the stage option is not available during group creation in Software AG cloud tenants.
   Status	Group the transactions on the basis of their status. Note: Click Select All to select all the listed options. It can be a combination of: Success Failed
   Starts with	Group the transactions on the basis of the application from where the business flow starts. You can choose one or more products when creating the group. For example, you can have a group where you have one transaction starting from webMethods.io API and another transaction starting from webMethods.io Integration. Note: Click Select All to select all the listed options. You can select from the following: webMethods.io API webMethods.io B2B Note: Currently, End-to-End Monitoring support for B2B is very basic. You may not get complete trace for all types of B2B transactions. For example, transactions invoked by the AS2 protocol may not provide you the complete trace results. webMethods.io Integration Integration Cloud webMethods Integration Server
   Duration (ms)	Group the transactions on the basis of their execution time in milliseconds. You can select from the following: – Select this operator to group the transactions that have an execution time which is lesser than or equal to the value specified here. – Select this operator to group the transactions that have an execution time which is greater than or equal to the value specified here. – Select this operator to group the transactions that have an execution time which is between the values specified here. For example, if you specify 20 and 30 (ms), then both the values 20 and 30 are also taken into consideration.
   Product(s) involved	Group the transactions on the basis of the product(s) involved in a business transaction. Use the drop-down list to select one or more products. The group includes all the transactions for the selected product. If you select more than one product, it will list the transactions for the standalone products along with the transactions that involve both the products. For example, if you select webMethods.io API and webMethods.io Integration, the group will include the following: Note: Click Select All to select all the listed options. all transactions that involve only webMethods.io Integration all transactions that involve only webMethods.io API all transactions that involve both webMethods.io Integration and webMethods.io API.
   Transaction name(s)	Group the transactions on the basis of their names. The transaction name relates to an asset that produces transactions as defined by the chosen product types. For example, webMethods.io Integration generates transactions based on the name of services being run, whereas API Gateway generates transactions based on the name of the associated API. You can choose more than one transaction name and it can either be a complete match or a partial match. Note: Click X to remove the added names. Transaction name autocomplete list is based on the products selected in the Starts with criteria. For example, if you have selected webMethods.io API Gateway and webMethods.io B2B in Starts with, then the autocomplete list will contain all transactions starting within webMethods.io API Gateway or webMethods.io B2B. Transaction name autocomplete list is also based on the products selected in the Products involved criteria. For example, if you have selected webMethods.io API Gateway and webMethods.io B2B in Products involved, then the autocomplete list will contain all transactions passing through webMethods.io API Gateway or webMethods.io B2B.
   Error Message(s)	Group the transactions on the basis of an error message. Click + to add more than one entry.

3. Click Save.

Viewing the Transactions List

1. On the Dashboard page, select the group for which you want to view the transactions. On the Transactions list page, you will see all the transactions for the selected time frame and the set filter(s).

2. Click on the top-right corner of the screen to open the Show Columns dialog box.

3. Select the parameters for which you want to view the transaction details and click Save. The transactions list shows the following parameters:

   Parameter	Description
   Status	Status of the transaction. Currently, we have the following states: – Transaction successful. – Transaction failed.
   Name	Name of the transaction. This name is based on the starting point of the transaction. For example, if the transaction starts from webMethods.io API, then this name would be the name of the API.
   Duration (ms)	Total time taken by the transaction to complete.
   Starts with	The application from which the transaction starts. For example, webMethods.io Integration.
   Started on	Start time for the transaction.
   Error message	Message with which the transaction failed.
   Trace ID	This is a unique identifier for the transaction.
   Transaction ID	This is a custom transaction ID set by you for better traceability and usability.

This is an example of the transaction list view:

You can also filter the transactions from this page. For more information, see Filtering the Business Transactions.

Editing a Transactions Group

1. On the Dashboard page, select in the row for the transaction that you want to edit.

   Note: Editing a transaction group impacts all the rules associated with it.

2. Make the necessary changes and click Update.

Deleting a Transactions Group

1. On the Dashboard page, select in the row for the transaction that you want to delete.

   Note: Deleting a transaction group deletes all the rules and the rule violations associated with it.

2. Click Delete. On successful deletion, you will see a confirmation message on the screen.

Filtering the Business Transactions

You can filter the business transactions within any group to view a custom set of transactions.

Note: Filters that were set during group creation are already applied when you open the group.

To filter the transactions

1. On the Transactions list page, click as shown in the following example:
   
   

2. In the Filter dialog box, you can choose from the following filters:

   Note: Filters set during group creation are greyed out and not available for selection.

   Value	Description
   Status	Select the transaction status. You can select from the following and click Save: Success – Lists all the completed business transactions. Failed – Lists all the failed transactions.
   Name	Type the name of the business transaction and click Save. To add more than one transaction name, click +.
   Duration	Select from the following operators using the drop-down list and provide a valid time in milliseconds: – Select this operator to filter the transactions that have an execution time which is lesser than or equal to the value specified here. – Select this operator to filter the transactions that have an execution time which is greater than or equal to the value specified here. – Select this operator to group the transactions that have an execution time which is between the values specified here. For example, if you specify 20 and 30 (ms), then both the values 20 and 30 are also taken into consideration.
   Transaction starts with	Filter the transactions on the basis of the application from where the business flow starts. Select the checkbox next to the product. You can select more than one product.
   Error message	Filter the transactions based on the error message with which it failed. Type the error message and click Save. For example, server not found. You can filter based on more than one error message by clicking+next to the error message box.
   Product(s) involved	Filter the transactions based on the products that are involved with this transaction. Select the checkbox next to the product and click Save. You can select more than one product.
   Trace ID	Filter the transactions based on this unique identifier associated with a transaction. Enter the trace ID and click Save. You can filter based on more than one Trace ID by clicking + next to the trace ID box.
   Transaction ID	Filter the transactions based on the custom transaction identifier set by you. Enter the transaction ID and click Save.

3. The applied filters show up on the top of the page as shown in the following example:
   
   

   You can close a particular filter by selecting the filter and clicking X adjacent to the filter name.

4. After you apply the filters and retrieve the list of transactions, you can further sort the list by clicking the column headings. Sort in ascending or descending order by toggling the column header.

Note: When you add a custom transaction ID to your end-to-end business transaction, you can filter based on the custom transaction ID from the End-to-End Monitoring user interface. This is useful when you need to tag a specific transaction, beyond the filtering capability offered by service name, trace ID, status or duration.

Business Transaction Details Page

The Business Transaction Details page provides the following information about the transaction:

This is an example of a business transactions details page:

Duration

Duration is the total time taken by a transaction from the time a client initiates a request to the time a response is received by the client. Let us consider the following scenario where the duration for a transaction to complete is 200ms. In the following image, 200ms is the time difference between INT22 and INT11. The time taken by the transaction within the API component which is 50ms is already taken into account as part of this 200ms value. End-to-End Monitoring records the duration only from the time a client request enters the first component of the cloud platform (INT11) and the time the response exits the last component of the cloud platform (INT22).

Note: The application does not note the time the client sends the request (T1) and the time the client receives the response (T2), as this happens outside the Software AG environment.

Business Flow Map

This provides a logical representation of the business flow showing the path taken by the transaction through the various cloud components. Also, the processing time for the transaction within each component is visible.

A legend is available to identify the status of a transaction. An example of a business flow is as follows:

Viewing the Component Details for a Transaction

To view the component details for a transaction

1. On the Application dashboard, select the transaction group for which you want to view the details.

2. On the Business Transaction Details page, click the component from the business flow map for which you want to view the details. The details open in a separate pane and includes the following information:

   Field	Description
   Tenant name	Provides the tenant of the transaction.
   Status	Provides the status of the transaction. Status can be: Success Failed
   Transaction ID	Transaction identifier set by you.
   Cause of failure	Provides the reason for the error to occur. If a transaction is successful, then this parameter is not visible.
   Processing time	Time spent by the transaction within this component.
   Stage	Phase at which the failure has occurred during the execution of a transaction. Stage represents an activity in the life cycle of the transaction.
   Environment group	The mapped environment group where the failure has occured during the execution of the transaction.
   More details	Click this URL to open the details of the component highlighted in the Business Flow Map. Currently, this link is available for B2B and WebMethods Integration Cloud nodes only. Clicking this link redirects you to an instance of the respective component in a new tab. For example, if you click on an Integration Cloud component, you are redirected to the instance of webMethods Integration Cloud in a new tab.

This is an example of component details pane:

This is an example of the More details page of a workflow:

Alert

The Alert page lists all the Rule violations and the Rule list. From the Alert page, you can create rules. These rules are a set of conditions. When these conditions are met by a group of transactions, the rule violation occurs. You can configure a rule such that it triggers an alert in the form of an on-screen notification or email or both. The rules that you create are applicable to all the transactions of a group with which it is associated. For information on creating a rule, see Creating a Rule.

• Notification Alerts – Notifications for End-to-End Monitoring are currently shown on the top-right corner of your application screen. For example, when there are two new rule violations, you will see .

• Email Alerts – When you create a rule, you have an option to be notified through an email. Whenever a group violates the conditions defined in a rule, you will receive system generated emails with the details of the violation.

• Rule Violations – This includes a list of all the rules that are violated by the transactions in a group. For more information, see Working with Rule Violations.

• Rule List – This includes the list of all the rules that you create in the application. It also lists all the pre-defined rules available with the application. For more information on pre-defined rules, see Default Rules.

Viewing Alerts

To view the alerts

1. Click at the top of the screen to open the Rule violations pane. Here, you will see a list of all the new rule violations along with the older ones. If there are new rule violations, you will see a number with . For example, two new rule violations are indicated by at the top of the screen.

2. In the Rule violations pane, click View all to go to the Rule violations section of the Alert page.

3. In the Rule violations tab, you will see all the rules violations listed in a table format.

Creating a Rule

Note: The rules are evaluated by the system for every 10 minutes interval.

The application triggers alerts when a group of transactions violates the conditions defined in a rule associated with it.

To create a rule

1. On the Alert page, click Rule list.

2. Select Create rule.

   Note: The application provides you a set of default rules. For more information, see Default Rules.

3. In the Create rule dialog box, provide the following details:

   • In the General information section, provide the details for the following and click Next.
     
     

     Value	Description
     Name	Provide a name for the rule. Note: You cannot change the rule name once you save the rule.
     Description	Provide a meaningful description for the rule.
     Environment	Select the environment that the rule is applicable to, from the drop-down list. You can select only one environment at a time.
     Stage	Provide the stage to which this rule is applicable. For webMethods.io Integration Cloud tenants this can be either of the following options: Development Test Pre-Live Live However, in case of Software AG cloud tenants, the stage is automatically mapped to the environment. Hence, the stage option is not available during rule creation in Software AG cloud tenants.
     Group	Select the group from the drop-down list. This group list will only contain the groups associated with the tenant selected in the Environment drop-down. Note: Currently, you can associate a rule only to a single group. However, a group can be associated with multiple rules.

   This is an example of the General information section:

   

   • In the Rule & Action section, provide the details for the following and click Next.
     
     

     Value	Description
     KPI	This is the key performance index. Select from the following options: Error count Error rate Average response time
     Time range	The time range on the basis of which this rule should trigger an alert when violated. Select from the following options: In last 30 mins In last 1 hour In last 12 hours In last 24 hours
     Operator	Select the operator on the basis of which this rule should trigger an alert when violated. Select from the following options: Less than (<) – lesser than the value specified in the Value field. Greater than (>) – greater than the value specified in the Value field. Equals to (=) – equal to the value specified in the Value field. Note: Value could be a number, percentage, or time and changes on the basis of the KPI you select.
     Value	This field varies on the basis of your selection in the KPI field. The value field changes on the basis of the KPI you select: For Error count, it is set to Value(number). For Error rate, it is set to Value(%). For Average response time, it is set to Value(ms).

   • In the Action section, select your notification preference and click Next. You can select either or both of the following options:

   Value	Description
   Show app notification	Select this option to get on-screen app notifications.
   Send email	Select this option to subscribe for email alerts whenever a group violates a rule. Provide the recipient email address. You can enter more than one email address separated by a comma.

   This is an example of the Rule & Action section:

   

   • In the Summary section, verify all the details provided by you and click Save.

   

Default Rules

The application provides you with a set of pre-defined rules. By default, these rules are disabled. These pre-defined rules are as follows:

Note: Other than the Name, you can modify all the other values as per your requirements.

Note: Each of these default rules will be available for every environment in your tenant. For instance, if you have four environments then the Rule list will show 12 default rules.

Working with Rule Violations

The Rule violations page provides details of all the violations. Use this page to search for specific violations and to view all the violations within the various groups for a specified duration. The following table provides you details of all the actions possible in the Rule violations page:

Action	Description
Show all	Use this drop-down list to filter all the violations on the basis of duration or status. The values include: All open All resolved Last 1 hour Last 12 hours Today Yesterday Last 7 days
	This is the refresh button. Click this to refresh the list of rule violations. It shows the time when the list was last refreshed.
	This is the column view settings button. Click this to customize the columns you want to view for the rule violations.
	Select this drop-down list to filter the rule violations on the basis of group name. You also get a search box when you click this drop-down list. Use this search box to type a group name that you want to search. To search the rule violations across all the available groups, click Select All. Note: You can select more than one group.
Search by name	Type a rule name in this search box to filter all the rule violations on the basis of rule name.

To sort the rule violations, click on the header row of any column. You can sort in both ascending and descending order.

Working with Rule List

The application lists all the rules that you create in the Rule list page. The following table provides you details of all the actions possible in the Rule list page:

Action	Description
Create rule	Use this action button to create a new rule. For more information, see Creating a Rule.
	To edit a rule, click this action button on the row of the corresponding rule. Note: You cannot modify the rule name.
	To delete a rule, click this action button on the row of the corresponding rule.
	Select this toggle button in the Status column to enable or disable a rule. indicates that a rule is enabled and indicates that a rule is disabled. Note: When you use this button, you will see corresponding system messages on screen to indicate if a rule is enabled or disabled successfully.
	This is the column view settings button. Click this to customize the columns you want to view for the listed rules. In the Show Columns dialog, select the columns you want to view and click Save.
	Select this drop-down list to filter the rules on the basis of group name. You also get a search box when you click this drop-down list. Use this search box to type a group name that you want to search. Note: You can select more than one group.
Search by name	Type a rule name in this search box to filter the rules on the basis of rule name.

To sort the rules, click on the header row of any column. You can sort in both ascending and descending order.

Where can I find a detailed Step-by-Step Tutorial?

For a detailed tutorial click Step-by-Step Tutorial

Environment Groups and Stages

Get a granular view of the transaction data by filtering the transactions based on the environment group and the related stages.

Overview

Customers implement their solutions with Software AG Cloud products spanning multiple environments (one product installed in each environment) as well as each environment having stages like Development, Test and Production. This feature helps in providing a more granular view of the transaction data by allowing you to filter the transactions based on the environment group and the related stages. Using the environment group, you can view the transactions that span across more than one environment. Using the stages filter, you can further filter the transaction data based on the different stages the environments of the selected environment group are mapped with.

For example, let us assume there are two environments T1 and T2. A transaction starts from T1 and moves to T2. If the tenant is logged into T1, then the transaction details can be traced as the transaction starts from T1. However, if the tenant is logged into T2, then the transaction details are unavailable. To solve this, we have the Environment groups using which you can get the transaction details.

Creating an Environment Group

When you access End-to-End Monitoring from the App Switcher, you will see the following screen providing you the details on how to create environment groups. Select Next to start the environment creation process. This Environment group creation flow window will appear until you disable it using the Don’t show this message again checkbox.

You can skip this step and create the Environment groups at a later stage, by clicking the Later option.

Note: If you are a non-Software AG Cloud user, contact your Application Administrator to create the Environment groups.

If you selected Later on the onboarding screen, you can also create environment groups by following these steps:

1. Select End-to-End Monitoring. Go to User Profile > Administrator > Manage environment groups.

   

2. On the Manage environment groups page, click Create.

   

3. In the Create environment group window, provide an environment group name.

   Note: You cannot change this name once you create the environment group.

   

4. In Filter by stage, select the stages based on which you want to filter the environments. You can select more than one stage. Stages may include the following:

   • Development
   • Testing
   • Production

   The application auto-populates the environments list based on your selection of Stages.

5. Select all the environments that you want to include in your group and click Save.

   Note: After you select an environment, it will no longer be available for selection when you create another environment group. You can map an environment to a single environment group only.

6. On successful creation of the environment group, the application shows you a confirmation message and you will see the newly created environment group in the Environment group listing table.

   

Editing an Existing Environment Group

1. Go to User Profile > Administrator > Manage environment groups.

2. On the Manage environment groups page, click in the row of the environment group that you want to edit.

   

3. Make the required edits.

   Note: You cannot edit the environment group name.

4. After you make the changes, click Save.

Deleting an existing Environment Group

1. Go to User Profile > Administrator > Manage environment groups.

2. In the Manage environment groups section, click in the row of the environment group that you want to delete.

   Note: The deletion of an environment group is a permanent action.

   

3. On successful deletion, the application shows you a confirmation message and the environment group listing is updated.

Note:

• Environment groups are currently specific to the region of the logged-in environment. Any listing shows environment groups from that specific region only.

• After you add an environment to an environment group, it is no longer available for addition to any other environment group. You will receive the below prompt when attempting to create an environment group when all environments have already been mapped.

Non-Software AG Cloud tenants

If you are a webMethods Integration Cloud (WMIC) tenant, the environment creation process is different from the above mentioned steps. Non-Software AG Cloud tenants will need to contact Software AG Cloud Operations with the following questionnaire to assist in the creation of the environment groups.

1. What is the Account Name?

   The account name links the various environment groups together and is usually the customer name or the master account name in Software AG Cloud.

2. What are the environment groups, and the environments within each group?

   Specify a list of environment groups and the list of environments within each environment group and the stages (Development, Test, Pre-Live, Live) each environment belongs to. The list of environments within an environment group are used to filter the information when a given environment group is selected as part of the stages filter. All the environment groups will be linked to the same account name. Environment is also referred to as tenants in webMethods Integration Cloud (WMIC) terminology.

Note:

• A webMethods Integration Cloud (WMIC) tenant can have multiple stages. However, an API Gateway(APIGW) tenant can have only one stage.
• If information regarding environments and environment groups needs to be updated, contact Software AG Global support with information in the above questionnaire.
• Only four pre-defined stages are available for Non-Software AG Cloud tenants:
  • Development
  • Test
  • Pre-Live
  • Live

This table explains some valid and invalid configurations. You can use the same table template when you contact Software AG global support. Consider all these configurations for a given Account ID UPS:

Valid and Invalid Configurations

Dashboard and Alert Changes

The main dashboard page of End-to-End Monitoring contains a listing of all groups. There are new column entries available on this table as a part of the environment groups and stages feature. To view these entries, click . You can now add the following additional columns for table sorting purposes:

• Environment
• Stage
• Environment Group

On Dashboard page All groups table, click .

You will notice the additional column entries:

On Alert page Rule violations table, click .

You will notice the same additional column entries as Dashboard page above.

On Alert page Rule list table, click .

You will notice the same additional column entries as Dashboard page above.

Masthead Filters

After environments have been grouped as per your need, this feature provides filters on the Mast head to conveniently refine information listed on the Dashboard or Alert pages as per the Environment group or Stage in view or Both.

Dashboard page All groups table listing filtered as per Logistic Environment group on the masthead:

Dashboard page All groups table listing filtered as per Development Stage in view on the masthead:

Alert page Rule list table listing filtered as per Sales Environment group on the masthead:

Where can I find a detailed Step-by-Step Tutorial?

For a detailed tutorial click Step-by-Step Tutorial - Stages Feature

Example - Environment Groups and Stages

Consider a scenario where the Environment groups in your account are mapped in the following manner:

Now, on the masthead when you select All Environment group and All Stage in view:

You will notice that all dashboard page widgets and All groups listing automatically update information as per the masthead selections.

Next, create a group test2 which is linked to environment devuhm02 and save the group. Change the masthead selections to Fin1 Environment group:

You will now notice that the four widgets reflect new data and the new listing only shows test2 apart from All transactions.

You will notice similar filtering capabilities on Alert page as well:

Next, create a group test1 which is linked to environment prodenv1 and save the group. Change the masthead selections to Logistics Environment group:

You will now notice that the four widgets reflect new data and the new listing only shows test1 apart from All transactions.

Corresponding changes in Alert page:

Migration - Environment Groups and Stages

Consider a scenario where you have an account with groups prior to account mapping. None of these groups are mapped to an environment group or stage. Let us take the below journey of migrating these groups to an environment groups and stages mapped account.

Let us start with creating two groups, Old_Success and Old_fail with the below selected criteria:

You will notice at this stage that the masthead is empty and does not provide any options related to environment groups or stages:

Now let us create environment group mapping with the below criteria:

Let us now select edit button against Old_Success and Old_fail groups and select Development stages for both groups:

The masthead filters are now successfully able to filter the Old_Success and Old_fail groups. Hence, these groups have been successfully migrated to the new mapping.

Hybrid Monitoring for Integration Server and Microservices Runtime

A hybrid integration consists of a transaction that spans multiple platforms and products including both cloud and on-premises setups. Software AG’s End-to-End Monitoring allows you to track your integrations from the cloud through to your on-premises installation to give you a complete view of your integration landscape regardless of the topology of your IT infrastructure.

A hybrid integration use case is one where the transaction starts in a Software AG cloud product webMethods.io Integration and this cloud product calls a service on an on-premises Software AG product Integration Server to complete the transaction.

Hybrid monitoring is the capability that monitors the end-to-end hybrid transaction and displays it in the End-To-End Monitoring user interface.

The End-To-End Monitoring node on the Software AG installer allows you to setup the on-premises monitoring agents that enable the end-to-end transaction monitoring of hybrid integration.

Note:
For this release hybrid monitoring is supported for transactions from Integration Server and Microservices Runtime. In the future we will add support for all our products and even your own products, through an open tracing API.

Pre-requisites

1. Install Integration Server or Microservices Runtime, if not done already. You will need to have hybrid transaction configured for Integration Server 10.7 and above versions, and Microservices Runtime 10.7 and above versions.

2. Configure webMethods Cloud tenant connection in the Integration Server Administration UI. Prior to version 10.11, this configuration supported a connection to just a single webMethods.io Integration tenant. Versions 10.11 and above webMethods Integration Server, provides the capability to connect to multiple webMethods.io Integration tenants from the same Integration Server installation supporting hybrid integration transactions from multiple tenants. This functionality is also available on version 10.7 with advanced fixes. Use one of the two configuration options provided below, based on if this fix release is available at the time of using this functionality.

webMethods Integration Server without multiple tenant connection support

1. In the Integration Server Administration UI, navigate to webMethods Cloud >> Tenant Connections.

   

2. Configure the User Name, Password and webMethods Cloud URL.

webMethods Integration Server with multiple tenant connection support

• In webMethods Integration Server, navigate to webMethods Cloud >> Tenant Connections. Select the pre-existing tenant connection named default, provide the tenant details and Save. You can use this default connection to create your account.

  

• If a custom tenant connection is being used, create a new tenant connection with an alias name and tenant details.

  

Configuration

Following hybrid monitoring configurations are required in Integration Server and Microservices Runtime.

Integration Server

1. Make the following selections during Software AG Installer setup. If you have Integration Server pre-installed, only select End-to-End Monitoring.

   

   • If you are on release version 10.11, select End-to-End Monitoring Core 10.11 under Infrastructure.

     

2. Using Software AG update manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core, Integration Server and Microservices runtime plug-in fix.

   

3. Post fix installation, copy the plugin jar file:

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\lib\jars\
     

4. Copy log4j properties file:

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
   • To directory:
     <INSTALL-DIR>\IntegrationServer\instances\default\
     

5. Add the following lines at the end of file:
   <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf

   • Microsoft Windows
     

     wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\

   • Linux
     

     wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/                                                                                                    

6. Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, edit the property Dlog4j.configurationFile and append e2ecustomlog4j2.properties to it.

   • Microsoft Windows:

     • Existing

       wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties"

     • Appended

       wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"

   • Linux:

     • Existing

       wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties"

     • Appended

       wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"

7. If multiple tenant connection is supported in webMethods Integration Server, and a custom tenant connection is being used then:

   • Open the file: <INSTALL-DIR>\E2EMonitoring\agent\config\agent.config
   • Un-comment the property
     agent.onprem_multitenant_alias = ${SW_AGENT_ONPREM_MULTITENANT_ALIAS:default}
   • Replace default with tenant connection name.
   • Restart webMethods Integration Server.

Note:

• When a custom tenant connection is not used, End-to-End Monitoring uses default tenant connection value for agent.onprem_multitenant_alias in agent.config. You need to save default tenant connection details in webMethods Integration Server when there is no custom connection.
• Hybrid data transfer takes place within tenants of one given region only. When multiple tenant connections are created for different tenants belonging to the same region, specifying one of the tenant connection names for agent.onprem_multitenant_alias in agent.config is sufficient to trace hybrid transactions of other tenants belonging to the same region.

Microservices Runtime

1. Make the following selections during Software AG Installer setup. If you have Microservices Runtime pre-installed, only select End-to-End Monitoring.

   

2. Using Software AG update manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core, Integration Server and Microservices runtime plug-in fix.

   

3. Post fix installation, copy the plugin jar file:

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\lib\jars\
     

4. Copy log4j properties file:

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
   • To directory:
     <INSTALL-DIR>\IntegrationServer\
     

5. Add the following lines at the end of file:
   

   • Microsoft Windows
     

     <INSTALL-DIR>\IntegrationServer\bin\setenv.bat

     set JAVA_UHM_OPTS=-javaagent:"../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/" -Xbootclasspath/a:"../E2EMonitoring/agent/uhm-apm-agent.jar"
     set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHM_OPTS%
     set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% -Dlog4j.configurationFile="./e2ecustomlog4j2.properties"

   • Linux
     

     <INSTALL-DIR>\IntegrationServer\bin\setenv.sh

     JAVA_UHM_OPTS="-javaagent:../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/ -Xbootclasspath/a:../E2EMonitoring/agent/uhm-apm-agent.jar"
     JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} ${JAVA_UHM_OPTS}"
     JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} -Dlog4j.configurationFile=./e2ecustomlog4j2.properties"

What about multiple runtimes?

There are scenarios when multiple runtimes like multiple Integration Server cluster nodes or Integration Server installations are pointing to the same end-to-end agent jar file. In such situations, you need to make hybrid monitoring configuration changes in each runtime. This is applicable to both Integration server and Microservices Runtime.

Setting up what to trace

You now need to choose the on-premise services that you wish to trace in End-to-End Monitoring. This is done by changing the auditing setting of each service. You can choose to trace both top level services or private sub-level services depending on how much information you want to trace. However, take caution not to trace too much as this will consume your cloud storage and effectively reduce the number of historical transactions that you can view. It can also potentially impact the performance of your integrations and increase network traffic. Note that you do not need to trace every sub-service to ensure that the transaction is monitored*, any errors are propagated up the chain until they are caught by an audited service.

Note:

• * Excludes asynchronous services that are invoked using messaging or sub-threads. In which case you need to have these services audited. They are considered top level services in their own right.
• This setup piggybacks the on-premises service auditing feature.

Using the Software AG Designer

1. In Software AG Designer, for the on-premises Integration Server services associated with hybrid transaction including nested services, enable auditing with the following values:

   Enable auditing: Always
   Log on: Error, Success and start

   

2. Here is an example of a hybrid transaction in the End-to-End Monitoring user interface:

   

   

Using WATT properties in Integration Server

Enabling auditing in numerous services of Integration Server can be tedious and time consuming when you use the Software AG Designer. You can now also decide what services to audit using the extended settings of your webMethods runtime. This feature compliments the auditing options that can be set for each service individually. The advantage is that you can have different logging levels between different environments without having to modify service settings. This significantly simplifies your service auditing.

1. In Integration Server settings, select Extended and click Edit Extended Settings.

   

2. The following new keys are now available for auditing purposes. Use these keys to set audit levels.

   Include service names	Exclude service names
   watt.server.audit.service.include	watt.server.audit.service.exclude

   

Note:
Any change in these values is refreshed within ten minutes.

Including service names

1. To include service names with exact match, use the following syntax. After you set this extended property in Integration Server, these services are included for tracing. This list is captured during the first invocation of the service.

   Syntax	Description
   watt.server.audit.service.include=svc1,svc2	svc1 and svc2 are service names in this example where svc1 and svc2 are the fully qualified service names including the folder namespace.

2. To include service names starting with a certain text, use the following syntax. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service. You can also include partial names to specify services sharing the same name-space.

   Syntax	Description
   watt.server.audit.service.include=svc1*,svc2*	All service names starting with svc1 and svc2 are traced, where svc1 and svc2 could be jc.helloworld:* and acme.ordermgmt:pub:* respectively.

3. When you are tracing many services, using a file makes it easier. You can also choose to specify a file if you have a long list of services to specify. To include service names through a file, use the following syntax. Save your file with a .txt extension. List your service names in include.txt.
   
   Place each service name on a separate line in the file. Use # to comment out. Save the file in the relative or absolute path as per the table below. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

   Syntax	File location
   watt.server.audit.service.include=$file{./include.txt}	<install-dir>/IntegrationServer/instances/default
   watt.server.audit.service.include=$file{c:/home/include.txt}	c:/home/include.txt

4. Certain keywords are provided to simplify auditing. The keyword $toplevelis provided to simply trace all services that are called from an external client. Child services will only be invoked in this case if they are flagged individually for audit. This property can also be used in a comma separated list with all of the aforementioned features. To include all top level services, use the following syntax. When you include top level services, it is recommended to use $wmservices in the exclude list. This reduces the transaction logs overhead of all background webMethods.io services in the e2etrace.log file.

   Syntax	Description
   watt.server.audit.service.include=$toplevel	All top level services are traced.

Note: We recommend that you use the keyword $wmservices with your exclude list if you are using this property, in order to avoid too much tracing.

Excluding service names

1. To exclude service names, use the following syntax. After you set this extended property in Integration Server, these services are excluded from tracing. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

   Syntax	Description
   watt.server.audit.service.exclude=svc3,svc4	svc3 and svc4 are service names in this example.

2. When you are tracing many services, using a file makes it easier. To exclude service names using a file, use the following syntax in the given location. Save your file with a .txt extension. List your service names in exclude.txt.
   
   Place each service name on a separate line in the file. Use # to comment out. Save the file in the relative or absolute path as per the table below. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

   Syntax	File location
   watt.server.audit.service.exclude=$file{./exclude.txt}	<install-dir>/IntegrationServer/instances/default
   watt.server.audit.service.exclude=$file{c:/home/exclude.txt}	c:/home/exclude.txt

3. To exclude service names through the agent.config file, use the following syntax. When you set this property, all service names in the agent.config file and agent.exclude_serv_list property are excluded from tracing.

   Syntax	Description
   watt.server.audit.service.exclude=$wmservices	Excludes services through the use of agent.config.

   Example

   agent.exclude_serv_list=${SW_AGENT_EXCLUDE_SERV_LIST:wm.,com.,pub.}	wmservices represents any webMethods services. These service names start with wm. or pub. or com.. However, you can update this list with namespaces of your choice.

   

Key Combinations

These key properties can be combined. Following are some simple combination examples.

Important:

• What is eventually traced is always an effective difference of all services listed in the following two keys :
  watt.server.audit.service.include and watt.server.audit.service.exclude.
• However, the include property precludes the exclude property. The values in the include list will always take precedence even if a match is found in the exclude list. These properties are complimentary to the service audit properties and do not replace them, unless you have changed the agent property agent.global_incl_serv_flag.

What are the configurable properties?

Following are the configurable properties in the End-to-End Monitoring agent available at <INSTALL-DIR>\E2EMonitoring\agent\config

SW_AGENT_COLLECTOR_BACKEND_SERVICES values are based on the Software AG Cloud Regions :

Note:
Hybrid Monitoring data transfer from on-premises Integration Server or Microservices Runtime to cloud takes place through the tenant endpoint configured under webMethods Cloud Tenant Connections on port 11800.

Troubleshooting

Where can I find the log files?

• Integration Server log file location: <InstallDir>/IntegrationServer/instances/default/logs
• Microservices Runtime log file location: <InstallDir>/IntegrationServer/logs
• On-premises Integration Server transaction traces: e2etraces.log

How can I view traces information in the log files?

1. Go to the file e2ecustomlog4j2.properties at the following location:

   • Integration Server

     <INSTALL-DIR>\IntegrationServer\instances\default

   • Microservices Runtime

     <INSTALL-DIR>\IntegrationServer

2. Apply DEBUG logging level to the following properties:

   appender.FILE_OUT.filter.threshold.level = DEBUG
   appender.FILE_OUT1.filter.threshold.level = DEBUG
   appender.FILE_OUT2.filter.threshold.level = DEBUG
   logger.20.level = DEBUG
   logger.21.level = DEBUG
   logger.22.level = DEBUG

   
   

3. Here is a sample of the on-premises transaction trace:

   2021-08-06T19:49:50,328 DEBUG [e2eAgentAppender] - 
   [ENTRY]SpanInfo [traceId=[26655816.49441.45841902585488], segmentId:26655969.276.162825958992328, spanId:0, serviceName=forTesting:javaAsyncService1, startTime=1628259589923, duration=4ms, component=ONPREMISEIS, tenant=VMWODEPLAT01:5555, status=Success, requestID=5745b59f-089a-4b13-9038-f82188d68a32, parentLandscape=WMIO]

Are there exception cases where hybrid transaction traces will not work?

• The trace cannot be guaranteed to be coherent if you have child services that are audited, but called from a parent service that is not audited.

Detailed step-by-step tutorial

For a detailed tutorial, see Step-by-Step Tutorial - Hybrid Monitoring

Hybrid Monitoring for on-premises API Gateway

End-to-End Monitoring supports hybrid monitoring involving on-premises API gateway version 10.11 and above.

Note:
On-premises Integration Server and API Gateway plugins cannot be installed in the same installation. This is a known limitation.

1. Using Software AG Installer install on-premises API Gateway, End-to-End Monitoring Core and API Gateway Plugin.

   

2. Using Software AG update manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core and API Gateway plug-in fix.

3. webMethods Integration Server supports multiple tenant connections with 10.11 and above versions. This functionality is also available on version 10.7 with advanced fixes. Configure your tenant connection as per either of the below 3 arrangements:

   • webMethods Integration Server without multiple tenant connection support

     • In the Integration Server Administration UI, navigate to webMethods Cloud >> Tenant Connections.

       

   • webMethods Integration Server with multiple tenant connection support

     • In webMethods Integration Server, navigate to webMethods Cloud >> Tenant Connections. Tenant connections can be used in two ways in webMethods Integration Server with multiple tenant connection support.

     • Default tenant connection - Select the pre-existing tenant connection named default, provide the tenant details and Save. You can use this default connection to create your account.

       

     • New custom tenant conection - If you need to use a custom tenant connection, create a new tenant connection with a different alias name. This custom tenant connection is to be used while creating your webMethods cloud account.

       

4. Copy file: uhm-onpremise-is-http-plugin.jar

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-http-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\lib\jars\
     

5. Copy file: uhm-api-onpremise-plugin.jar

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\ uhm-api-onpremise-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\instances\default\packages\WmAPIGateway\ code\jars
     

6. Copy file: e2ecustomlog4j2.properties

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
   • To directory:
     <INSTALL-DIR>\IntegrationServer\instances\default\
     

7. Add the following lines at the end of file:
   <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf

   • Microsoft Windows
     

     wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\

   • Linux
     

     wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/    

8. Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, edit the property Dlog4j.configurationFile and append e2ecustomlog4j2.properties to it.

   • Microsoft Windows:

     • Existing

       wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties"

     • Appended

       wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"

   • Linux:

     • Existing

       wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties"

     • Appended

       wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"

9. Open the file: <INSTALL-DIR>\E2EMonitoring\agent\config\agent.config, and edit below properties:

   • Replace
     agent.service_name=${SW_AGENT_NAME: ONPREMISEIS} with
     agent.service_name=${SW_AGENT_NAME:ONPREMISEAPI}
   • Replace
     agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:default} with
     agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:<TenantName}

What are the configurable properties?

SW_AGENT_COLLECTOR_BACKEND_SERVICES values are based on the Software AG Cloud Regions :

You will have these property changes in agent.config take effect only after restarting the Integration Server.

Note:

1. To see segment traces of on-premises API gateway transaction in logs set all log levels to DEBUG in
   <INSTALL-DIR>\IntegrationServer\instances\default\e2ecustomlog4j2.properties
   
2. End-to-End Monitoring can trace transactions belonging to single tenant, even if end-to-end transactions involve multiple API gateway tenants at a time.
   
3. Hybrid monitoring works with on-premises Integration Server as well as on-premises API gateway. However, End-to-End Monitoring does not support usage of both products together as part of Hybrid integration.
4. End-to-End Monitoring, by default, monitors all the on-premises webMethods API Gateway calls with tag e2em.