Hybrid Monitoring

Common configuration

All combinations of hybrid monitoring have the following common configuration.

Hybrid Integration Installation Setup

1. Install Integration Server or Microservices Runtime along with End-to-End Monitoring, if not done already. If you have Integration Server pre-installed, select only End-to-End Monitoring.

   • For version 10.7

   • From version 10.11 onwards

Note:
Make your selections as per the below criteria:

• For Hybrid Monitoring with Integration Server, select:
  • End-to-End Monitoring Core
  • Integration Server or Microservices Runtime Plug-in
• For Hybrid Monitoring with API Gateway, select:
  • End-to-End Monitoring Core
  • API Gateway Plug-in

2. Using Software AG Update Manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core. Based on the product installed, install either Integration Server and Microservices Runtime plug-in fix, or API Gateway plug-in fix.

Setting up webMethods Cloud tenant connection in webMethods Integration Server

Without multiple tenant connection support

1. In the Integration Server Administration UI, go to webMethods Cloud > Tenant Connections.

   

2. Configure the User Name, Password and webMethods Cloud URL.

With multiple tenant connection support

1. In webMethods Integration Server, go to webMethods Cloud > Tenant Connections. Select the pre-existing tenant connection named default, provide the tenant details, and Save. You can use this default connection to create your account.

   

2. If a custom tenant connection is being used, create a new tenant connection with an alias name and tenant details.

   

Note:
Multiple tenant connection support is available with latest fixes installed.

Setting up what to trace

For on-premises services to be traced in End-to-End Monitoring audit settings have to be enabled. There are 2 ways of doing this. You can choose to trace both top level services or private sub-level services depending on how much information you want to trace. However, take caution not to trace too much as this will consume your cloud storage and effectively reduce the number of historical transactions that you can view. It can also potentially impact the performance of your integrations and increase network traffic. Note that you do not need to trace every sub-service to ensure that the transaction is monitored. Errors are propagated up the chain until they are caught by an audited service.

Enabling audit in the Software AG Designer

You now need to choose the on-premises services that you wish to trace in End-to-End Monitoring. This is done by changing the auditing setting of each service.

1. In Software AG Designer, for the on-premises Integration Server services associated with hybrid transaction including nested services, enable auditing with the following values:

   Enable auditing: Always
   Log on: Error, Success and start

   

2. Here is an example of a hybrid transaction in the End-to-End Monitoring user interface:

   

   

Using WATT properties in Integration Server

Enabling auditing in numerous services of Integration Server can be tedious and time consuming when you use the Software AG Designer. You can now also decide what services to audit using the extended settings of your webMethods runtime. This feature complements the auditing options that can be set for each service individually. The advantage is that you can have different logging levels between different environments without having to modify the service settings. This significantly simplifies your service auditing.

1. In Integration Server settings, select Extended and click Edit Extended Settings.

   

2. The following new keys are now available for auditing purposes. Use these keys to set the audit levels.

   Include service names	Exclude service names
   watt.server.audit.service.include	watt.server.audit.service.exclude

   

Note:
Any change in these values is refreshed within ten minutes.

Including service names

1. To include service names with exact match, use the following syntax. After you set this extended property in Integration Server, these services are included for tracing. This list is captured during the first invocation of the service.

   Syntax	Description
   watt.server.audit.service.include=svc1,svc2	svc1 and svc2 are service names in this example where svc1 and svc2 are the fully qualified service names including the folder namespace.

2. To include service names starting with a certain text, use the following syntax. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service. You can also include partial names to specify services sharing the same namespace.

   Syntax	Description
   watt.server.audit.service.include=svc1*,svc2*	All service names starting with svc1 and svc2 are traced, where svc1 and svc2 could be jc.helloworld:* and acme.ordermgmt:pub:* respectively.

3. When you are tracing many services, using a file makes it easier. You can also choose to specify a file if you have a long list of services to specify. To include service names through a file, use the following syntax. Save your file with a .txt extension. List your service names in include.txt.
   
   Place each service name on a separate line in the file. Use # to comment out. Save the file in the relative or absolute path as per the table below. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

   Syntax	File location
   watt.server.audit.service.include=$file{./include.txt}	<install-dir>/IntegrationServer/instances/default
   watt.server.audit.service.include=$file{c:/home/include.txt}	c:/home/include.txt

4. Certain keywords are provided to simplify auditing. The keyword $toplevelis provided to trace all services that are called from an external client. Child services will only be invoked in this case if they are flagged individually for audit. This property can also be used in a comma separated list with all of the aforementioned features. To include all top level services, use the following syntax. When you include top level services, it is recommended to use $wmservices in the exclude list. This reduces the transaction logs overhead of all background webMethods.io services in the e2etrace.log file.

   Syntax	Description
   watt.server.audit.service.include=$toplevel	All top level services are traced.

Note:
It is recommended to use the keyword $wmservices with your exclude list if you are using this property, in order to avoid too much tracing.

Excluding service names

1. To exclude service names, use the following syntax. After you set this extended property in Integration Server, these services are excluded from tracing. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

   Syntax	Description
   watt.server.audit.service.exclude=svc3,svc4	svc3 and svc4 are service names in this example.

2. When you are tracing many services, using a file makes it easier. To exclude service names using a file, use the following syntax in the given location. Save your file with a .txt extension. List your service names in exclude.txt.
   
   Place each service name on a separate line in the file. Use # to comment out. Save the file in the relative or absolute path as per the following table. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

   Syntax	File location
   watt.server.audit.service.exclude=$file{./exclude.txt}	<install-dir>/IntegrationServer/instances/default
   watt.server.audit.service.exclude=$file{c:/home/exclude.txt}	c:/home/exclude.txt

3. To exclude service names through the agent.config file, use the following syntax. When you set this property, all service names in the agent.config file and agent.exclude_serv_list property are excluded from tracing.

   Syntax	Description
   watt.server.audit.service.exclude=$wmservices	Excludes services through the use of agent.config.

   Example

   agent.exclude_serv_list=${SW_AGENT_EXCLUDE_SERV_LIST:wm.,com.,pub.}	wmservices represents any webMethods services. These service names start with wm. or pub. or com.. However, you can update this list with namespaces of your choice.

   

Key Combinations

These key properties can be combined. Following are some simple combination examples.

Note:

• * Excludes asynchronous services that are invoked using messaging or sub-threads, in which case you need to have these services audited. They are considered top level services in their own right.
• This setup piggybacks the on-premises service auditing feature.

Important:

• What is eventually traced is always an effective difference of all services listed in the following two keys :
  watt.server.audit.service.include and watt.server.audit.service.exclude.
• However, the include property precludes the exclude property. The values in the include list will always take precedence even if a match is found in the exclude list. These properties are complementary to the service audit properties and do not replace them, unless you have changed the agent property agent.global_incl_serv_flag.

Collector end-points

SW_AGENT_COLLECTOR_BACKEND_SERVICES values are based on the Software AG Cloud Regions :

Setting up on-premises Integration Server for Cloud Container access

webMethods Cloud Container supports hybrid integration. This integration takes place when Cloud Container accesses the on-premises webMethods Integration Server through messaging.

For a hybrid integration in Cloud Container, the on-premises Integration Server and Cloud Container need to share the same messaging server. To achieve this, the on-premises Integration Server messaging service is pointed to use Cloud Container messaging service.

There are 2 types of messaging service in Integration Server.

• webMethods messaging service
• JMS

In both these cases JNDI provided alias needs to be configured in the on-premises Integration Server. End-to-End Monitoring uses this for authentication purposes in Cloud Container hybrid integration monitoring.

1. In Integration Server, navigate to Messaging > JNDI settings > JNDI provider alias.

   

2. Create a new JNDI provider alias or use the existing default alias DEFAULT_IS_JNDI_PROVIDER. Verify that the provider URL of your JNDI provider alias contains the Cloud Container messaging server URL.

   

3. You can find the Cloud Container messaging URL in Cloud Container Solution > Manage > Messaging Access Settings.

   

webMethods messaging service

1. In Integration Server, navigate to Messaging > webMethods settings, to configure on-premises Integration Server with Cloud Container messaging server.

   

2. Disable webMethods Messaging Connection Aliasfor your existing messaging services, and edit the webMethods Messaging Connection Alias Definitions. If you are not using the messaging service, then create a new Messaging Connection alias based on your requirement.

   

3. Replace Realm URL with Cloud Container messaging server URL. Save and enable the alias.

4. Verify that the Cloud Container solution and the on-premises Integration Server have the same type of messaging services enabled, before triggering your hybrid integration. This image shows webMethods messaging being enabled in Cloud Container.

   

JMS

1. Navigate to Messaging > JMS settings, to configure on-premises Integration Server with Cloud Container messaging server.

   

2. Disable JMS Connection Alias Definitions for your existing messaging services, and edit the JMS Connection Alias Definitions. If you are not using the messaging service, then create a new JMS Connection alias based on your requirement.

   

3. Select the JNDI Provider Alias Name created above. Select DEFAULT_IS_JNDI_PROVIDER, if default JNDI alias is used. Save and enable the JNDI provider alias.

4. Verify that the Cloud Container solution and the on-premises Integration Server have the same type of messaging services enabled, before triggering your hybrid integration. This image shows JMS messaging being enabled in Cloud Container.

   

Note:

• Setup and usage of webMethods Messaging Connection Alias definition or JMS Provider Alias definition requires setting up connection factory channels and topics at the messaging server.
• For more information on configuring Integration Server for webMethods Messaging, see webMethods Integration Server Administrators Guide.
• For more information on configuring Integration Server for JMS Messaging, see webMethods Integration Server Administrators Guide.
• For more information on configuring Integration Server to work with JNDI Providers, see webMethods Integration Server Administrators Guide.

Common configurable properties

Following are the configurable properties in the End-to-End Monitoring agent available at

<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Troubleshooting

Where can I find the log files?

• Integration Server log file location: <InstallDir>/IntegrationServer/instances/default/logs
• Microservices Runtime log file location: <InstallDir>/IntegrationServer/logs
• On-premises Integration Server transaction traces: e2etraces.log

How can I view traces information in the log files?

Note:

The steps outlined below are applicable only if you have already completed the configuration for hybrid monitoring with Integration Server or with webMethods API Gateway.

1. Go to the file e2ecustomlogback.xml at the following location:

   • Integration Server

     <INSTALL-DIR>\IntegrationServer\instances\default

   • Microservices Runtime

     <INSTALL-DIR>\IntegrationServer

   Note:
   If you cannot locate the file e2ecustomlogback.xml in the installation, then install latest fixes.

2. Apply the DEBUG logging level to the following properties:

   "e2eAgentSummary" level="DEBUG"  
   "e2eAgentSegments" level="DEBUG" 
   "e2eagentFailedSegments" level="DEBUG" 

   
   

3. Here is a sample of the on-premises transaction trace:

   2021-08-06T19:49:50,328 DEBUG [e2eAgentAppender] - 
   [ENTRY]SpanInfo [traceId=[26655816.49441.45841902585488], segmentId:26655969.276.162825958992328, spanId:0, serviceName=forTesting:javaAsyncService1, startTime=1628259589923, duration=4ms, component=ONPREMISEIS, tenant=VMWODEPLAT01:5555, status=Success, requestID=5745b59f-089a-4b13-9038-f82188d68a32, parentLandscape=WMIO]

Are there exception cases where hybrid transaction traces will not work?

• The trace cannot be guaranteed to be coherent if you have child services that are audited, but called from a parent service that is not audited.

Detailed step-by-step tutorial

For a detailed tutorial, see Step-by-Step Tutorial - Hybrid Monitoring.

With Integration Server and Microservices Runtime through webMethods.io Integration on-premise connectors

Pre-requisites

1. For Hybrid Monitoring installation setup click here.
2. For Hybrid Monitoring to work, webMethods Cloud tenant connection needs to be setup in Integration Server. Click here for more details.
3. Audit needs to be enabled for on-premises services. Click here for more details.

Note:

• From v10.15 onwards, product code UHM has changed to UHA. If you come across a file name with UHA in it, consider it the same as a UHM file.

Configuration

The following hybrid monitoring configurations are required in Integration Server and Microservices Runtime.

Integration Server

1. After fix installation, copy the plugin jar file:

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\lib\jars\
     

2. Setting up logs: Copy logback xml file

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml
   • To directory:
     <INSTALL-DIR>\IntegrationServer\instances\default

   Note:
   If you cannot locate the file e2ecustomlogback.xml in the installation, then install latest fixes.

3. Add the following lines at the end of the file:
   <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf

   • Microsoft Windows
     

     wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\

   • Linux
     

     wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/

   Note:
   From v10.15 onwards, product code UHM has changed to UHA. Accordingly, replace uhm-apm-agent.jar with uha-apm-agent.jar in the above lines.

4. Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, add the property:

   • Microsoft Windows:

         wrapper.java.additional.503=-Dlogback.configurationFile=".\e2ecustomlogback.xml"

   • Linux:

         wrapper.java.additional.503=-Dlogback.configurationFile="./e2ecustomlogback.xml"

5. If multiple tenant connections are supported in Integration Server and a custom tenant connection is being used, then open the file: <INSTALL-DIR>\E2EMonitoring\agent\config\agent.configand set the below properties:

Note:

• When a custom tenant connection is not used, End-to-End Monitoring uses the default tenant connection value for agent.onprem_multitenant_alias in agent.config. You need to save the default tenant connection details in webMethods Integration Server when there is no custom connection.
• Hybrid data transfer takes place within tenants of one given region only. When multiple tenant connections are created for different tenants belonging to the same region, specifying one of the tenant connection names for agent.onprem_multitenant_alias in agent.config is sufficient to trace hybrid transactions of other tenants belonging to the same region.
• To see segment traces of on-premises API Gateway transaction in logs, set all log levels to DEBUG in versions below 10.15
  <INSTALL-DIR>\IntegrationServer\instances\default\e2ecustomlog4j2.properties
  
• From v10.15 onwards, set log levels to debug in
  <INSTALL-DIR>\IntegrationServer\instances\default\e2elogback.xml

Click here for more details on End-to-End Monitoring agent configurable properties for hybrid monitoring with Integration Server or Microservices Runtime.

Click here for more details on traces information in logs files.

Microservices Runtime

1. After fix installation, copy the plugin jar file:

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\lib\jars\
     

2. Setting up logs: Copy logback xml file

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml
   • To directory:
     <INSTALL-DIR>\IntegrationServer\

   Note:
   If you cannot locate the file e2ecustomlogback.xml in the installation, then install latest fixes.

3. Add the following lines at the end of the file:
   

   • Microsoft Windows
     

     <INSTALL-DIR>\IntegrationServer\bin\setenv.bat

     set JAVA_UHM_OPTS=-javaagent:"../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/" -Xbootclasspath/a:"../E2EMonitoring/agent/uhm-apm-agent.jar"
     set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHM_OPTS%
     set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% -Dlogback.configurationFile="./e2ecustomlogback.xml"

   • Linux
     

     <INSTALL-DIR>\IntegrationServer\bin\setenv.sh

     JAVA_UHM_OPTS="-javaagent:../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/ -Xbootclasspath/a:../E2EMonitoring/agent/uhm-apm-agent.jar"
     JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} ${JAVA_UHM_OPTS}"
     JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} -Dlogback.configurationFile=./ e2ecustomlogback.xml"

What about multiple runtimes?

There are scenarios when multiple runtimes like multiple Integration Server cluster nodes or Integration Server installations are pointing to the same end-to-end agent jar file. In such situations, you need to make hybrid monitoring configuration changes in each runtime. This is applicable to both Integration server and Microservices Runtime.

What are the End-to-End Monitoring agent configurable properties?

Click Here for the common configurable properties in the End-to-End Monitoring agent available at
<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Note:

• agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000}
• This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes if you prefer a faster sync time.

Click here for more details on traces information in logs files.

With on-premises webMethods API Gateway

End-to-End Monitoring supports hybrid monitoring involving on-premises API Gateway v10.11 and later versions.

Note:

• On-premises Integration Server and API Gateway plugins cannot be installed in the same installation. This is a known limitation.
• From v10.15 onwards, product code UHM has changed to UHA. If you come across a file name with UHA in it, consider it the same as a UHM file.

Pre-requisites

1. For Hybrid Monitoring installation setup click Here.
2. For Hybrid Monitoring to work, webMethods Cloud tenant connection needs to be setup in Integration Server. Click Here for more details.

Configuration

1. Copy file: uhm-onpremise-is-http-plugin.jar

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-http-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\lib\jars\
     

2. Copy file: uhm-api-onpremise-plugin.jar

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\plugins\ uhm-api-onpremise-plugin.jar
   • To directory:
     <INSTALL-DIR>\IntegrationServer\instances\default\packages\WmAPIGateway\code\jars
     

3. Setting up logs: Copy logback xml file

   • From directory:
     <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml
   • To directory:
     <INSTALL-DIR>\IntegrationServer\instances\default\

   Note:
   If you cannot locate the file e2ecustomlogback.xml in the installation, then install latest fixes.

4. Add the following lines at the end of the file:
   <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf

   • Microsoft Windows
     

     wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\

   • Linux
     

     wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
     wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/   

   Note:
   From v10.15 onwards, product code UHM has changed to UHA. Accordingly, replace uhm-apm-agent.jar with uha-apm-agent.jar in the above lines.

5. Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, add the property:

   • Microsoft Windows:

         wrapper.java.additional.503=-Dlogback.configurationFile=".\e2ecustomlogback.xml"

   • Linux:

         wrapper.java.additional.503=-Dlogback.configurationFile="./e2ecustomlogback.xml"

What are the API Gateway specific configurable properties?

Click Here for the common configurable properties in the End-to-End Monitoring agent available at
<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Notes:

• To see segment traces of on-premises API Gateway transaction in logs, set all log levels to DEBUG in
  <INSTALL-DIR>\IntegrationServer\instances\default\e2ecustomlog4j2.properties (For versions below 10.15)
  <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml (From v10.15 onwards)
  
• End-to-End Monitoring can trace transactions belonging to a single tenant, even if end-to-end transactions involve multiple API Gateway tenants at a time.
  
• Hybrid monitoring works with on-premises Integration Server as well as on-premises API Gateway. However, End-to-End Monitoring does not support usage of both products together as part of hybrid integration.
• End-to-End Monitoring by default monitors all the on-premises webMethods API Gateway calls with the tag e2em.
• agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000}: This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes if you prefer a faster sync time.

Click here for more details on traces information in logs files.

With Integration Server through Cloud Container services

End-to-End Monitoring supports hybrid monitoring through Cloud Container solutions involving only Integration Server version 10.11.

How to enable support for webMethods Cloud Container?

End-to-End Monitoring support for Cloud Container use cases, is currently only available on demand for this initial release version 10.15.2. These use cases have been verified on Integration Server version 10.11. Support for all Cloud Container use cases will be available by default, post performance optimizations planned for a release later in 2023.

To enable this on demand End-to-End Monitoring support for Cloud Container use cases, contact Software AG Global Support with the below details:

• Tenant name and region.

Note:

• Cloud Container server restart is required for your tenant, to enable the End-to-End Monitoring agent. Software AG Global Support will coordinate with you for this restart to ensure a minimal downtime.

Pre-requisites

1. For Hybrid Monitoring installation setup click here.
2. For Hybrid Monitoring to work, webMethods Cloud tenant connection needs to be setup in Integration Server. Click here for more details.
3. Audit needs to be enabled for on-premises services. Click here for more details.
4. For Cloud Container to communicate with on-premises Integration Server, you need to configure Integration Server to use Cloud Container messaging services. Click here for more details.

Note:

• From v10.15 onwards, product code UHM has changed to UHA. If you come across a file name with UHA in it, consider it the same as a UHM file.

Cloud Container hybrid monitoring configuration

1. Configuration for hybrid monitoring with Integration Server through Cloud Container, is the same as the configuration for hybrid monitoring through webmethods.io Integration on-premises connector. Click here for more details.
2. Set Cloud Container specific agent.config properties instead of webmethods.io Integration properties. Click here for more details.

What are the Cloud Container specific configurable properties?

Click here for the common configurable properties in the End-to-End Monitoring agent available at
<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Note:

• agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000}
• This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes if you prefer a faster sync time.