Hybrid Monitoring

End-to-End Monitoring allows you to track your integrations from the cloud through to your on-premises installation to give you a complete view of your integration landscape, regardless of the topology of your IT infrastructure. A hybrid integration consists of a transaction that spans multiple platforms and products including both cloud and on-premises setups.


Hybrid monitoring is the capability that monitors the end-to-end hybrid transaction and displays it in the End-To-End Monitoring user interface.

A hybrid integration use case is one where the transaction starts in a Software AG cloud product, for example, webMethods.io Integration, and this cloud product calls a service on an on-premises Software AG product, for example, Integration Server, to complete the transaction.

End-To-End Monitoring node on the Software AG installer allows you to setup the on-premises monitoring agents, which enable the end-to-end transaction monitoring of hybrid integration.

Note:

  • From v10.15 onwards, product code UHM has changed to UHA. If you come across a file name with UHA in it, consider it the same as a UHM file.

Common configuration

All combinations of hybrid monitoring have the following common configuration.

Hybrid Integration Installation Setup

  1. Install Integration Server or Microservices Runtime along with End-to-End Monitoring, if not done already. If you have Integration Server pre-installed, select only End-to-End Monitoring.

    • For version 10.7

    • From version 10.11 onwards

Note:
Make your selections as per the below criteria:

  • For Hybrid Monitoring with Integration Server, select:
    • End-to-End Monitoring Core
    • Integration Server or Microservices Runtime Plug-in
  • For Hybrid Monitoring with API Gateway, select:
    • End-to-End Monitoring Core
    • API Gateway Plug-in


2. Using Software AG Update Manager, install the latest fixes for all installed Software AG products along with End-to-End Monitoring core. Based on the product installed, install either Integration Server and Microservices Runtime plug-in fix, or API Gateway plug-in fix.

Setting up webMethods Cloud tenant connection in webMethods Integration Server

Without multiple tenant connection support

  1. In the Integration Server Administration UI, go to webMethods Cloud > Tenant Connections.

  2. Configure the User Name, Password and webMethods Cloud URL.

With multiple tenant connection support

  1. In webMethods Integration Server, go to webMethods Cloud > Tenant Connections. Select the pre-existing tenant connection named default, provide the tenant details, and Save. You can use this default connection to create your account.

  2. If you use a custom tenant connection, create a new tenant connection with an alias name and tenant details.

Note:
Multiple tenant connection support is available with latest fixes installed.

Setting up what to trace

For on-premises services to be traced in End-to-End Monitoring audit settings have to be enabled. There are 2 ways of doing this. You can choose to trace both top level services or private sub-level services depending on how much information you want to trace. However, take caution not to trace too much as this will consume your cloud storage and effectively reduce the number of historical transactions that you can view. It can also potentially impact the performance of your integrations and increase network traffic. Note that you do not need to trace every sub-service to ensure that the transaction is monitored. Errors are propagated up the chain until they are caught by an audited service.

Enabling audit in the Software AG Designer

You now need to choose the on-premises services that you wish to trace in End-to-End Monitoring. This is done by changing the auditing setting of each service.

  1. In Software AG Designer, for the on-premises Integration Server services associated with hybrid transaction including nested services, enable auditing with the following values:

    Enable auditing: Always
    Log on: Error, Success and start

  2. Here is an example of a hybrid transaction in the End-to-End Monitoring user interface:

Using WATT properties in Integration Server

Enabling auditing in numerous services of Integration Server can be tedious and time consuming when you use the Software AG Designer. You can now also decide what services to audit using the extended settings of your webMethods runtime. This feature complements the auditing options that can be set for each service individually. The advantage is that you can have different logging levels between different environments without having to modify the service settings. This significantly simplifies your service auditing.

  1. In Integration Server settings, select Extended and click Edit Extended Settings.

  2. The following new keys are now available for auditing purposes. Use these keys to set the audit levels.

    Include service names Exclude service names
    watt.server.audit.service.include watt.server.audit.service.exclude

Note:
Any change in these values is refreshed within ten minutes.

Including service names

  1. To include service names with exact match, use the following syntax. After you set this extended property in Integration Server, these services are included for tracing. This list is captured during the first invocation of the service.

    Syntax Description
    watt.server.audit.service.include=svc1,svc2 svc1 and svc2 are service names in this example where svc1 and svc2 are the fully qualified service names including the folder namespace.
  2. To include service names starting with a certain text, use the following syntax. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service. You can also include partial names to specify services sharing the same namespace.

    Syntax Description
    watt.server.audit.service.include=svc1*,svc2* All service names starting with svc1 and svc2 are traced, where svc1 and svc2 could be jc.helloworld:* and acme.ordermgmt:pub:* respectively.
  3. When you are tracing many services, using a file makes it easier. You can also choose to specify a file if you have a long list of services to specify. To include service names through a file, use the following syntax. Save your file with a .txt extension. List your service names in include.txt.

    Place each service name on a separate line in the file. Use # to comment out. Save the file in the relative or absolute path as per the table below. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

    Syntax File location
    watt.server.audit.service.include=$file{./include.txt} <install-dir>/IntegrationServer/instances/default
    watt.server.audit.service.include=$file{c:/home/include.txt} c:/home/include.txt
  4. Certain keywords are provided to simplify auditing. The keyword $toplevelis provided to trace all services that are called from an external client. Child services will only be invoked in this case if they are flagged individually for audit. This property can also be used in a comma separated list with all of the aforementioned features. To include all top level services, use the following syntax. When you include top level services, it is recommended to use $wmservices in the exclude list. This reduces the transaction logs overhead of all background webMethods.io services in the e2etrace.log file.

    Syntax Description
    watt.server.audit.service.include=$toplevel All top level services are traced.

Note:
It is recommended to use the keyword $wmservices with your exclude list if you are using this property, in order to avoid too much tracing.

Excluding service names

  1. To exclude service names, use the following syntax. After you set this extended property in Integration Server, these services are excluded from tracing. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

    Syntax Description
    watt.server.audit.service.exclude=svc3,svc4 svc3 and svc4 are service names in this example.
  2. When you are tracing many services, using a file makes it easier. To exclude service names using a file, use the following syntax in the given location. Save your file with a .txt extension. List your service names in exclude.txt.

    Place each service name on a separate line in the file. Use # to comment out. Save the file in the relative or absolute path as per the following table. These service names can either be complete service names or partial namespaces. This list is captured during the first invocation of the service.

    Syntax File location
    watt.server.audit.service.exclude=$file{./exclude.txt} <install-dir>/IntegrationServer/instances/default
    watt.server.audit.service.exclude=$file{c:/home/exclude.txt} c:/home/exclude.txt
  3. To exclude service names through the agent.config file, use the following syntax. When you set this property, all service names in the agent.config file and agent.exclude_serv_list property are excluded from tracing.

    Syntax Description
    watt.server.audit.service.exclude=$wmservices Excludes services through the use of agent.config.

    Example

    agent.exclude_serv_list=${SW_AGENT_EXCLUDE_SERV_LIST:wm.,com.,pub.} wmservices represents any webMethods services. These service names start with wm. or pub. or com.. However, you can update this list with namespaces of your choice.

Key Combinations

These key properties can be combined. Following are some simple combination examples.

Combination example Description
watt.server.audit.service.include=$toplevel,svc1,svc2,$file{./include.txt} All top level services are traced, along with svc1 and svc2, and services listed in the file include.txt.
watt.server.audit.service.exclude=svc1,svc2,$wmservices,${file{./exclude.txt} svc1 and svc2 are excluded from tracing. Services listed in exclude.txt are also excluded from tracing. Services listed in agent.config file and agent.exclude_serv_list property are also excluded from tracing.

Note:

  • Excludes asynchronous services that are invoked using messaging or sub-threads, in which case you need to have these services audited. They are considered top level services in their own right.
  • This setup piggybacks the on-premises service auditing feature.


Important:

  • What is eventually traced is always an effective difference of all services listed in the following two keys :
    watt.server.audit.service.include and watt.server.audit.service.exclude.
  • However, the include property precludes the exclude property. The values in the include list will always take precedence even if a match is found in the exclude list. These properties are complementary to the service audit properties and do not replace them, unless you have changed the agent property agent.global_incl_serv_flag.

Collector end-points

SW_AGENT_COLLECTOR_BACKEND_SERVICES values are based on the Software AG Cloud Regions :

Azure End-Point Value
US2 East Azure uhm.e2em-hybrid-az-us.webmethods.io:443
AU1 Australia East Azure uhm.e2em-hybrid-az-au.webmethods.io:443
EU3 West Azure uhm.e2em-hybrid-az-eu.webmethods.io:443
AWS End-Point Value
US1 Oregon AWS uhm.e2em-hybrid-aw-us.webmethods.io:443
EU2 Frankfurt AWS uhm.e2em-hybrid-aw-eu.webmethods.io:443
AU2 Sydney AWS uhm.e2em-hybrid-aw-au.webmethods.io:443

Common configurable properties

Following are the configurable properties in the End-to-End Monitoring agent available at

<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Property Description
agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000} This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes if you prefer a faster sync time.
collector.establish_cloud_communication=${SW_AGENT_COLLECTOR_ESTABLISH_CLOUD_COMMUNICATION:true} This property defines connectivity with the cloud collector. Default value is true, which means that data is sent to the cloud collector.
agent.onprem_multitenant_alias Hybrid data transfer takes place within tenants of one given region only. When multiple tenant connections are created for different tenants belonging to the same region, specifying one of the tenant connection names for agent.onprem_multitenant_alias in agent.config is sufficient to trace hybrid transactions of other tenants belonging to the same region.

However, this is required only if you have multiple tenants in the same region and you want to route the trace using a specific tenant. Note that End-to-End Monitoring can only trace to one tenant but is able to determine the original tenant for each trace.
agent.global_incl_serv_flag=${SW_AGENT_GLOBAL_INCL_SERV_FLAG:GLOBAL_OR_LOCAL} This property allows you to select between Software AG Designer audit property or Integration Server extended settings WATT properties based tracing. Default Value is GLOBAL_OR_LOCAL. Possible values are GLOBAL_OR_LOCAL, GLOBAL, LOCAL.

  • LOCAL: Only services with audit service enabled on the Software AG Designer are traced.

  • GLOBAL: WATT property include-exclude list is considered for tracing.

  • GLOBAL_OR_LOCAL: Both WATT property and Software AG Designer settings, are considered for include-exclude list.

agent.exclude_serv_list=${SW_AGENT_EXCLUDE_SERV_LIST:wm.,com.,pub.} This property is considered only if WATT exclude is set with $wmservices. $wmservices represents any webMethods services. These service names start with wm. or pub. or com., and are excluded from tracing.
agent.watt_check_interval=${SW_AGENT_WATT_CHECK_INTERVAL:600000} Any changes in WATT property or the content in include or exclude files is refreshed in cache at this interval. Default value is 600000 ms=10 minutes.
agent.service_name=${SW_AGENT_NAME:ONPREMISEIS} Indicates service name that monitors product calls.
Set SW_AGENT_NAME to ONPREMISEAPI, to monitor on-premise API Gateway calls.
agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:default} Indicates cloud tenant from which transactions to on-premise API Gateway takes place.
Tenant name is the value for SW_AGENT_ONPPREM_TENANT_ID.
agent.apigw_tag=${SW_AGENT_APIGW_TAG:<custom tag>} End-to-End Monitoring, by default, monitors all the on-premises webMethods API Gateway calls with tag e2em. If you need to monitor an API containing custom tags , add this property in agent.config with custom tag value. To enable tracing, add the e2em tags to the webMethods API Gateway API, in case you do not have them already.
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES: uhm.e2em-hybrid-az-us.webmethods.io:443} In case of agent to collector connection issues, this property can be used to set collector endpoint so that the agent can connect to the specified collector. This property is not required to be set explicitly from v10.15 onwards, for v10.11 or v1.3.0 latest fixes (Fix 5 and above).

Note: Refer Collector end-point section for specific collector end-point values.
agent.onprem_multijndi_alias = ${SW_AGENT_ONPREM_MULTIJNDI_ALIAS:DEFAULT_IS_JNDI_PROVIDER} This property defines the Multi-JNDI support for specific alias. This property is set to DEFAULT_IS_JNDI_PROVIDER by default. If you use custom JNDI provider alias in the Integration Server, then use the custom name as value instead of DEFAULT_IS_JNDI_PROVIDER.
agent.onprem_jndi_auth_enable=${SW_AGENT_ONPREM_JNDI_AUTH_ENABLE:true} This property defines on-premises to cloud JNDI based authentication. It is set to false by default.

Troubleshooting

Where can I find the log files?

How can I view traces information in the log files?

For versions below 10.15

  1. Go to the e2ecustomlog4j2.properties file at the following location:

    • Integration Server

      <INSTALL-DIR>\IntegrationServer\instances\default
    • Microservices Runtime

      <INSTALL-DIR>\IntegrationServer
  2. Apply the DEBUG logging level to the following properties:

    appender.FILE_OUT.filter.threshold.level = DEBUG
    appender.FILE_OUT1.filter.threshold.level = DEBUG
    appender.FILE_OUT2.filter.threshold.level = DEBUG
    logger.20.level = DEBUG
    logger.21.level = DEBUG
    logger.22.level = DEBUG


  3. Here is a sample of the on-premises transaction trace:

    2021-08-06T19:49:50,328 DEBUG [e2eAgentAppender] - 
    [ENTRY]SpanInfo [traceId=[26655816.49441.45841902585488], segmentId:26655969.276.162825958992328, spanId:0, serviceName=forTesting:javaAsyncService1, startTime=1628259589923, duration=4ms, component=ONPREMISEIS, tenant=VMWODEPLAT01:5555, status=Success, requestID=5745b59f-089a-4b13-9038-f82188d68a32, parentLandscape=WMIO]

From v10.15 onwards

  1. Go to the file e2ecustomlogback.xml at the following location:

    • Integration Server

      <INSTALL-DIR>\IntegrationServer\instances\default
    • Microservices Runtime

      <INSTALL-DIR>\IntegrationServer
  2. Apply the DEBUG logging level to the following properties:

    "e2eAgentSummary" level="DEBUG"  
    "e2eAgentSegments" level="DEBUG" 
    "e2eagentFailedSegments" level="DEBUG" 


  3. Here is a sample of the on-premises transaction trace:

    2021-08-06T19:49:50,328 DEBUG [e2eAgentAppender] - 
    [ENTRY]SpanInfo [traceId=[26655816.49441.45841902585488], segmentId:26655969.276.162825958992328, spanId:0, serviceName=forTesting:javaAsyncService1, startTime=1628259589923, duration=4ms, component=ONPREMISEIS, tenant=VMWODEPLAT01:5555, status=Success, requestID=5745b59f-089a-4b13-9038-f82188d68a32, parentLandscape=WMIO]

Are there exception cases where hybrid transaction traces will not work?

Detailed step-by-step tutorial

For a detailed tutorial, see Step-by-Step Tutorial - Hybrid Monitoring.

With Integration Server and Microservices Runtime through webMethods.io Integration on-premise connectors

Pre-requisites

  1. Click here for Hybrid Monitoring installation setup.
  2. For Hybrid Monitoring to work, webMethods Cloud tenant connection needs to be setup in Integration Server. Click here for more details.
  3. Audit needs to be enabled for on-premises services. Click here for more details.

Note:

  • From v10.15 onwards, product code UHM has changed to UHA. If you come across a file name with UHA in it, consider it the same as a UHM file.

Configuration

The following hybrid monitoring configurations are required in Integration Server and Microservices Runtime.

Integration Server

  1. After fix installation, copy the plugin jar file:

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
    • To directory:
      <INSTALL-DIR>\IntegrationServer\lib\jars\
  2. Setting up logs:

    For versions below 10.15: Copy log4j properties file:

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
    • To directory:
      <INSTALL-DIR>\IntegrationServer\

    From v10.15 onwards: Copy logback xml file

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml
    • To directory:
      <INSTALL-DIR>\IntegrationServer\
  3. Add the following lines at the end of the file:
    <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf

    • Microsoft Windows

      wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
      wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\
    • Linux

      wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
      wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/

    Note:
    From v10.15 onwards, product code UHM has changed to UHA. Accordingly, replace uhm-apm-agent.jar with uha-apm-agent.jar in the above lines.

  4. For versions below 10.15,

    Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, edit the property Dlog4j.configurationFile and append e2ecustomlog4j2.properties to it.

    • Microsoft Windows:

      • Existing

        wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties"
      • Appended

        wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"
    • Linux:

      • Existing

        wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties"
      • Appended

        wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"

    From v10.15 onwards,

    Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, add the property:

    • Microsoft Windows:

          wrapper.java.additional.503=-Dlogback.configurationFile=".\e2ecustomlogback.xml"
    • Linux:

          wrapper.java.additional.503=-Dlogback.configurationFile="./e2ecustomlogback.xml"
  5. If multiple tenant connections are supported in Integration Server and a custom tenant connection is being used, then open the file: <INSTALL-DIR>\E2EMonitoring\agent\config\agent.configand set the below properties:

Property Description
agent.onprem_multitenant_alias = ${SW_AGENT_ONPREM_MULTITENANT_ALIAS:default} Un-comment this property and replace default with the tenant connection name. Restart Integration Server.

Note:

  • When a custom tenant connection is not used, End-to-End Monitoring uses the default tenant connection value for agent.onprem_multitenant_alias in agent.config. You need to save the default tenant connection details in webMethods Integration Server when there is no custom connection.
  • Hybrid data transfer takes place within tenants of one given region only. When multiple tenant connections are created for different tenants belonging to the same region, specifying one of the tenant connection names for agent.onprem_multitenant_alias in agent.config is sufficient to trace hybrid transactions of other tenants belonging to the same region.
  • To see segment traces of on-premises API Gateway transaction in logs, set all log levels to DEBUG in versions below 10.15
    <INSTALL-DIR>\IntegrationServer\instances\default\e2ecustomlog4j2.properties
  • From v10.15 onwards, set log levels to debug in
    <INSTALL-DIR>\IntegrationServer\instances\default\e2elogback.xml

Click here for more details on Integration Server configurable properties.

Microservices Runtime

  1. After fix installation, copy the plugin jar file:

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-plugin.jar
    • To directory:
      <INSTALL-DIR>\IntegrationServer\lib\jars\
  2. Setting up logs:

    For versions below 10.15: Copy log4j properties file:

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
    • To directory:
      <INSTALL-DIR>\IntegrationServer\

    From v10.15 onwards: Copy logback xml file

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml
    • To directory:
      <INSTALL-DIR>\IntegrationServer\
  3. Add the following lines at the end of the file:

    • For versions below 10.15: Microsoft Windows

      <INSTALL-DIR>\IntegrationServer\bin\setenv.bat

      set JAVA_UHM_OPTS=-javaagent:"../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/" -Xbootclasspath/a:"../E2EMonitoring/agent/uhm-apm-agent.jar"
      set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHM_OPTS%
      set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% -Dlog4j.configurationFile="./e2ecustomlog4j2.properties"
    • For versions below 10.15: Linux

      <INSTALL-DIR>\IntegrationServer\bin\setenv.sh

      JAVA_UHM_OPTS="-javaagent:../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/ -Xbootclasspath/a:../E2EMonitoring/agent/uhm-apm-agent.jar"
      JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} ${JAVA_UHM_OPTS}"
      JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} -Dlog4j.configurationFile=./e2ecustomlog4j2.properties"
    • From v10.15 onwards: Microsoft Windows

      <INSTALL-DIR>\IntegrationServer\bin\setenv.bat

      set JAVA_UHM_OPTS=-javaagent:"../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/" -Xbootclasspath/a:"../E2EMonitoring/agent/uhm-apm-agent.jar"
      set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% %JAVA_UHM_OPTS%
      set JAVA_CUSTOM_OPTS=%JAVA_CUSTOM_OPTS% -Dlogback.configurationFile="./e2ecustomlogback.xml"
    • From v10.15 onwards: Linux

      <INSTALL-DIR>\IntegrationServer\bin\setenv.sh

      JAVA_UHM_OPTS="-javaagent:../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/ -Xbootclasspath/a:../E2EMonitoring/agent/uhm-apm-agent.jar"
      JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} ${JAVA_UHM_OPTS}"
      JAVA_CUSTOM_OPTS="${JAVA_CUSTOM_OPTS} -Dlogback.configurationFile=./ e2ecustomlogback.xml"

What about multiple runtimes?

There are scenarios when multiple runtimes like multiple Integration Server cluster nodes or Integration Server installations are pointing to the same end-to-end agent jar file. In such situations, you need to make hybrid monitoring configuration changes in each runtime. This is applicable to both Integration Server and Microservices Runtime.

What are the Integration Server and Microservices Runtime configurable properties?

Property Description
agent.onprem_jndi_auth_enable=${SW_AGENT_ONPREM_JNDI_AUTH_ENABLE:false} This property defines on-premises to cloud JNDI based authentication. Verify that it is set to false.
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES: uhm.e2em-hybrid-az-us.webmethods.io:443} In case of agent to collector connection issues, this property can be used to set collector endpoint so that the agent can connect to the specified collector. This property is not required to be set explicitly from v10.15 onwards, for v10.11 or v1.3.0 latest fixes (Fix 5 and above).

Note: Refer Collector end-point section for specific collector end-point values.

Click here for the common configurable properties in the End-to-End Monitoring agent available at
<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Note:

  • agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000}
  • This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes if you prefer a faster sync time.

With on-premises webMethods API Gateway

End-to-End Monitoring supports hybrid monitoring involving on-premises API Gateway v10.11 and later versions.

Note:

  • On-premises Integration Server and API Gateway plugins cannot be installed in the same installation. This is a known limitation.
  • From v10.15 onwards, product code UHM has changed to UHA. If you come across a file name with UHA in it, consider it the same as a UHM file.

Pre-requisites

  1. Click here for Hybrid Monitoring installation setup.
  2. For Hybrid Monitoring to work, webMethods Cloud tenant connection needs to be setup in Integration Server. Click here for more details.

Configuration

  1. Copy file: uhm-onpremise-is-http-plugin.jar

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\plugins\uhm-onpremise-is-http-plugin.jar
    • To directory:
      <INSTALL-DIR>\IntegrationServer\lib\jars\
  2. Copy file: uhm-api-onpremise-plugin.jar

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\plugins\ uhm-api-onpremise-plugin.jar
    • To directory:
      <INSTALL-DIR>\IntegrationServer\instances\default\packages\WmAPIGateway\code\jars
  3. Setting up logs:

    For versions below 10.15: Copy log4j properties file:

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlog4j2.properties
    • To directory:
      <INSTALL-DIR>\IntegrationServer\instances\default\

    From v10.15 onwards: Copy logback xml file

    • From directory:
      <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml
    • To directory:
      <INSTALL-DIR>\IntegrationServer\instances\default\
  4. Add the following lines at the end of the file:
    <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf

    • Microsoft Windows

      wrapper.java.additional.501=-Xbootclasspath/a:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar;"%JAVA_BOOT_CLASSPATH%"
      wrapper.java.additional.502=-javaagent:..\..\..\E2EMonitoring\agent\uhm-apm-agent.jar=logging.dir=.\logs\
    • Linux

      wrapper.java.additional.501=-Xbootclasspath/a:../../../E2EMonitoring/agent/uhm-apm-agent.jar:"%JAVA_BOOT_CLASSPATH%"
      wrapper.java.additional.502=-javaagent:../../../E2EMonitoring/agent/uhm-apm-agent.jar=logging.dir=./logs/   

    Note:
    From v10.15 onwards, product code UHM has changed to UHA. Accordingly, replace uhm-apm-agent.jar with uha-apm-agent.jar in the above lines.

  5. For versions below 10.15,

    Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, edit the property Dlog4j.configurationFile and append e2ecustomlog4j2.properties to it.

    • Microsoft Windows:

      • Existing

        wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties"
      • Appended

        wrapper.java.additional.204=Dlog4j.configurationFile="<INSTALL-DIR>\profiles\IS_default\configuration\logging\log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"
    • Linux:

      • Existing

        wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties"
      • Appended

        wrapper.java.additional.204=-Dlog4j.configurationFile="<INSTALL-DIR>/profiles/IS_default/configuration/logging/log4j2.properties,.tc.custom.log4j2.properties,e2ecustomlog4j2.properties"

    From v10.15 onwards,

    Open the file: <INSTALL-DIR>\profiles\IS_default\configuration\custom_wrapper.conf, add the property:

    • Microsoft Windows:

          wrapper.java.additional.503=-Dlogback.configurationFile=".\e2ecustomlogback.xml"
    • Linux:

          wrapper.java.additional.503=-Dlogback.configurationFile="./e2ecustomlogback.xml"

What are the API Gateway specific configurable properties?

Property Description
agent.service_name=${SW_AGENT_NAME:ONPREMISEIS} Indicates service name that monitors product calls.
Set SW_AGENT_NAME to ONPREMISEAPI, to monitor on-premise API Gateway calls.
agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:default} Indicates cloud tenant from which transactions to on-premise API Gateway takes place.
Tenant name is the value for SW_AGENT_ONPPREM_TENANT_ID.

New value: agent.onprem_tenant_id=${SW_AGENT_ONPPREM_TENANT_ID:<TenantName}
agent.onprem_multitenant_alias = ${SW_AGENT_ONPREM_MULTITENANT_ALIAS:default} Indicates name of the custom tenant connection used in Integration Server.
Custom tenant connection name created in the Integration Server is the value for the SW_AGENT_ONPREM_MULTITENANT_ALIAS.
This property is commented out by default, un-comment it if custom tenant connection is used in the Integration Server.
agent.apigw_tag=${SW_AGENT_APIGW_TAG:<custom tag>} End-to-End Monitoring, by default, monitors all the on-premises webMethods API Gateway calls with tag e2em. If you need to monitor an API containing custom tags , add this property in agent.config with custom tag value. To enable tracing, add the e2em tags to the webMethods API Gateway API, in case you do not have them already.
agent.onprem_jndi_auth_enable=${SW_AGENT_ONPREM_JNDI_AUTH_ENABLE:false} This property defines on-premises to cloud JNDI based authentication. Verify that it is set to false.
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES: uhm.e2em-hybrid-az-us.webmethods.io:443} In case of agent to collector connection issues, this property can be used to set collector endpoint so that the agent can connect to the specified collector. This property is not required to be set explicitly from v10.15 onwards, for v10.11 or v1.3.0 latest fixes (Fix 5 and above).

Note: Refer Collector end-point section for specific collector end-point values.

Click here for the common configurable properties in the End-to-End Monitoring agent available at
<INSTALL-DIR>\E2EMonitoring\agent\config\agent.config

Notes:

  • To see segment traces of on-premises API Gateway transaction in logs, set all log levels to DEBUG in
    <INSTALL-DIR>\IntegrationServer\instances\default\e2ecustomlog4j2.properties (For versions below 10.15)
    <INSTALL-DIR>\E2EMonitoring\agent\config\e2ecustomlogback.xml (From v10.15 onwards)
  • End-to-End Monitoring can trace transactions belonging to a single tenant, even if end-to-end transactions involve multiple API Gateway tenants at a time.
  • Hybrid monitoring works with on-premises Integration Server as well as on-premises API Gateway. However, End-to-End Monitoring does not support usage of both products together as part of hybrid integration.
  • End-to-End Monitoring by default monitors all the on-premises webMethods API Gateway calls with the tag e2em.
  • agent.trace_transfer_interval=${SW_AGENT_TRACE_TRANSFER_INTERVAL:180000}: This property defines the interval between each write operation to the file. The same property is used for data send interval through the gRPC channel. Default value is 18000 ms. It is recommended to reduce this time interval to shorter segment trace writes if you prefer a faster sync time.