Cloud Foundry (Tanzu)

Overview

This document details the Cavisson observability agent installation process in CloudFoundry platform to monitor its components and applications deployed in Tanzu Application Service. VMware Tanzu Application Service is an app development and deployment platform for public and private clouds. The observability agents provide metrics across the infrastructure and application components.

Installation Steps

Machine Agent Installation using BOSH release

1. Login to a VM which has BOSH CLI available and download the latest version of machine agent release . You can contact Cavisson sales to get the latest version.

wget https://nde.cav-test.com/HUB/CavMachineAgent-4.x.x.x.tar.gz –no-check-certificate

2.Untar the package downloaded in step1.

tar -xvzf CavMachineAgent-4.x.x.x.tar.gz

3. Go to the directory CavMachineAgent extracted after untar.

4. Clone the java release for BOSH Packages from the official bosh git repository.

git clone https://github.com/bosh-packages/java-release

Upon successful completion of the clone command, a new directory with the name ‘java-release’ will be created in the CavMachine agent directory. .

5. Vendor package the java-release package to Cavisson machine agent. This creates a local blobstore in /opt that contains the vendored package .

bosh vendor-package openjdk-8 java-release

6. Create bosh release for Cavisson machine agent

bosh create-release –sha2 –tarball=CavMachineAgent-4.x.x.x.tgz –version=4.x.x –name=CavMachineAgent

7. Upload the BOSH release to the BOSH Director.

bosh upload-release CavMachineAgent-4.x.x.x.tgz

8. Modify machine agent manifest file (located in the CavMachineAgent directory) to add the controller details and stemcell version.

Stemcells version can be obtained using command ‘bosh stemcells’.

Same version should be mentioned in the manifest file.

—

name: CavMachineAgent
version: 4.13.0.46
releases:
– name: CavMachineAgent
    version: 4.13.0.46
stemcells:
– alias: bosh-google-kvm-ubuntu-xenial-go_agent
    os: ubuntu-xenial
    version: “621.176”
instance_groups:
– name: cmon
    azs:
      – asia-south1-c
    instances: 1
    jobs:
      – name: cmon
        version: 4.13.0.46
        release: CavMachineAgent
    properties:
      cav_controller: nspoc2.cav-test.com:6666
      cav_tier: Tanzu
    vm_type: micro
    stemcell: bosh-google-kvm-ubuntu-xenial-go_agent
    networks:
      – name: pas
addons: []
update:
canaries: 1
max_in_flight: 1
canary_watch_time: 1000-60000
update_watch_time: 1000-60000

Sample Controller parameters:-

cav_controller	Mention Cavisson Server IP/Host where data is collected.
tiername	Mention the tier name as per your application architecture.
nfdbip and nfdbport(optional)	If user wants to add logs in a separate server.Mention Cavisson log collector IP/Host IP and port.
env_name(optional)	Mention Environment Name based on which environment is to be monitored. ex if it is preprod you can mention pre-prod env.

9. Deploy release using updated manifest file.

bosh -n -d CavMachineAgent deploy manifest.yml

10. Add same controller details in runtime.yml as mentioned in manifest.yml

addons:
– name: CavMachineAgentAddon
jobs:
– name: cmon
    release: CavMachineAgent
properties:
    cav_controller: nspoc2.cav-test.com:6666
    cav_tier: Tanzu
include:
    stemcell:
    – os: ubuntu-xenial
releases:
– name: CavMachineAgent
version: 4.13.0.46

11. Update the BOSH director runtime configuration .This runtime configuration applies to all new BOSH deployments going forward.

bosh update-runtime-config runtime.yml

12. Deploy your changes.

Since existing BOSH deployments won’t be automatically updated with the jobs specified in the runtime configuration, you need to redeploy them so that BOSH rolls out the Cavisson agent

Note:-Replace <cf-name> with your cf deployment name and <deployment yml path> with path of your cf deployment yml.

bosh -d <cf-name> deploy <deployment yml path>

Here is the video link for installation

https://www.cavisson.com/product-resources/netdiagnostics-enterprise-resources/videos-netdiagnostics/agent-installation-on-tas/

Youtube Video Link for Cavisson Agent installation on cloud foundry using BOSH release:-https://youtu.be/Pa7pNR6ewLk

Application Agent Installation using Sidecar

1. Login to a VM which has cf CLI available and download the latest version of application agent sidecar . You can contact Cavisson sales to get the latest version.

wget https://nde.cav-test.com/HUB/cavisson-agent-sidecar_4xx.zip –no-check-certificate

2. Create cavisson agent build pack

cf create-buildpack cavisson-agent-sidecar-buildpack cavisson-agent-sidecar_4xx.zip 1

3. Validate if buildpacks got successfully created

cf buildpacks |grep cavisson-agent-sidecar-buildpack

4. Modify your application manifest.yml to include cavisson buildpack created in step2 and add controller details.

The below snippet shows a sample configuration for an application called shopapp

applications:
– name: shopapp
    application: shopapp.war
    disk_quota: 2G
    instances: 2
    memory: 1G
    buildpacks:
    – cavisson-agent-sidecar-buildpack
    – java_buildpack_offline
    env:
      JBP_CONFIG_OPEN_JDK_JRE: ‘{ jre: { version: 11.+ }, memory_calculator: { stack_threads: 30 } }’
      JAVA_OPTS: ‘-XX:MaxDirectMemorySize=100M -javaagent:/home/vcap/deps/netdiagnostics/lib/ndmain.jar=time,ndAgentJar=/home/vcap/deps/netdiagnostics/lib/ndagent-with-dep.jar,ndHome=/home/vcap/deps/netdiagnostics’
      CAV_APP_AGENT_TIER: ShopApp
      CAV_APP_AGENT_NDCHOST: nspoc2.cav-test.com
      CAV_APP_AGENT_NDCPORT: 6666
      CAV_APP_AGENT_NDC_COMM_PROTOCOL: WSS
      CAV_APP_AGENT_SERVER: server1
      CAV_APP_AGENT_INSTANCE: instance

Parameters

*JAVA_OPTS*	JAVA_OPTS is an environment variable often used with Java applications to set various command-line options and system properties for the Java Virtual Machine (JVM) when the application is launched. This variable allows you to customize the behavior and configuration of the JVM according to your application’s needs.
*CAV_APP_AGENT_NDCHOST*	Mention Cavisson Server IP/Host where data is collected.
*CAV_APP_AGENT_TIER*	Tier is a logical entity that constitutes grouping of individual servers. The value entered here will be visible as a Tier in the metric tree in the product UI.
*CAV_APP_AGENT_SERVER*	Server constitutes grouping of instances. The server mentioned here comes under the tier defined in the above step in the metric tree.
*CAV_APP_AGENT_INSTANCE*	Provide an appropriate instance name. The instance(s) will be shown under Tier>Server in the metric tree and all the captured metrics will be visible under the instances.

5. Redeploy the application to get it effective.

Here is the video link for installation
https://www.cavisson.com/product-resources/netdiagnostics-enterprise-resources/videos-netdiagnostics/agent-installation-on-tas/

Youtube link for Java Application Agent Installation using sidecar:https://youtu.be/Q1C4EHjrJxg

Uninstallation Steps

1. To uninstall the machine agent, delete the release from your BOSH

bosh delete-release <Release Name>

2. To uninstall the Application agent , delete the buildpack

cf delete-buildpack cavisson-agent-sidecar-buildpack

Monitoring

Once you’ve completed the agent Installation you can able to see all listed metrics in Appendix section with in pre-bundled dashboard.

The dashboard provides deep level insights and facilitates a deeper understanding of trends and patterns which includes

Appendix

Metrics Group

Metrics Group	Metric
Cloud Foundry CPU Memory Stats Cloud Foundry CPU Memory Stats Cloud Foundry CPU Memory Stats
	Memory Active Anonymous (MB)
	Memory Active Anonymous Bytes/Sec
	Memory Inactive Anonymous (MB)
	Memory Inactive Anonymous Bytes/Sec
	Active File Memory (MB)
	Active File Bytes/Sec
	Inactive File Memory (MB)
	Inactive File Bytes/Sec
	Mapped File Memory (MB)
	Mapped File Memory Bytes/Sec
	Page Faults
	Page Faults/Sec
	Major Page Faults
	Major Page Faults/Sec
	PageIn
	PageIn/Sec
	PageOut
	PageOut/Sec
	RSS Memory (MB)
	RSS Memory Bytes/Sec
	Unevictable Memory(MB)
	Unevictable Memory Bytes/Sec
	Swap Memory (MB)
	Total Swap Memory Bytes/Sec
	Total Memory Usage Toward Limit
	Hierarchical Memory Limit (MB)
	Hierarchical Swap Memory Limit (MB)
	Cache Memory (MB)
	Total Cache Memory Bytes/Sec
	Memory Usage(Pct)
	Memory Usage(MB)
	CPU Usage(Cores)
	CPU Usage Time/Sec
	CPU User Usage Time/Sec
	CPU Usage User(Cores)
	CPU System Usage Time/Sec
	CPU Usage System(Cores)
	Disk Usage (MB)
	Total INodes Used
	Exclusive Disk Used (MB)
	Exclusive INodes Used
	Transferred Bytes/Sec
	Received Bytes/Sec
	Process Count
	Process Limit
	CPU Limit(Shares)
	Memory Limit(MB)
	CPU Cores

Cloud Foundry Garden Container IO Stats	IO Read Bytes/Sec
	IO Write Bytes/Sec
	IO Sync Bytes/Sec
	IO Async Bytes/Sec
	Total IO Bytes/Sec
CF Disk Stats	Disk Partition Size(MB)
	Disk Partition Used (MB)
	Disk Partition Available(MB)
	Disk Partition Used (Pct)
Cloud Foundry App Stats
Cloud Foundry App Stats	App Stats
Cloud Foundry Process Stats	Process CPU Usage
	Process Memory Usage
	Process Disk Usage
	Process Memory Quota
	Process Disk Quota
	Process State
Cloud Foundry Task Stats	Task State
	Task Memory Limit (MB)
	Task Disk Limit (MB)

Application Metrics Group

Metrics Group	Metric
Integration Points	Total Calls
	Calls/Sec
	Avg. Response Time
	Errors
	Average Network Delay in Request
	Average Network Delay in Response
	Errors/Sec
	Errors Response Time
	TCP Latency
	Bandwidth(KBPS)
Business Transaction	Requests per sec
	Average Response Time
	Average Request Size
	Errors/sec
	Normal Average Response Time
	Slow Average Response Time
	Very Slow Average Response Time
	Error Average Response Time
	Percent Slow and Very Slow Transactions
	Average Cpu Time
	Average Wait Time
	Average Sync Time
	Average Queue Time
Thread Hotspot Stats	Number Of Threads
	Number Of All Hotspot Threads
	All Hotspot Threads Duration (Secs)
	Number Of New Hotspot Threads
	New Hotspot Threads Duration (Secs)
	Number Of Blocked Threads
	Number Of Runnable Threads
	Number Of Time Waiting Threads
	Number Of Waiting Threads
	Number Of Terminated Threads

For details on stem cells refer to https://bosh.cloudfoundry.org/stemcells/.