{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rickymoorhouse.uk/blog/2020/trawler/index",
  "path": "/blog/2020/trawler/index",
  "publishedAt": "2020-06-02T00:00:00.000Z",
  "site": "at://did:plc:r53zv4vpzeihop3aliwyejlu/site.standard.publication/3mos5q3a7jf2w",
  "tags": [
    "apiconnect",
    "monitoring",
    "trawler"
  ],
  "textContent": "As part of our work in running and monitoring our API Connect cloud deployments we've built some of our own tooling to assist with monitoring what is going on within the deployments.  Trawler is one of these items which is used to gather metrics from a Kubernetes based deployment of API Connect. \n\nTrawler runs within kubernetes alongside API Connect and identifies the API Connect components and exposes metrics to prometheus (or other compatible monitoring tooling)\n\nThis data can then be used to feed into dashboards such as this one in Grafana:\n\nTrawler is open-source and available on github and docker hub - See the installation guide for more information on using trawler for yourself.\n\n\n\nThe kind of metrics that trawler collects are currently as follows:\n\nManagement subsystem:\n - API Connect version information (apiconnect_build_info)\n - Total users (apiconnect_users_total)\n - Number of provider_orgs (apiconnect_provider_orgs_total)\n - Number of consumer orgs (apiconnect_consumer_orgs_total)\n - Number of catalogs (apiconnect_catalogs_total)\n - Number of draft products / apis (apiconnect_draft_products_total / apiconnect_draft_apis_total)\n - Number of products / apis (apiconnect_products_total / apiconnect_apis_total)\n - Number of subscriptions (apiconnect_subscriptions_total)\n\nDataPower subsystem:\n - TCP connection stats (datapower_tcp...)\n - Log target stats: events processed, dropped, pending (datapower_logtarget...)\n - Object counts e.g. SSLClientProfile, APICollection, APIOperation etc. (datapower_{object}_total) \n - HTTP Stats (datapower_http_tenSeconds/oneMinute/tenMinutes/oneDay)\n\nAnalytics subsystem\n\n - Cluster health status (analytics_cluster_status)\n - Number of nodes in the cluster (analytics_data_nodes_total/analytics_nodes_total)\n - Number of shards in states - active, relocating, initialising, unassigned (analytics_{state}_shards_total)\n - Number of pending tasks (analytics_pending_tasks_total)",
  "title": "Trawler - Metric gathering for API Connect"
}