GCP
|
5 Troubleshooting Commands
Contributed by Shea Stewart
Codecollection:
Uses promql on the Ops Suite API to determine the health of a Kong managed ingress resource
and pushes the result as an SLI metric. Produces a 1 for a healthy resource, or 0 for an unhealthy resource.
Tasks:
Tasks:
- Get Access Token
- Get HTTP Error Rate
- Get Upstream Health
- Get Request Latency Rate
- Generate Kong Ingress Score
Source Code
Run arbitrary gcloud commands and parse their output for arbitrary values such as json to be submitted as a metric.
Tasks:
Tasks:
- Run Gcloud CLI Command and Push metric
Run arbitrary gcloud commands and capture the stdout in a report.
Tasks:
Tasks:
- Run Gcloud CLI Command and Push metric
This codebundle sets up a monitor for a specific region and GCP Product, which is then periodically checked for
ongoing incidents based on the history available at https://status.cloud.google.com/incidents.json filtered based on severity level.
Tasks:
Tasks:
- Get Number of GCP Incidents Effecting My Workspace
Performs a metric query using a PromQL statement on the Ops Suite API
and pushes the result as an SLI metric.
Tasks:
Tasks:
- Run Prometheus Instant Query Against Google Prom API Endpoint
Performs a metric query using a Google MQL statement on the Ops Suite API
and pushes the result as an SLI metric.
Tasks:
Tasks:
- Running GCP OpsSuite Metric Query
Generate a link to the GCP Log Explorer.
Tasks:
Tasks:
- Get GCP Log Dashboard URL For Given Log Query
Uses promql on the Ops Suite API to determine the health of a MongoDB database instance
and pushes the result as an SLI metric. Produces a 1 for a healthy resource, or 0 for an unhealthy resource.
Tasks:
Tasks:
- Get Access Token
- Get Instance Status
- Get Connection Utilization Rate
- Get MongoDB Member State Health
- Get MongoDB Replication Lag
- Get MongoDB Queue Size
- Get Assertion Rate
- Generate MongoDB Score
Retrieve the number of results of a GCP Log Explorer query.
Tasks:
Tasks:
- Running GCE Logging Query And Pushing Result Count Metric
Troubleshoot GCE Ingress Resources related to GCP HTTP Load Balancer in GKE
Tasks:
Tasks:
- Search For GCE Ingress Warnings in GKE Context `CONTEXT`
- Identify Unhealthy GCE HTTP Ingress Backends in GKE Namespace `NAMESPACE`
- Validate GCP HTTP Load Balancer Configurations in GCP Project `GCP_PROJECT_ID`
- Fetch Network Error Logs from GCP Operations Manager for Ingress Backends in GCP Project `GCP_PROJECT_ID`
- Review GCP Operations Logging Dashboard in GCP project `GCP_PROJECT_ID`
Raises Issues
Collects Nginx ingress host controller metrics from GMP on GCP and inspects the results for ingress with a HTTP error code rate greater than zero
over a configurable duration and raises issues based on the number of ingress with error codes.
Tasks:
Tasks:
- Fetch Nginx HTTP Errors From GMP for Ingress `INGRESS_OBJECT_NAME`
- Find Owner and Service Health for Ingress `INGRESS_OBJECT_NAME`
Counts nodes that have been preempted within the defined time interval.
Tasks:
Tasks:
- Count the number of nodes in active preempt operation in project `${GCP_PROJECT_ID}`
List all GCP nodes that have been preempted in the previous time interval.
Tasks:
Tasks:
- List all nodes in an active preempt operation for GCP Project `GCP_PROJECT_ID` within the last `AGE` hours Show More
Discoverable
Collects Kong ingress host metrics from GMP on GCP and inspects the results for ingress with a HTTP error code rate greater than zero
over a configurable duration and raises issues based on the number of ingress with error codes.
Tasks:
Tasks:
- Check If Kong Ingress HTTP Error Rate Violates HTTP Error Threshold in GCP Project `GCP_PROJECT_ID`
- Check If Kong Ingress HTTP Request Latency Violates Threshold in GCP Project `GCP_PROJECT_ID`
- Check If Kong Ingress Controller Reports Upstream Errors in GCP Project `GCP_PROJECT_ID`
This SLI uses the GCP API or gcloud to score bucket health. Produces a value between 0 (completely failing thet test) and 1 (fully passing the test). Looks for usage above a threshold and public buckets.
Tasks:
Tasks:
- Fetch GCP Bucket Storage Utilization for `${PROJECT_IDS}`
- Check GCP Bucket Security Configuration for `${PROJECT_IDS}`
- Fetch GCP Bucket Storage Operations Rate for `${PROJECT_IDS}`
- Generate Bucket Score in Project `${PROJECT_IDS}`
Inspect GCP Storage bucket usage and configuration.
Tasks:
Tasks:
- Fetch GCP Bucket Storage Utilization for `PROJECT_IDS`
- Add GCP Bucket Storage Configuration for `PROJECT_IDS` to Report Show More
- Check GCP Bucket Security Configuration for `PROJECT_IDS`
- Fetch GCP Bucket Storage Operations Rate for `PROJECT_IDS`
Identify issues affecting GKE Clusters in a GCP Project and creates a health score. A score of 1 is healthy, a score between 0 and 1 indicates unhealthy components.
Tasks:
Tasks:
- Identify GKE Service Account Issues in GCP Project `${GCP_PROJECT_ID}`
- Fetch GKE Recommendations for GCP Project `${GCP_PROJECT_ID}`
- Fetch GKE Cluster Health for GCP Project `${GCP_PROJECT_ID}`
- Check for Quota Related GKE Autoscaling Issues in GCP Project `${GCP_PROJECT_ID}`
- Generate GKE Cluster Health Score
Identify issues affecting GKE Clusters in a GCP Project
Tasks:
Tasks:
- Identify GKE Service Account Issues in GCP Project `GCP_PROJECT_ID`
- Fetch GKE Recommendations for GCP Project `GCP_PROJECT_ID`
- Fetch GKE Cluster Health for GCP Project `GCP_PROJECT_ID`
- Check for Quota Related GKE Autoscaling Issues in GCP Project `GCP_PROJECT_ID`
- Validate GKE Node Sizes for GCP Project `GCP_PROJECT_ID`
- Fetch GKE Cluster Operations for GCP Project `GCP_PROJECT_ID`
Count the number of Cloud Functions in an unhealthy state for a GCP Project.
Tasks:
Tasks:
- Count unhealthy GCP Cloud Functions in GCP Project `${GCP_PROJECT_ID}`
Identify problems related to GCP Cloud Function deployments
Tasks:
Tasks:
- List Unhealthy Cloud Functions in GCP Project `GCP_PROJECT_ID`
- Get Error Logs for Unhealthy Cloud Functions in GCP Project `GCP_PROJECT_ID`
Fetches logs from a GCP using a configurable query and raises an issue with details on the most common issues.
Tasks:
Tasks:
- Inspect GCP Logs For Common Errors in GCP Project `GCP_PROJECT_ID`
Runs a user provided gcloud command and pushes the metric to the RunWhen Platform. The supplied command must result in distinct single metric. Command line tools like jq are available.
Tasks:
Tasks:
- ${TASK_TITLE}
Runs a user provided gcloud command
Tasks:
Tasks:
- TASK_TITLE