KUBERNETES
|
1 Troubleshooting Commands
Contributed by stewartshea
Codecollection:
This taskset runs a user provided kubectl command and adds the output to the report. Command line tools like jq are available.
Tasks:
Tasks:
- TASK_TITLE
Source Code
This taskset runs a user provided kubectl command and pushes the metric. The supplied command must result in distinct single metric. Command line tools like jq are available.
Tasks:
Tasks:
- ${TASK_TITLE}
Runs an ad-hoc user-provided command, and if the provided command outputs a non-empty string to stdout then an issue is generated with a configurable title and content.
User commands should filter expected/healthy content (eg: with grep) and only output found errors.
Tasks:
Tasks:
- TASK_TITLE
Runs an ad-hoc user-provided command, and if the provided command outputs a non-empty string to stdout then a health score of 0 (unhealthy) is pushed, otherwise if there is no output, indicating no issues, then a 1 is pushed.
User commands should filter expected/healthy content (eg: with grep) and only output found errors.
Tasks:
Tasks:
- ${TASK_TITLE}
Retreieve aggregate data via kubectl top command.
Tasks:
Tasks:
- Running Kubectl Top And Extracting Metric Data
Uses kubectl to query the state of a ingestor ring. Returns the json of injester id, status and timestamp.
Tasks:
Tasks:
- Fetch Ingestor Ring Member List and Status
Uses kubectl to query the state of a ingestor ring and determine if it's healthy. Returns 1 if healthy, 0 if unhealthy.
Tasks:
Tasks:
- Determine Cortex Ingester Ring Health
Uses kubectl (or equivalent) to query the state of a patroni cluster and determine if it's healthy.
Tasks:
Tasks:
- Determine Patroni Health
Check the health of a Kubernetes API server using kubectl.
Returns 1 when OK, or a 0 in the case of an unhealthy API server.
Tasks:
Tasks:
- Running Kubectl Check Against API Server
A taskset for troubleshooting general issues associated with typical kubernetes deployment resources.
Supports API interactions via both the API client and Kubectl binary through RunWhen Shell Services.
Tasks:
Tasks:
- Troubleshoot Resourcing
- Troubleshoot Events
- Troubleshoot PVC
- Troubleshoot Pods
This codebundle runs an arbitrary kubectl command and writes the stdout to a report.
Typically used in conjunction with other codebundles.
Tasks:
Tasks:
- Running Kubectl And Adding Stdout To Report
This taskset runs general troubleshooting checks against all applicable objects in a namespace, checks error events, and searches pod logs for error entries.
Tasks:
Tasks:
- Trace Namespace Errors
- Fetch Unready Pods
- Triage Namespace
- Object Condition Check
- Namespace Get All
This SLI uses kubectl to score namespace health. Produces a value between 0 (completely failing thet test) and 1 (fully passing the test). Looks for container restarts, events, and pods not ready.
Tasks:
Tasks:
- Get Event Count and Score
- Get Container Restarts and Score
- Get NotReady Pods
- Generate Namspace Score
This codebundle runs a kubectl get command that produces a value and pushes the metric.
Uses jmespath for filtering and allows calculations such as count, sum, avg on specified fields.
Tasks:
Tasks:
- Running Kubectl get and push the metric
A taskset for troubleshooting issues for StatefulSets and their related resources.
Tasks:
Tasks:
- Check StatefulSets Replicas Ready
- Get Events For The StatefulSet
- Get StatefulSet Logs
- Get StatefulSet Manifests Dump
Runs multiple Kubernetes and psql commands to report on the health of a postgres cluster.
Tasks:
Tasks:
- Get Standard Resources
- Describe Custom Resources
- Get Pod Logs & Events
- Get Pod Resource Utilization
- Get Running Configuration
- Get Patroni Output
- Run DB Queries
Searches a namespace for matching objects and provides the commands to decommission them.
Tasks:
Tasks:
- Generate Decomission Commands
Returns the number of events with matching messages as an SLI metric.
Tasks:
Tasks:
- Get Number Of Matching Events
Triages issues related to a deployment's replicas.
Tasks:
Tasks:
- Fetch Logs
- Get Related Events
- Check Deployment Replicas
Detects and reinitializes laggy Patroni cluster members which are unable to catchup in replication using kubectl and patronictl.
Tasks:
Tasks:
- Determine Patroni Health
Measures the maximum replica lag across a Patroni cluster.
Tasks:
Tasks:
- Measure Patroni Member Lag
Checks that the current state of a daemonset is healthy and returns a score of either 1 (healthy) or 0 (unhealthy).
Tasks:
Tasks:
- Health Check Daemonset
Creates an adhoc one-shot job which mounts a PVC as a canary test, which is polled for success before being torn down.
Tasks:
Tasks:
- Run Canary Job
Taskset to triage issues related to patroni.
Tasks:
Tasks:
- Get Patroni Status
- Get Pods Status
- Fetch Logs
Runs a postgres SQL query and pushes the returned result into a report.
During execution, the SQL query should be passed to a Kubernetes workload that has access to the psql binary.
The workload will run the query and return the results from stdout.
Tasks:
Tasks:
- Run Postgres Query And Results to Report
Triages issues related to a StatefulSet and its pods, including persistent volumes and ordered deployment characteristics.
Tasks:
Tasks:
- Analyze Application Log Patterns for StatefulSet `STATEFULSET_NAME` in Namespace `NAMESPACE`
- Detect Log Anomalies for StatefulSet `STATEFULSET_NAME` in Namespace `NAMESPACE`
- Check Liveness Probe Configuration for StatefulSet `STATEFULSET_NAME`
- Check Readiness Probe Configuration for StatefulSet `STATEFULSET_NAME` in Namespace `NAMESPACE`
- Inspect StatefulSet Warning Events for `STATEFULSET_NAME` in Namespace `NAMESPACE`
- Fetch StatefulSet Workload Details For `STATEFULSET_NAME` in Namespace `NAMESPACE`
- Inspect StatefulSet Replicas for `STATEFULSET_NAME` in namespace `NAMESPACE`
- Check StatefulSet PersistentVolumeClaims for `STATEFULSET_NAME` in Namespace `NAMESPACE`
- Identify Recent Configuration Changes for StatefulSet `STATEFULSET_NAME` in Namespace `NAMESPACE`
Discoverable
Runs diagnostic checks against an AKS cluster.
Tasks:
Tasks:
- Check for Resource Health Issues Affecting AKS Cluster `AKS_CLUSTER` In Resource Group `AZ_RESOURCE_GROUP`
- Check Configuration Health of AKS Cluster `AKS_CLUSTER` In Resource Group `AZ_RESOURCE_GROUP`
- Check Network Configuration of AKS Cluster `AKS_CLUSTER` In Resource Group `AZ_RESOURCE_GROUP`
- Fetch Activities for AKS Cluster `AKS_CLUSTER` In Resource Group `AZ_RESOURCE_GROUP`
Provides a list of tasks that can remediate configuraiton issues with manifests in GitHub based GitOps repositories.
Tasks:
Tasks:
- Remediate Readiness and Liveness Probe GitOps Manifests in Namespace `NAMESPACE`
- Increase ResourceQuota Limit for Namespace `NAMESPACE` in GitHub GitOps Repository
- Adjust Pod Resources to Match VPA Recommendation in `NAMESPACE`
- Expand Persistent Volume Claims in Namespace `NAMESPACE`
Checks the overall health of certificates in a namespace that are managed by cert-manager.
Tasks:
Tasks:
- Get Namespace Certificate Summary for Namespace `NAMESPACE`
- Find Unhealthy Certificates in Namespace `NAMESPACE`
- Find Failed Certificate Requests and Identify Issues for Namespace `NAMESPACE`
Raises Issues
Counts the number of unhealthy cert-manager managed certificates in a namespace.
Tasks:
Tasks:
- Count Unready and Expired Certificates in Namespace `${NAMESPACE}`
This taskset collects information about perstistent volumes and persistent volume claims to
validate health or help troubleshoot potential issues.
Tasks:
Tasks:
- Query The Jenkins Kubernetes Workload HTTP Endpoint in Kubernetes StatefulSet `STATEFULSET_NAME`
- Query For Stuck Jenkins Jobs in Kubernetes Statefulset Workload `STATEFULSET_NAME`
This taskset provides tasks to troubleshoot service accounts in a Kubernetes namespace.
Tasks:
Tasks:
- Test Service Account Access to Kubernetes API Server in Namespace `NAMESPACE`
Provides chaos injection tasks for specific workloads like your apps in a Kubernetes namespace. These are destructive tasks and the expectation is that you can heal these changes by enabling your GitOps reconciliation.
Tasks:
Tasks:
- Test `WORKLOAD_NAME` High Availability in Namespace `NAMESPACE`
- OOMKill `WORKLOAD_NAME` Pod
- Mangle Service Selector For `WORKLOAD_NAME` in `NAMESPACE`
- Mangle Service Port For `WORKLOAD_NAME` in `NAMESPACE`
- Fill Tmp Directory Of Pod From `WORKLOAD_NAME`
This codebundle runs a series of tasks to identify potential helm release issues related to Flux managed Helm objects.
Tasks:
Tasks:
- List all available FluxCD Helmreleases in Namespace `NAMESPACE`
- Fetch Installed FluxCD Helmrelease Versions in Namespace `NAMESPACE`
- Fetch Mismatched FluxCD HelmRelease Version in Namespace `NAMESPACE`
- Fetch FluxCD HelmRelease Error Messages in Namespace `NAMESPACE`
- Check for Available Helm Chart Updates in Namespace `NAMESPACE`
Perform oprational tasks for a Kubernetes deployment.
Tasks:
Tasks:
- Restart Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Force Delete Pods in Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Rollback Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE` to Previous Version
- Scale Down Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Scale Up Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE` by SCALE_UP_FACTORx
- Clean Up Stale ReplicaSets for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Scale Down Stale ReplicaSets for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
Runs a series of tasks to check the overall health of a postgres cluster and to provide detailed information useful for debugging or reviewing configurations.
Tasks:
Tasks:
- List Resources Related to Postgres Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Get Postgres Pod Logs & Events for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Get Postgres Pod Resource Utilization for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Get Running Postgres Configuration for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Get Patroni Output and Add to Report for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Fetch Patroni Database Lag for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Check Database Backup Status for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Run DB Queries for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
Runs multiple Kubernetes and psql commands to report on the health of a postgres cluster. Produces a value between 0 (completely failing thet test) and 1 (fully passing the test). Checks for database lag & backup health.
Tasks:
Tasks:
- Check Patroni Database Lag in Namespace `${NAMESPACE}` on Host `${HOSTNAME}` using `patronictl`
- Check Database Backup Status for Cluster `${OBJECT_NAME}` in Namespace `${NAMESPACE}`
- Generate Namespace Score for Namespace `${NAMESPACE}`
Generates a report of the reconciliation errors for fluxcd in your cluster.
Tasks:
Tasks:
- Check FluxCD Reconciliation Health in Kubernetes Namespace `FLUX_NAMESPACE`
Measures failing reconciliations for fluxcd
Tasks:
Tasks:
- Health Check Flux Reconciliation
Triages issues related to a Azure Loadbalancers and its activity logs.
Tasks:
Tasks:
- Check Activity Logs for Azure Load Balancer `AZ_LB_NAME`
This taskset restarts a resource with a given set of labels, typically used with other tasksets.
Tasks:
Tasks:
- Get Current Resource State with Labels `LABELS`
- Get Resource Logs with Labels `LABELS`
- Restart Resource with Labels `LABELS` in `CONTEXT`
Detects and analyzes stacktraces/tracebacks in Kubernetes workload logs for troubleshooting application issues.
Tasks:
Tasks:
- Analyze Workload Stacktraces for WORKLOAD_TYPE `WORKLOAD_NAME` in Namespace `NAMESPACE`
This SLI monitors stacktrace health in kubernetes workload application logs. Produces a value between 0 (stacktraces detected) and 1 (no stacktraces found). Focuses specifically on application error detection through stacktrace analysis.
Tasks:
Tasks:
- Get Stacktrace Health Score for ${WORKLOAD_TYPE} `${WORKLOAD_NAME}`
- Generate Stacktrace Health Score for `${WORKLOAD_NAME}`
This taskset investigates the logs, state and health of Kubernetes Prometheus operator.
Tasks:
Tasks:
- Check Prometheus Service Monitors in namespace `NAMESPACE`
- Check For Successful Rule Setup in Kubernetes Namespace `NAMESPACE`
- Verify Prometheus RBAC Can Access ServiceMonitors in Namespace `PROM_NAMESPACE`
- Inspect Prometheus Operator Logs for Scraping Errors in Namespace `NAMESPACE`
- Check Prometheus API Healthy in Namespace `PROM_NAMESPACE`
Evaluate cluster node health using kubectl
Tasks:
Tasks:
- Check for Node Restarts in Cluster `CONTEXT` within Interval `INTERVAL`
Evaluate cluster node health using kubectl.
Tasks:
Tasks:
- Check for Node Restarts in Cluster `${CONTEXT}`
- Generate Namespace Score in Kubernetes Cluster `$${CONTEXT}`
This taskset collects information and runs general troubleshooting checks against argocd application objects within a namespace.
Tasks:
Tasks:
- Fetch ArgoCD Application Sync Status & Health for `APPLICATION`
- Fetch ArgoCD Application Last Sync Operation Details for `APPLICATION`
- Fetch Unhealthy ArgoCD Application Resources for `APPLICATION`
- Scan For Errors in Pod Logs Related to ArgoCD Application `APPLICATION`
- Fully Describe ArgoCD Application `APPLICATION`
Performs a triage on the Open Source version of Artifactory in a Kubernetes cluster.
Tasks:
Tasks:
- Check Artifactory Liveness and Readiness Endpoints in `NAMESPACE`
Triages issues related to a ingress objects and services.
Tasks:
Tasks:
- Fetch Ingress Object Health in Namespace `NAMESPACE`
- Check for Ingress and Service Conflicts in Namespace `NAMESPACE`
Performs application-level troubleshooting by inspecting the logs of a workload for parsable exceptions,
and attempts to determine next steps.
Tasks:
Tasks:
- Get `CONTAINER_NAME` Application Logs in Namespace `NAMESPACE`
- Tail `CONTAINER_NAME` Application Logs For Stacktraces
Measures the number of exception stacktraces present in an application's logs over a time period.
Tasks:
Tasks:
- Tail `${CONTAINER_NAME}` Application Logs For Stacktraces
Identify resource constraints or issues in a cluster.
Tasks:
Tasks:
- Identify High Utilization Nodes for Cluster `CONTEXT`
- Identify Pods Causing High Node Utilization in Cluster `CONTEXT`
- Identify Pods with Resource Limits Exceeding Node Capacity in Cluster `CONTEXT`
Counts the number of nodes above 90% CPU or Memory Utilization from kubectl top.
Tasks:
Tasks:
- Identify High Utilization Nodes for Cluster `${CONTEXT}`
- Identify Pods with Resource Limits Exceeding Node Capacity in Cluster `${CONTEXT}`
- Generate Cluster Resource Health Score
Provides chaos injection tasks for Kubernetes namespaces. These are destructive tasks and the expectation is that you can heal these changes by enabling your GitOps reconciliation.
Tasks:
Tasks:
- Kill Random Pods In Namespace `NAMESPACE`
- OOMKill Pods In Namespace `NAMESPACE`
- Mangle Service Selector In Namespace `NAMESPACE`
- Mangle Service Port In Namespace `NAMESPACE`
- Fill Random Pod Tmp Directory In Namespace `NAMESPACE`
PostgreSQL Operations Runbook for Kubernetes clusters
Supports CrunchyDB and Zalando PostgreSQL operators
Primary focus on reinitializing failed cluster members
Tasks:
Tasks:
- Reinitialize Failed PostgreSQL Cluster Members for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Perform PostgreSQL Cluster Failover Operation for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Restart PostgreSQL Cluster with Rolling Update for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
- Verify Cluster Recovery and Generate Summary for Cluster `OBJECT_NAME` in Namespace `NAMESPACE`
This taskset performs diagnostic checks on a OpenTelemetry Collector to ensure it's pushing metrics.
Tasks:
Tasks:
- Query Collector Queued Spans in Namespace `NAMESPACE`
- Check OpenTelemetry Collector Logs For Errors In Namespace `NAMESPACE`
- Query OpenTelemetry Logs For Dropped Spans In Namespace `NAMESPACE`
This taskset is used to suspend a flux resource for the purposes of executing chaos tasks.
Tasks:
Tasks:
- Suspend the Flux Resource Reconciliation for `FLUX_RESOURCE_NAME` in namespace `FLUX_RESOURCE_NAMESPACE`
- Select Random FluxCD Workload for Chaos Target in Namespace `FLUX_RESOURCE_NAMESPACE`
- Execute Chaos Command on `TARGET_RESOURCE` in Namespace `TARGET_NAMESPACE`
- Execute Additional Chaos Command on FLUX_RESOURCE_TYPE 'FLUX_RESOURCE_NAME' in namespace 'FLUX_RESOURCE_NAMESPACE'
- Resume Flux Resource Reconciliation in `TARGET_NAMESPACE`
This codebundle runs a series of tasks to identify potential Kustomization issues related to Flux managed Kustomization objects.
Tasks:
Tasks:
- List All FluxCD Kustomization objects in Namespace `NAMESPACE` in Cluster `CONTEXT`
- List Suspended FluxCD Kustomization objects in Namespace `NAMESPACE` in Cluster `CONTEXT`
- List Unready FluxCD Kustomizations in Namespace `NAMESPACE` in Cluster `CONTEXT`
This codebundle checks for unhealthy or suspended FluxCD Kustomization objects.
Tasks:
Tasks:
- List Suspended FluxCD Kustomization objects in Namespace `${NAMESPACE}` in Cluster `${CONTEXT}`
- List Unready FluxCD Kustomizations in Namespace `${NAMESPACE}` in Cluster `${CONTEXT}`
- Generate FluxCD Kustomization Health Score for Namespace `${NAMESPACE}` in Cluster `${CONTEXT}`
This codebundle runs a series of tasks to identify potential helm release issues related to ArgoCD managed Helm objects.
Tasks:
Tasks:
- Fetch all available ArgoCD Helm releases in namespace `NAMESPACE`
- Fetch Installed ArgoCD Helm release versions in namespace `NAMESPACE`
Inspects the resources provisioned for a given set of pods and raises issues or recommendations as necessary.
Tasks:
Tasks:
- Show Pods Without Resource Limit or Resource Requests Set in Namespace `NAMESPACE`
- Check Pod Resource Utilization with Top in Namespace `NAMESPACE`
- Identify VPA Pod Resource Recommendations in Namespace `NAMESPACE`
- Identify Overutilized Pods in Namespace `NAMESPACE`
This taskset queries Jaeger API directly for trace details and parses the results
Tasks:
Tasks:
- Query Traces in Jaeger for Unhealthy HTTP Response Codes in Namespace `NAMESPACE`
A suite of tasks that can be used to triage potential issues in your vault namespace.
Tasks:
Tasks:
- Fetch Vault CSI Driver Logs in Namespace `NAMESPACE`
- Get Vault CSI Driver Warning Events in `NAMESPACE`
- Check Vault CSI Driver Replicas Show More
- Fetch Vault Pod Workload Logs in Namespace `NAMESPACE` with Labels `LABELS`
- Get Related Vault Events in Namespace `NAMESPACE`
- Fetch Vault StatefulSet Manifest Details in `NAMESPACE`
- Fetch Vault DaemonSet Manifest Details in Kubernetes Cluster `NAMESPACE`
- Verify Vault Availability in Namespace `NAMESPACE` and Context `CONTEXT`
- Check Vault StatefulSet Replicas in `NAMESPACE`
Triages issues related to a DaemonSet and its pods, including node scheduling and resource constraints.
Tasks:
Tasks:
- Analyze Application Log Patterns for DaemonSet `DAEMONSET_NAME` in Namespace `NAMESPACE`
- Detect Log Anomalies for DaemonSet `DAEMONSET_NAME` in Namespace `NAMESPACE`
- Identify Recent Configuration Changes for DaemonSet `DAEMONSET_NAME` in Namespace `NAMESPACE`
- Check Liveness Probe Configuration for DaemonSet `DAEMONSET_NAME`
- Check Readiness Probe Configuration for DaemonSet `DAEMONSET_NAME` in Namespace `NAMESPACE`
- Inspect DaemonSet Warning Events for `DAEMONSET_NAME` in Namespace `NAMESPACE`
- Fetch DaemonSet Workload Details For `DAEMONSET_NAME` in Namespace `NAMESPACE`
- Inspect DaemonSet Status for `DAEMONSET_NAME` in namespace `NAMESPACE`
- Check Node Affinity and Tolerations for DaemonSet `DAEMONSET_NAME` in Namespace `NAMESPACE`
Checks istio proxy sidecar injection status, high memory and cpu usage, warnings and errors in logs, valid certificates, configuration and verify istio installation.
Tasks:
Tasks:
- Verify Istio Sidecar Injection for Cluster `CONTEXT`
- Check Istio Sidecar Resource Usage for Cluster `CONTEXT`
- Validate Istio Installation in Cluster `CONTEXT`
- Check Istio Controlplane Logs For Errors in Cluster `CONTEXT`
- Fetch Istio Proxy Logs in Cluster `CONTEXT`
- Verify Istio SSL Certificates in Cluster `CONTEXT`
- Check Istio Configuration Health in Cluster `CONTEXT`
Checks istio proxy sidecar injection status, high memory and cpu usage, warnings and errors in logs, valid certificates, configuration and verify istio installation.
Tasks:
Tasks:
- Verify Istio Sidecar Injection for Cluster `${CONTEXT}`
- Check Istio Sidecar Resource Usage for Cluster `${CONTEXT}`
- Validate Istio Installation in Cluster `${CONTEXT}`
- Check Istio Controlplane Logs For Errors in Cluster `${CONTEXT}`
- Fetch Istio Proxy Logs in Cluster `${CONTEXT}`
- Verify Istio SSL Certificates in Cluster `${CONTEXT}`
- Check Istio Configuration Health in Cluster `${CONTEXT}`
- Generate Health Score for Cluster ${CONTEXT}
Performs application-level troubleshooting by inspecting the logs of a workload for parsable exceptions,
and attempts to determine next steps.
Tasks:
Tasks:
- Get `CONTAINER_NAME` Application Logs from Workload `WORKLOAD_NAME` in Namespace `NAMESPACE`
- Scan `CONTAINER_NAME` Application For Misconfigured Environment
- Tail `CONTAINER_NAME` Application Logs For Stacktraces in Workload `WORKLOAD_NAME`
Measures the number of exception stacktraces present in an application's logs over a time period.
Tasks:
Tasks:
- Measure Application Exceptions in `${NAMESPACE}`
This taskset provides detailed information about the images used in a Kubernetes namespace.
Tasks:
Tasks:
- Check Image Rollover Times for Namespace `NAMESPACE`
- List Images and Tags for Every Container in Running Pods for Namespace `NAMESPACE`
- List Images and Tags for Every Container in Failed Pods for Namespace `NAMESPACE`
- List ImagePullBackOff Events and Test Path and Tags for Namespace `NAMESPACE`
This taskset collects information on your redis workload in your Kubernetes cluster and raises issues if any health checks fail.
Tasks:
Tasks:
This taskset runs general troubleshooting checks against all applicable objects in a namespace. Looks for warning events, odd or frequent normal events, restarting containers and failed or pending pods.
Tasks:
Tasks:
- Inspect Warning Events in Namespace `NAMESPACE`
- Inspect Container Restarts In Namespace `NAMESPACE`
- Inspect Pending Pods In Namespace `NAMESPACE`
- Inspect Failed Pods In Namespace `NAMESPACE`
- Inspect Workload Status Conditions In Namespace `NAMESPACE`
- Get Listing Of Resources In Namespace `NAMESPACE`
- Check Event Anomalies in Namespace `NAMESPACE`
- Check Missing or Risky PodDisruptionBudget Policies in Namepace `NAMESPACE`
- Check Resource Quota Utilization in Namespace `NAMESPACE`
This SLI uses kubectl to score namespace health. Produces a value between 0 (completely failing thet test) and 1 (fully passing the test). Looks for container restarts, events, and pods not ready.
Tasks:
Tasks:
- Get Error Event Count within ${EVENT_AGE} and calculate Score
- Get Container Restarts and Score in Namespace `${NAMESPACE}`
- Get NotReady Pods in `${NAMESPACE}`
- Generate Namespace Score in `${NAMESPACE}`
This taskset collects information about storage such as PersistentVolumes and PersistentVolumeClaims to
validate health or help troubleshoot potential storage issues.
Tasks:
Tasks:
- Fetch Events for Unhealthy Kubernetes PersistentVolumeClaims in Namespace `NAMESPACE`
- List PersistentVolumeClaims in Terminating State in Namespace `NAMESPACE`
- List PersistentVolumes in Terminating State in Namespace `NAMESPACE`
- List Pods with Attached Volumes and Related PersistentVolume Details in Namespace `NAMESPACE`
- Fetch the Storage Utilization for PVC Mounts in Namespace `NAMESPACE`
- Check for RWO Persistent Volume Node Attachment Issues in Namespace `NAMESPACE`
This SLI collects information about storage such as PersistentVolumes and PersistentVolumeClaims and generates an aggregated health score for the namespace. 1 = Healthy, 0 = Failed, >0 <1 = Degraded
Tasks:
Tasks:
- Fetch the Storage Utilization for PVC Mounts in Namespace `${NAMESPACE}`
- Generate Namespace Score for Namespace `${NAMESPACE}`
Troubleshoot GCE Ingress Resources related to GCP HTTP Load Balancer in GKE
Tasks:
Tasks:
- Search For GCE Ingress Warnings in GKE Context `CONTEXT`
- Identify Unhealthy GCE HTTP Ingress Backends in GKE Namespace `NAMESPACE`
- Validate GCP HTTP Load Balancer Configurations in GCP Project `GCP_PROJECT_ID`
- Fetch Network Error Logs from GCP Operations Manager for Ingress Backends in GCP Project `GCP_PROJECT_ID`
- Review GCP Operations Logging Dashboard in GCP project `GCP_PROJECT_ID`
Triages issues related to a deployment and its replicas.
Tasks:
Tasks:
- Analyze Application Log Patterns for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Detect Log Anomalies for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Fetch Deployment Logs for `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Check Liveness Probe Configuration for Deployment `DEPLOYMENT_NAME`
- Check Readiness Probe Configuration for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Inspect Deployment Warning Events for `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Check Deployment Replica Status for `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Inspect Container Restarts for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
- Identify Recent Configuration Changes for Deployment `DEPLOYMENT_NAME` in Namespace `NAMESPACE`
This SLI uses kubectl to score deployment health. Produces a value between 0 (completely failing the test) and 1 (fully passing the test). Looks for container restarts, critical log errors, pods not ready, deployment status, and recent events.
Tasks:
Tasks:
- Get Container Restarts and Score for Deployment `${DEPLOYMENT_NAME}`
- Get Critical Log Errors and Score for Deployment `${DEPLOYMENT_NAME}`
- Get NotReady Pods Score for Deployment `${DEPLOYMENT_NAME}`
- Get Deployment Replica Status and Score for `${DEPLOYMENT_NAME}`
- Get Recent Warning Events Score for `${DEPLOYMENT_NAME}`
- Generate Deployment Health Score for `${DEPLOYMENT_NAME}`
This codebundle fetches the number of running pods with the set of provided labels, letting you measure the number of running pods.
Tasks:
Tasks:
- Measure Number of Running Pods with Label in `${NAMESPACE}`