gRPC

Setup

gRPC Go recently acquired support for Interceptors, i.e. middleware that is executed by a gRPC Server before the request is passed onto the user's application logic. It is a perfect way to implement common patterns: auth, logging, and... monitoring.

To use Interceptors in chains, please see link

This library requires Go 1.9 or later.

For enabling prometheus metrics interceptor in your application you can refer the following link here

Once you enable it verify your setup by checking the following metrics

  • grpc_server_handled_total

  • grpc_server_handling_seconds_bucket

Metrics

Metric

Key Performance Indicator(KPI)

Requests

grpc_server_handled_total

Request Rate

rate(grpc_server_handled_total[5m])

Errors

grpc_server_handled_total{grpc_code!="OK"}

Error Ratio

rate(grpc_server_handled_total{grpc_code!="OK"}[5m])/ rate(grpc_server_handled_total[5m])

Latency

grpc_server_handling_seconds_count

grpc_server_handling_seconds_bucket

Latency Average

rate(grpc_server_handling_seconds_count[5m])/ rate(grpc_server_handling_seconds_count[5m])

Latency P99

histogram_quantile (

0.99,

sum(rate(grpc_server_handling_seconds_bucket[1m])) by (le)

)

Alerts

KPI

Alerts

Request Rate

RequestRateAnomaly

Error Rate

ErrorRatioBreach

ErrorBuildup based on a 99.9 SLO

Latency Average

Latency P99

LatencyAverageBreach

LatencyAverageAnomaly

LatencyP99ErrorBuildup

Dashboards

Service KPI Dashboard

This dashboard has the following KPIs for resources and requests:

  • Request Rate

  • Latency Average

  • Latency P99

  • Error Rate

  • CPU %

  • CPU Cores Used

  • CPU Throttle

  • Memory %

  • Memory Bytes

  • Disk Usage

  • Network Usage

Last updated