Prometheus Metrics for On-prem Users

BuildBuddy exposes Prometheus metrics that allow monitoring the four golden signals: latency, traffic, errors, and saturation.

Prometheus metrics are exposed under the path metrics/ on port 9090 by default.

To view these metrics in a live-updating dashboard, we recommend using a tool like Grafana.

SSL metrics

`buildbuddy_ssl_certificate_reload_failures_total` (Counter)

Number of failed TLS certificate reloads.

The server retains the last successfully loaded certificate on failure.

Invocation build event metrics

All invocation metrics are recorded at the end of each invocation.

`buildbuddy_invocation_count` (Counter)

The total number of invocations whose logs were uploaded to BuildBuddy.

Labels

invocation_status: Invocation status: success, failure, disconnected, or unknown.
bazel_exit_code: Exit code of a completed bazel command
bazel_command: Command provided to the Bazel daemon: run, test, build, coverage, mobile-install, ...

Examples

# Number of invocations per second by invocation status
sum by (invocation_status) (rate(buildbuddy_invocation_count[5m]))

# Invocation success rate
sum(rate(buildbuddy_invocation_count{invocation_status="success"}[5m]))
/
sum(rate(buildbuddy_invocation_count[5m]))

`buildbuddy_invocation_discarded_anonymous_build_event_stream_count` (Counter)

The total number of anonymous build event streams discarded because they did not contain an API key.

`buildbuddy_invocation_log_live_chunk_written_bytes` (Counter)

Total number of bytes written to the key-value store for live (in-progress) invocation log tail chunks.

`buildbuddy_invocation_duration_usec` (Histogram)

The total duration of each invocation, in microseconds.

Labels

invocation_status: Invocation status: success, failure, disconnected, or unknown.
bazel_command: Command provided to the Bazel daemon: run, test, build, coverage, mobile-install, ...

Examples

# Median invocation duration in the past 5 minutes
histogram_quantile(
0.5,
sum(rate(buildbuddy_invocation_duration_usec_bucket[5m])) by (le)
)

`buildbuddy_invocation_open_streams` (Gauge)

Number of build event streams currently being handled by the server.

`buildbuddy_invocation_build_event_count` (Counter)

Number of build events uploaded to BuildBuddy.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

Examples

# Build events uploaded per second
sum(rate(buildbuddy_invocation_build_event_count[5m]))

# Approximate error rate of build event upload handler
sum(rate(buildbuddy_invocation_build_event_count{status="0"}[5m]))
/
sum(rate(buildbuddy_invocation_build_event_count[5m]))

`buildbuddy_invocation_stats_recorder_workers` (Gauge)

Number of invocation stats recorder workers currently running.

`buildbuddy_invocation_stats_recorder_duration_usec` (Histogram)

How long it took to finalize an invocation's stats, in microseconds.

This includes the time required to wait for all BuildBuddy apps to flush their local metrics to Redis (if applicable) and then record the metrics to the DB.

`buildbuddy_invocation_webhook_invocation_lookup_workers` (Gauge)

Number of webhook invocation lookup workers currently running.

`buildbuddy_invocation_webhook_invocation_lookup_duration_usec` (Histogram)

How long it took to lookup an invocation before posting to the webhook, in microseconds.

`buildbuddy_invocation_webhook_invocation_payload_size_bytes` (Histogram)

Size in bytes of the payload posted to invocation webhook endpoints.

`buildbuddy_invocation_webhook_notify_workers` (Gauge)

Number of webhook notify workers currently running.

`buildbuddy_invocation_webhook_notify_duration_usec` (Histogram)

How long it took to post an invocation proto to the webhook, in microseconds.

Remote cache metrics

NOTE: Cache metrics are recorded at the end of each invocation, which means that these metrics provide approximate real-time signals.

`buildbuddy_remote_cache_events` (Counter)

Number of cache events handled.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
cache_event_type: Cache event type: hit, miss, or upload.
group_id: Group (organization) ID associated with the request.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_remote_cache_download_size_bytes` (Histogram)

Number of bytes downloaded from the remote cache in each download.

Use the _sum suffix to get the total downloaded bytes and the _count suffix to get the number of downloaded files.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
group_id: Group (organization) ID associated with the request.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

Examples

# Cache download rate (bytes per second)
sum(rate(buildbuddy_cache_download_size_bytes_sum[5m]))

`buildbuddy_remote_cache_download_duration_usec` (Histogram)

Download duration for each file downloaded from the remote cache, in microseconds.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

Examples

# Median download duration for content-addressable store (CAS)
histogram_quantile(
0.5,
sum(rate(buildbuddy_remote_cache_download_duration_usec{cache_type="cas"}[5m])) by (le)
)

`buildbuddy_remote_cache_upload_size_bytes` (Histogram)

Number of bytes uploaded to the remote cache in each upload.

Use the _sum suffix to get the total uploaded bytes and the _count suffix to get the number of uploaded files.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
group_id: Group (organization) ID associated with the request.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

Examples

# Cache upload rate (bytes per second)
sum(rate(buildbuddy_cache_upload_size_bytes_sum[5m]))

`buildbuddy_remote_cache_upload_duration_usec` (Histogram)

Upload duration for each file uploaded to the remote cache, in microseconds.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

Examples

# Median upload duration for content-addressable store (CAS)
histogram_quantile(
0.5,
sum(rate(buildbuddy_remote_cache_upload_duration_usec{cache_type="cas"}[5m])) by (le)
)

`buildbuddy_remote_cache_method_handled_total` (Counter)

Total number of methods completed on the cache, regardless of success or failure.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
cache_method: The Methods exposed in interfaces.Cache.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_method_handling_usec` (Histogram)

Histogram of response latency(microseconds) of cache methods

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
cache_method: The Methods exposed in interfaces.Cache.

`buildbuddy_remote_cache_disk_cache_last_eviction_age_usec` (Gauge)

The age of the item most recently evicted from the cache, in microseconds.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_eviction_age_msec` (Histogram)

Age of items evicted from the cache, in milliseconds.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_num_evictions` (Counter)

Number of items evicted.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_partition_size_bytes_evicted` (Counter)

Number of bytes in the partition evicted.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_partition_size_bytes` (Gauge)

Number of bytes in the partition.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_sampled_partition_group_size_bytes` (Gauge)

Number of bytes seen while sampling the partition for eviction, by group ID.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_cache_disk_cache_partition_capacity_bytes` (Gauge)

Number of bytes in the partition.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_partition_num_items` (Gauge)

Number of items in the partition.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
cache_type: Cache type: action for action cache, cas for content-addressable storage.

`buildbuddy_remote_cache_disk_cache_duplicate_writes` (Counter)

Number of writes for digests that already exist.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_added_file_size_bytes` (Histogram)

Size of artifacts added to the file cache, in bytes.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_filesystem_total_bytes` (Gauge)

Total size of the underlying filesystem.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_disk_cache_filesystem_avail_bytes` (Gauge)

Available bytes in the underlying filesystem.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

Examples

# Total number of duplicate writes.
sum(buildbuddy_remote_cache_duplicate_writes)

`buildbuddy_remote_cache_disk_cache_duplicate_writes_bytes` (Counter)

Number of bytes written that already existed in the cache.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_distributed_cache_peer_lookups` (Histogram)

Number of peers consulted (including the 'local peer') for a distributed cache read before returning a response.

For batch requests, one observation is recorded for each digest in the request.

Labels

op: Distributed cache operation name, such as "FindMissing" or "Get".
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_remote_cache_distributed_cache_find_missing_blob_status_count` (Counter)

Count of blobs checked by FindMissing at the distributed-cache layer, by logical present/absent status and originating purpose.

Labels

purpose: FindMissing purpose: which internal code path originated a FindMissing lookup (see repb.FindMissingBlobsRequest.Purpose), e.g. "ATIME_UPDATE".
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_remote_cache_distributed_cache_backfill_count` (Counter)

Count of distributed cache backfills, by whether the blob was sent to the peer as a reference or as inline bytes, and by status code.

Labels

request_type: How a distributed cache write's payload was sent: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_backfill_size_bytes` (Counter)

Total digest sizes of blobs backfilled to distributed cache peers, by whether the blob was sent as a reference or as inline bytes, and by status code.

Labels

request_type: How a distributed cache write's payload was sent: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_backfill_latency_usec` (Histogram)

Latency of backfilling a single digest.

Labels

group_id: Group (organization) ID associated with the request.
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_remote_cache_distributed_cache_read_count` (Counter)

Count of objects served by distributed cache reads, by operation and source.

Labels

op: Distributed cache operation name, such as "FindMissing" or "Get".
source: Where a distributed cache read was served from: "local" (this node is a replica), "read_through" (local read-through cache), "lookaside", "same_zone_peer" or "cross_zone_peer".

`buildbuddy_remote_cache_distributed_cache_read_digest_size` (Counter)

Total digest sizes of objects served by distributed cache reads, by operation and source.

Labels

op: Distributed cache operation name, such as "FindMissing" or "Get".
source: Where a distributed cache read was served from: "local" (this node is a replica), "read_through" (local read-through cache), "lookaside", "same_zone_peer" or "cross_zone_peer".

`buildbuddy_remote_cache_distributed_cache_read_response_count` (Counter)

Count of distributed cache peer reads, by whether the payload was received as a reference or as inline bytes, and by status code.

Labels

response_type: How a distributed cache read's payload was received: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_read_response_size_bytes` (Counter)

Total digest sizes of blobs read from distributed cache peers, by whether the payload was received as a reference or as inline bytes, and by status code.

Labels

response_type: How a distributed cache read's payload was received: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_get_multi_response_count` (Counter)

Count of values received from distributed cache peers via GetMulti, by whether the value was received as a reference or as inline bytes, and by status code.

Labels

response_type: How a distributed cache read's payload was received: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_get_multi_response_size_bytes` (Counter)

Total digest sizes of values received from distributed cache peers via GetMulti, by whether the value was received as a reference or as inline bytes, and by status code.

Labels

response_type: How a distributed cache read's payload was received: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_write_request_count` (Counter)

Count of distributed cache peer writes, by whether the payload was sent as a reference or as inline bytes, and by status code.

Labels

request_type: How a distributed cache write's payload was sent: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_write_request_size_bytes` (Counter)

Total digest sizes of blobs written to distributed cache peers, by whether the payload was sent as a reference or as inline bytes, and by status code.

Labels

request_type: How a distributed cache write's payload was sent: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_get_with_metadata_response_count` (Counter)

Count of distributed cache peer GetWithMetadata responses, by whether the payload was received as a reference or as inline bytes, and by status code.

Labels

response_type: How a distributed cache read's payload was received: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_distributed_cache_get_with_metadata_response_size_bytes` (Counter)

Total digest sizes of blobs fetched from distributed cache peers via GetWithMetadata, by whether the payload was received as a reference or as inline bytes, and by status code.

Labels

response_type: How a distributed cache read's payload was received: "reference" (a pointer to the blob in shared storage) or "bytes" (the blob's bytes, streamed inline).
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_migration_not_found_error_count` (Counter)

Number of not found errors from the destination cache during a cache migration.

Labels

type: Describes the type of cache request
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_cache_migration_double_read_hit_count` (Counter)

Number of double reads where the source and destination caches hold the same digests during a cache migration.

Labels

type: Describes the type of cache request
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_cache_migration_copy_chan_size` (Gauge)

Number of digests queued to be copied during a cache migration.

`buildbuddy_remote_cache_migration_bytes_copied` (Counter)

Number of bytes copied from the source to destination cache during a cache migration.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_cache_migration_blobs_copied` (Counter)

Number of blobs copied from the source to destination cache during a cache migration.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_cache_tree_cache_lookup_count` (Counter)

Total number of TreeCache lookups.

Labels

status: The TreeCache status: hit/miss/invalid_entry.
level: TreeCache directory depth: 0 for the root dir, 1 for a direct child of the root dir, and so on.

`buildbuddy_remote_cache_tree_cache_split_lookup_count` (Counter)

Total number of TreeCache split lookups.

Labels

status: The TreeCache split lookup status: hit/miss/failure

`buildbuddy_remote_cache_tree_cache_split_write_count` (Counter)

Total number of splits written to TreeCache.

`buildbuddy_remote_cache_tree_cache_set_count` (Counter)

Total number of TreeCache sets.

Labels

status: The TreeCache set status: success/deadline_exceeded/other_error

`buildbuddy_remote_cache_tree_cache_bytes_transferred` (Counter)

Number of bytes written or read from tree cache

Labels

op: TreeCache operation "read" or "write"

`buildbuddy_remote_cache_splice_blob_duration_usec` (Histogram)

Duration of the full SpliceBlob RPC handler, in microseconds.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_remote_cache_splice_blob_count` (Counter)

Number of SpliceBlob requests that reached the manifest store step, labeled by whether chunk contents were verified against the blob digest.

Labels

splice_validation: Whether a SpliceBlob request hashed the referenced chunks to verify the blob digest (full) or skipped that check (skipped).
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_cache_chunked_manifest_validation_count` (Counter)

Number of chunked manifest Store() calls, labeled by whether the shared validation marker was a hit (skipped re-hashing) or miss (full chunk verification performed).

Labels

cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_remote_cache_chunked_manifest_load_count` (Counter)

Number of successful chunked manifest LoadManifest() calls, labeled by manifest key scheme.

Labels

prefix:

`buildbuddy_remote_execution_get_tree_directory_lookup_count` (Counter)

Number of directories fetched by GetTree calls, split by where the directory was found.

Labels

location: Where a directory from a GetTree request was found: one of "uncached", "filecache", "remote"

`buildbuddy_remote_execution_get_tree_filecache_trees_read` (Counter)

Number of trees read from the local filecache.

`buildbuddy_remote_execution_get_tree_filecache_bytes_read` (Counter)

Total size in bytes of trees read from the local filecache.

`buildbuddy_remote_execution_get_tree_filecache_trees_written` (Counter)

Number of trees written to the local filecache.

`buildbuddy_remote_execution_get_tree_filecache_bytes_written` (Counter)

Total size in bytes of trees written to the local filecache.

`buildbuddy_remote_asset_fetch_singleflight_requests_total` (Counter)

Number of per-URI FetchBlob attempts entering process-local singleflight after cache lookup.

Includes callers that later fail or cancel; excludes cache hits and requests that bypass coalescing.

`buildbuddy_remote_asset_fetch_singleflight_executions_total` (Counter)

Number of origin download and CAS publication operations started by FetchBlob singleflight.

Includes failed or canceled operations. Compare with fetch_singleflight_requests_total to measure coalescing.

`buildbuddy_remote_cache_lookaside_cache_lookup_count` (Counter)

Total number of Lookaside Cache lookups.

Labels

status: The Lookaside cache status: hit/miss.

`buildbuddy_remote_cache_lookaside_cache_lookup_bytes` (Counter)

Total number bytes served from the Lookaside Cache by hit/miss status.

Labels

status: The Lookaside cache status: hit/miss.

`buildbuddy_remote_cache_lookaside_cache_eviction_age_msec` (Histogram)

Age of items evicted from the cache, in milliseconds.

Labels

eviction_reason: The reason an item was evicted from the lookaside cache. One of: "expired" or "size"

Remote execution metrics

`buildbuddy_remote_execution_count` (Counter)

Number of actions executed remotely.

This only includes actions which reached the execution phase. If an action fails before execution (for example, if it fails authentication) then this metric is not incremented.

Labels

exit_code: Process exit code of an executed action.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".
isolation: Effective workload isolation type used for an executed task, such as "docker", "podman", "firecracker", or "none".

Examples

# Total number of actions executed per second
sum(rate(buildbuddy_remote_execution_count[5m]))

`buildbuddy_remote_execution_tasks_started_count` (Counter)

Number of tasks started remotely, but not necessarily completed.

Includes retry attempts of the same task.

`buildbuddy_remote_execution_executed_action_metadata_durations_usec` (Histogram)

Time spent in each stage of action execution, in microseconds.

Queries should filter or group by the stage label, taking care not to aggregate different stages.

Labels

stage: Executed action stage. Action execution is split into stages corresponding to the timestamps defined in ExecutedActionMetadata: queued, input_fetch, execution, and output_upload. An additional stage, worker, includes all stages during which a worker is handling the action, which is all stages except the queued stage.
group_id: Group (organization) ID associated with the request.

Examples

# Median duration of all command stages
histogram_quantile(
0.5,
sum(rate(buildbuddy_remote_execution_executed_action_metadata_durations_usec_bucket[5m])) by (le, stage)
)

# p90 duration of just the command execution stage
histogram_quantile(
0.9,
sum(rate(buildbuddy_remote_execution_executed_action_metadata_durations_usec_bucket{stage="execution"}[5m])) by (le)
)

`buildbuddy_remote_execution_task_pressure_stall_duration_fraction` (Histogram)

Linux PSI stall time as a fraction of each action's execution duration (0-1).

Labels

resource: System resource: "cpu", "memory", or "io".
stall_type: Pressure stall type: "some" (task is partially stalled on the resource) or "full" (task is completely stalled on the resource).

`buildbuddy_remote_execution_task_size_read_requests` (Counter)

Number of read requests to the task sizer, which estimates action resource usage based on historical execution stats.

Labels

status: Status of the task size read request: hit, miss, or error.
isolation: Effective workload isolation type used for an executed task, such as "docker", "podman", "firecracker", or "none".
os: OS associated with the request.
arch: CPU architecture associated with the request.
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_task_size_write_requests` (Counter)

Number of write requests to the task sizer, which estimates action resource usage based on historical execution stats.

Labels

status: Status of the task size write request: ok, missing_stats or error.
isolation: Effective workload isolation type used for an executed task, such as "docker", "podman", "firecracker", or "none".
os: OS associated with the request.
arch: CPU architecture associated with the request.
group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_task_size_prediction_duration_usec` (Histogram)

Task size prediction model request duration in microseconds.

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_execution_buildroot_disk_usage_measurement_duration_usec` (Histogram)

Duration of the buildroot (workspace) disk usage measurement performed after a task finishes, in microseconds.

`buildbuddy_remote_execution_resource_usage_timeline_metadata_size_bytes` (Histogram)

Total size of resource usage timeline payloads received from executors.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_input_download_bitmap_metadata_size_bytes` (Histogram)

Total size of input download bitmaps received from executors.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_enqueued_task_milli_cpu` (Histogram)

Milli-CPU prediction of enqueued tasks.

`buildbuddy_remote_execution_enqueued_task_memory_bytes` (Histogram)

Memory prediction of enqueued tasks.

`buildbuddy_remote_execution_oom_killer_targeted_task_memory_bytes` (Histogram)

Observed task memory in bytes targeted by the executor OOM killer.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_waiting_execution_result` (Gauge)

Number of execution requests for which the client is actively waiting for results.

Labels

group_id: Group (organization) ID associated with the request.

Examples

# Total number of execution requests with client waiting for result.
sum(buildbuddy_remote_execution_waiting_execution_result)

`buildbuddy_remote_execution_lost_tasks` (Counter)

Number of tasks that the scheduler reported lost after they stayed missing from Redis while clients were waiting on them.

Examples

# Rate at which the app tells clients to retry executions whose task the scheduler lost.
rate(buildbuddy_remote_execution_lost_tasks[5m])

`buildbuddy_remote_execution_requests` (Counter)

Number of execution requests received.

Labels

group_id: Group (organization) ID associated with the request.
os: OS associated with the request.
arch: CPU architecture associated with the request.

`buildbuddy_remote_execution_executor_registration_count` (Counter)

Number of executor registrations on the scheduler.

Labels

version: Binary version. Example: v2.0.0.

Examples

# Rate of new execution requests by OS/Arch.
sum(rate(buildbuddy_remote_execution_requests[1m])) by (os, arch)

`buildbuddy_remote_execution_merged_actions` (Counter)

Number of identical execution requests that have been merged.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_hedged_actions` (Counter)

Number of identicial execution request which were merged for which a hedged execution was run in the background.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_merged_actions_per_execution` (Histogram)

Distribution of how many actions were submitted and merged against a single, canonical execution over the lifetime of that canonical execution.

Note that this metric is recorded once per merged-action, so distribution values are cumulative, or recorded n-times per canonical execution.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_remote_execution_merged_action_submit_time_offset_usec` (Histogram)

The offset, in microseconds of wall-time, between the time when a merged action was submitted to the execution server and when the original action was submitted to the execution server.

Labels

group_id: Group (organization) ID associated with the request.

Examples

# Rate of merged actions by group.
sum(rate(buildbuddy_remote_execution_merged_actions[1m])) by (group_id)

`buildbuddy_remote_execution_queue_length` (Gauge)

Number of actions currently waiting in the executor queue.

Labels

group_id: Group (organization) ID associated with the request.

Examples

# Median queue length across all executors
quantile(0.5, buildbuddy_remote_execution_queue_length)

`buildbuddy_remote_execution_tasks_executing` (Gauge)

Number of tasks currently being executed by the executor.

Labels

stage: Executed action stage. Action execution is split into stages corresponding to the timestamps defined in ExecutedActionMetadata: queued, input_fetch, execution, and output_upload. An additional stage, worker, includes all stages during which a worker is handling the action, which is all stages except the queued stage.

Examples

# Fraction of idle executors
count_values(0, buildbuddy_remote_execution_tasks_executing)
/
count(buildbuddy_remote_execution_tasks_executing)

`buildbuddy_remote_execution_assigned_ram_bytes` (Gauge)

Estimated RAM on the executor that is currently allocated for task execution, in bytes.

`buildbuddy_remote_execution_assigned_and_queued_estimated_ram_bytes` (Gauge)

Estimated RAM on the executor that is currently allocated for queued or executing tasks, in bytes.

Note that this is a fuzzy estimate because there's no guarantee that tasks queued on a machine will be handled by that machine.

`buildbuddy_remote_execution_assignable_ram_bytes` (Gauge)

Maximum total RAM that can be allocated for task execution, in bytes.

`buildbuddy_remote_execution_assignable_disk_bytes` (Gauge)

Maximum total workspace disk that can be allocated for task execution, in bytes.

`buildbuddy_remote_execution_assigned_milli_cpu` (Gauge)

Estimated CPU time on the executor that is currently allocated for task execution, in milliCPU (CPU-milliseconds per second).

`buildbuddy_remote_execution_assigned_and_queued_estimated_milli_cpu` (Gauge)

Estimated CPU time on the executor that is currently allocated for queued or executing tasks, in milliCPU (CPU-milliseconds per second).

Note that this is a fuzzy estimate because there's no guarantee that tasks queued on a machine will be handled by that machine.

`buildbuddy_remote_execution_assignable_milli_cpu` (Gauge)

Maximum total CPU time on the executor that can be allocated for task execution, in milliCPU (CPU-milliseconds per second).

`buildbuddy_remote_execution_assigned_custom_resources` (Gauge)

Custom resources on the executor currently allocated for task execution.

Custom resources are dimensionless values configured via executor.custom_resources.

Labels

resource_name: Name of a custom resource configured on an executor.

`buildbuddy_remote_execution_assignable_custom_resources` (Gauge)

Maximum custom resources that can be allocated for task execution.

Custom resources are dimensionless values configured via executor.custom_resources.

Labels

resource_name: Name of a custom resource configured on an executor.

`buildbuddy_remote_execution_file_download_count` (Histogram)

Number of files downloaded during remote execution.

`buildbuddy_remote_execution_file_download_size_bytes` (Histogram)

Total number of bytes downloaded during remote execution.

`buildbuddy_remote_execution_file_download_duration_usec` (Histogram)

Time spent downloading files during remote execution, in microseconds.

`buildbuddy_remote_execution_file_upload_count` (Histogram)

Number of files uploaded during remote execution.

`buildbuddy_remote_execution_file_upload_size_bytes` (Histogram)

Total number of bytes uploaded during remote execution.

`buildbuddy_remote_execution_skipped_output_bytes` (Counter)

Total number of output bytes that weren't uploaded after remote execution.

`buildbuddy_remote_execution_file_upload_duration_usec` (Histogram)

Time spent uploading files during remote execution, in microseconds.

`buildbuddy_remote_execution_networking_command_duration_usec` (Histogram)

Duration of networking commands, in microseconds.

Labels

command: Command being run. Specific arguments to the command are omitted to reduce metric cardinality.

`buildbuddy_remote_execution_networking_command_cpu_usage_usec` (Histogram)

CPU usage of networking commands, in CPU-microseconds.

Labels

command: Command being run. Specific arguments to the command are omitted to reduce metric cardinality.

`buildbuddy_remote_execution_vfs_cas_files_count` (Counter)

Total number of CAS files in VFS filesystems.

`buildbuddy_remote_execution_vfs_cas_files_accessed_count` (Counter)

Number of CAS files in VFS filesystems that were accessed by the action.

`buildbuddy_remote_execution_vfs_cas_files_size_bytes` (Counter)

Total size of CAS files in VFS filesystems.

`buildbuddy_remote_execution_vfs_cas_files_accessed_bytes` (Counter)

Size of CAS files in VFS filesystems that were accessed by the action.

`buildbuddy_remote_runner_requests` (Counter)

Number of remote runner executions initiated.

Labels

group_id: Group (organization) ID associated with the request.
op:
stage: Generic label to describe the stage the metric is capturing
os: OS associated with the request.
arch: CPU architecture associated with the request.
self_hosted: Whether the request is targeting self-hosted executors.

`buildbuddy_firecracker_stage_duration_usec` (Histogram)

The total duration of each firecracker stage, in microseconds.

Labels

stage: Generic label to describe the stage the metric is capturing

Stage label values

"init": Time for the VM to start up (either a new VM or from a snapshot)
"exec": Time to run the command inside the container
"task_lifecycle": Time from when the task if first assigned to the VM (beginning of init) to after it's finished execution. This roughly represents what a customer will wait for the task to complete after it's been scheduled to a firecracker runner
"pause": Time to pause the VM, save a snapshot, and cleanup resources

Examples

# P95 workflow lifecycle duration in the past 5 minutes, grouped by group_id
histogram_quantile(
0.95,
sum by(le, group_id) (
rate(buildbuddy_firecracker_stage_duration_usec_bucket{job="executor-workflows", stage="task_lifecycle"}[5m])
)
)

`buildbuddy_firecracker_snapshot_save_workloads_executing` (Gauge)

Number of snapshot save workloads currently being executed by the executor.

Labels

stage: Generic label to describe the stage the metric is capturing

`buildbuddy_firecracker_snapshot_source_count` (Counter)

The number of snapshot manifests fetched from the local vs remote cache.

Labels

chunk_source: For chunked snapshot files, describes the initialization source of the chunk (Ex. remote_cache or local_filecache)

`buildbuddy_firecracker_exec_dial_duration_usec` (Histogram)

Time taken to dial the VM guest execution server after it has been started or resumed, in microseconds.

Labels

created_from_snapshot: CreatedFromSnapshot indicates if a firecracker execution used a snapshot.

`buildbuddy_firecracker_execution_error_count` (Counter)

Count of Firecracker task execution errors, labeled by reason.

Labels

reason: Reason for a Firecracker task execution failure.

`buildbuddy_firecracker_snapshot_remote_cache_upload_size_bytes` (Counter)

After a copy-on-write snapshot has been used, the total count of compressed bytes written to the cache (i.e.

will be 0 if the artifact is already cached).

Labels

file_name: Name of a file.

`buildbuddy_firecracker_cow_snapshot_dirty_chunk_ratio` (Histogram)

After a copy-on-write snapshot has been used, the ratio of dirty/total chunks.

Labels

file_name: Name of a file.

Examples

# To view how many elements fall into each bucket
# Visualize with the Bar Gauge type
# Legend: {{le}}
# Format: Heatmap
sum(increase(buildbuddy_firecracker_cow_snapshot_dirty_chunk_ratio_bucket[5m])) by(le)

`buildbuddy_firecracker_cow_snapshot_dirty_bytes` (Counter)

After a copy-on-write snapshot has been used, the total count of bytes dirtied.

Labels

file_name: Name of a file.

`buildbuddy_firecracker_cow_snapshot_bytes_read` (Counter)

After a copy-on-write snapshot has been used, the number of bytes read from each source.

Labels

file_name: Name of a file.
chunk_source: For chunked snapshot files, describes the initialization source of the chunk (Ex. remote_cache or local_filecache)

`buildbuddy_firecracker_cow_snapshot_chunk_source_ratio` (Histogram)

After a copy-on-write snapshot has been used, the percentage of chunks that were initialized by the given source.

Labels

file_name: Name of a file.
chunk_source: For chunked snapshot files, describes the initialization source of the chunk (Ex. remote_cache or local_filecache)

`buildbuddy_firecracker_cow_snapshot_memory_mapped_bytes` (Gauge)

Total number of bytes currently memory-mapped.

Labels

file_name: Name of a file.

`buildbuddy_firecracker_cow_snapshot_page_fault_total_duration_usec` (Histogram)

For a snapshotted VM, total time spent fulfilling page faults.

Labels

stage: Generic label to describe the stage the metric is capturing

`buildbuddy_firecracker_cow_snapshot_chunk_operation_duration_usec` (Histogram)

For a COW snapshot, cumulative time spent on an operation type.

Labels

file_name: Name of a file.
name: The name used to identify the type of an unexpected event.
stage: Generic label to describe the stage the metric is capturing

`buildbuddy_firecracker_cow_bytes_read` (Counter)

Total number of bytes read from COW chunked files.

Labels

file_name: Name of a file.

`buildbuddy_firecracker_cow_bytes_written` (Counter)

Total number of bytes written to COW chunked files.

Labels

file_name: Name of a file.

`buildbuddy_firecracker_workspace_conversion_disk_write_ops` (Counter)

Total number of disk write operations performed converting Firecracker action workspaces to/from ext4 images.

Labels

command: Command being run. Specific arguments to the command are omitted to reduce metric cardinality.

`buildbuddy_remote_execution_max_recyclable_resource_usage_event` (Counter)

Counter for firecracker runners that reach max disk/memory usage and won't get recycled.

Labels

group_id: Group (organization) ID associated with the request.
name: The name used to identify the type of an unexpected event.
recycled_runner_status: For remote execution runners, describes the recycling status (Ex. 'clean' if the runner is not recycled or 'recycled')

`buildbuddy_remote_execution_recycle_runner_requests` (Counter)

Number of execution requests with runner recycling enabled (via the platform property recycle-runner=true).

Labels

status: Status of the recycle runner request: hit if the executor assigned a recycled runner to the action; miss otherwise.

`buildbuddy_remote_execution_runner_pool_count` (Gauge)

Number of command runners that are currently pooled (and available for recycling).

`buildbuddy_remote_execution_runner_pool_evictions` (Counter)

Number of command runners removed from the pool to make room for other runners.

`buildbuddy_remote_execution_runner_pool_failed_recycle_attempts` (Counter)

Number of failed attempts to add runners to the pool.

Labels

reason: Reason for a runner not being added to the runner pool.

`buildbuddy_remote_execution_runner_pool_memory_usage_bytes` (Gauge)

Total memory usage of pooled command runners, in bytes.

Currently only supported for Docker-based executors.

`buildbuddy_remote_execution_runner_pool_disk_usage_bytes` (Gauge)

Total disk usage of pooled command runners, in bytes.

`buildbuddy_remote_execution_file_cache_requests` (Counter)

Number of local executor file cache requests.

Labels

status: Status of the file cache request: hit if found in cache, miss otherwise.

`buildbuddy_remote_execution_file_cache_op_latency_usec` (Histogram)

Latency of individual file cache operations.

Labels

`buildbuddy_remote_execution_file_cache_last_eviction_age_usec` (Gauge)

Age of the last entry evicted from the executor's local file cache (relative to when it was added to the cache), in microseconds.

`buildbuddy_remote_execution_file_cache_added_file_size_bytes` (Histogram)

Size of artifacts added to the file cache, in bytes.

`buildbuddy_remote_execution_file_cache_added_file_bytes_count` (Counter)

Total number of bytes written to the filecache by groupid.

Labels

group_id: Group (organization) ID associated with the request.

Blobstore metrics

"Blobstore" refers to the backing storage that BuildBuddy uses to store objects in the cache, as well as certain pieces of temporary data (such as invocation events while an invocation is in progress).

`buildbuddy_blobstore_read_count` (Counter)

Number of files read from the blobstore.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
blobstore_type: gcs (Google Cloud Storage), aws_s3, or disk.

`buildbuddy_blobstore_read_size_bytes` (Histogram)

Number of bytes read from the blobstore per file.

Labels

blobstore_type: gcs (Google Cloud Storage), aws_s3, or disk.

Bytes downloaded per second

sum(rate(buildbuddy_blobstore_read_size_bytes[5m]))

### **`buildbuddy_blobstore_read_duration_usec`** (Histogram)

Duration per blobstore file read, in **microseconds**.

#### Labels

- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_blobstore_write_count`** (Counter)

Number of files written to the blobstore.

#### Labels

- **status**: Status code as defined by [grpc/codes](https://godoc.org/google.golang.org/grpc/codes#Code). This is a numeric value; any non-zero code indicates an error.
- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

# Bytes uploaded per second
sum(rate(buildbuddy_blobstore_write_size_bytes[5m]))

`buildbuddy_blobstore_write_size_bytes` (Histogram)

Number of bytes written to the blobstore per file.

Labels

blobstore_type: gcs (Google Cloud Storage), aws_s3, or disk.

`buildbuddy_blobstore_write_duration_usec` (Histogram)

Duration per blobstore file write, in microseconds.

Labels

blobstore_type: gcs (Google Cloud Storage), aws_s3, or disk.

`buildbuddy_blobstore_clone_count` (Counter)

Number of files cloned (copied storage-side, without the bytes leaving the blobstore).

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
blobstore_type: gcs (Google Cloud Storage), aws_s3, or disk.

Bytes cloned per second (these bytes never leave the blobstore)

sum(rate(buildbuddy_blobstore_clone_size_bytes[5m]))

### **`buildbuddy_blobstore_clone_size_bytes`** (Histogram)

Number of bytes cloned per file.

#### Labels

- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_blobstore_clone_duration_usec`** (Histogram)

Duration per blobstore file clone, in **microseconds**.

#### Labels

- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_blobstore_delete_count`** (Counter)

Number of files deleted from the blobstore.

#### Labels

- **status**: Status code as defined by [grpc/codes](https://godoc.org/google.golang.org/grpc/codes#Code). This is a numeric value; any non-zero code indicates an error.
- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_blobstore_delete_duration_usec`** (Histogram)

Delete duration per blobstore file deletion, in **microseconds**.

#### Labels

- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_blobstore_exists_count`** (Counter)

Number of existance checks in the blobstore.

#### Labels

- **status**: Status code as defined by [grpc/codes](https://godoc.org/google.golang.org/grpc/codes#Code). This is a numeric value; any non-zero code indicates an error.
- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_blobstore_exists_duration_usec`** (Histogram)

CheckExists duration, in **microseconds**.

#### Labels

- **blobstore_type**: `gcs` (Google Cloud Storage), `aws_s3`, or `disk`.

### **`buildbuddy_invocation_log_bytes_written`** (Counter)

Number of invocation log bytes uploaded, either via the build event stream or the WriteEventLog API (stdout+stderr).

#### Labels

- **name**: The name used to identify the type of an unexpected event.
- **group_id**: Group (organization) ID associated with the request.

# SQL metrics

The following metrics are for monitoring the SQL database configured
for BuildBuddy.

If you'd like to see an up-to-date catalog of what BuildBuddy stores in
its SQL database, see the table definitions [here](https://github.com/buildbuddy-io/buildbuddy/blob/master/server/tables/tables.go).

## Query / error rate metrics

### **`buildbuddy_sql_query_count`** (Counter)

Number of SQL queries executed.

#### Labels

- **sql_query_template**: SQL query before substituting template parameters.

#### Examples

```promql
# SQL queries per second (by query template).
sum by (sql_query_template) (rate(buildbuddy_sql_query_count[5m]))

`buildbuddy_sql_query_duration_usec` (Histogram)

SQL query duration, in microseconds.

Labels

sql_query_template: SQL query before substituting template parameters.

Examples

# Median SQL query duration
histogram_quantile(
0.5,
sum(rate(buildbuddy_sql_query_duration_usec_bucket[5m])) by (le)
)

`buildbuddy_sql_error_count` (Counter)

Number of SQL queries that resulted in an error.

Labels

sql_query_template: SQL query before substituting template parameters.

Examples

# SQL error rate
sum(rate(buildbuddy_sql_error_count[5m]))
/
sum(rate(buildbuddy_sql_query_count[5m]))

`database/sql` metrics

The following metrics directly expose DBStats from the database/sql Go package.

`buildbuddy_sql_max_open_connections` (Gauge)

Maximum number of open connections to the database.

Labels

sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

`buildbuddy_sql_open_connections` (Gauge)

The number of established connections to the database.

Labels

connection_status: Status of the database connection: in_use or idle
sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

`buildbuddy_sql_wait_count` (Counter)

The total number of connections waited for.

Labels

sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

`buildbuddy_sql_wait_duration_usec` (Counter)

The total time blocked waiting for a new connection, in microseconds.

Labels

sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

`buildbuddy_sql_max_idle_closed` (Counter)

The total number of connections closed due to SetMaxIdleConns.

Labels

sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

`buildbuddy_sql_max_idle_time_closed` (Counter)

The total number of connections closed due to SetConnMaxIdleTime.

Labels

sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

`buildbuddy_sql_max_lifetime_closed` (Counter)

The total number of connections closed due to SetConnMaxLifetime.

Labels

sql_db_role: SQL DB replica role: primary for read+write replicas, or read_replica for read-only DB replicas.

HTTP metrics

`buildbuddy_http_request_count` (Counter)

HTTP request count.

Labels

route: HTTP route before substituting path parameters (/invocation/:id, /settings, ...)
method: HTTP method: GET, POST, ...

Examples

# Requests per second, by status code
sum by (code) (rate(buildbuddy_http_request_count[5m]))

# 5xx error ratio
sum(rate(buildbuddy_http_request_count{code=~"5.."}[5m]))
/
sum(rate(buildbuddy_http_request_count[5m]))

`buildbuddy_http_request_handler_duration_usec` (Histogram)

Time taken to handle each HTTP request in microseconds.

Labels

route: HTTP route before substituting path parameters (/invocation/:id, /settings, ...)
method: HTTP method: GET, POST, ...
code: HTTP response code: 200, 302, 401, 404, 500, ...

Examples

# Median request duration for successfully processed (2xx) requests.
# Other status codes may be associated with early-exits and are
# likely to add too much noise.
histogram_quantile(
0.5,
sum by (le)	(rate(buildbuddy_http_request_handler_duration_usec{code=~"2.."}[5m]))
)

`buildbuddy_http_response_size_bytes` (Histogram)

Response size of each HTTP response in bytes.

Labels

route: HTTP route before substituting path parameters (/invocation/:id, /settings, ...)
method: HTTP method: GET, POST, ...
code: HTTP response code: 200, 302, 401, 404, 500, ...

Examples

# Median HTTP response size
histogram_quantile(
0.5,
sum by (le)	(rate(buildbuddy_http_response_size_bytes[5m]))
)

`buildbuddy_http_client_request_count` (Counter)

HTTP outgoing request count.

Labels

client_name:
host: Host (without any port number) for outgoing HTTP requests
method: HTTP method: GET, POST, ...

`buildbuddy_http_client_response_size_bytes` (Histogram)

Response size of response for each HTTP client request in bytes.

Labels

client_name:
host: Host (without any port number) for outgoing HTTP requests
method: HTTP method: GET, POST, ...
code: HTTP response code: 200, 302, 401, 404, 500, ...

Redis client metrics

`buildbuddy_redis_client_dial_errors` (Counter)

The number of errors when dialing redis.

Internal metrics

These metrics are for monitoring lower-level subsystems of BuildBuddy.

Build event handler

The build event handler logs all build events uploaded to BuildBuddy as part of the Build Event Protocol.

`buildbuddy_build_event_handler_duration_usec` (Histogram)

The time spent handling each build event in microseconds.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

In-memory LRU caches

These metrics are reported by in-memory LRU caches (see server/util/lru) that are configured with a name.

`buildbuddy_lru_lookup_count` (Counter)

Number of LRU cache lookups (Get or Contains calls), by operation and hit/miss status.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
op: LRU cache operation: get or contains.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_lru_last_eviction_age_usec` (Gauge)

Age of the entry most recently automatically evicted from an LRU cache (time since the entry was added or last updated), in microseconds.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
eviction_reason: The reason an entry was automatically evicted from an LRU cache: size (evicted because LRU reached max size) or ttl (expired).

Usage tracker

`buildbuddy_usage_tracker_missing_collection_counts_count` (Counter)

The number of times the usage tracking system was missing counts for a particular collection period.

This may happen if there are transient redis errors.

Webhooks

Webhooks are HTTP endpoints exposed by BuildBuddy server which allow it to respond to repository events. These URLs are created as part of BuildBuddy workflows.

`buildbuddy_webhook_handler_workflows_started` (Counter)

The number of workflows triggered by the webhook handler.

Labels

event: Type of event sent to BuildBuddy's webhook handler: push or pull_request.

Cache

"Cache" refers to the cache backend(s) that BuildBuddy uses to accelerate file IO operations, which are common in different subsystems such as the remote cache and the fetch server (for downloading invocation artifacts).

BuildBuddy can be configured to use multiple layers of caching (an in-memory layer, coupled with a cloud storage layer).

`get` metrics

get metrics track non-streamed cache reads (all data is fetched from the cache in a single request).

`buildbuddy_cache_get_count` (Counter)

Number of cache get requests.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_get_duration_usec` (Histogram)

The time spent retrieving each entry from the cache, in microseconds.

This is recorded only for successful gets.

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_get_size_bytes` (Histogram)

Size of each entry retrieved from the cache, in bytes.

This is recorded only for successful gets.

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`read` metrics

read metrics track streamed cache reads.

`buildbuddy_cache_read_count` (Counter)

Number of streamed cache reads started.

This is incremented once for each started stream, not for each chunk in the stream.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_read_duration_usec` (Histogram)

The total time spent for each read stream, in microseconds.

This is recorded only for successful reads, and measures the entire read stream (not just individual chunks).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_read_size_bytes` (Histogram)

Total size of each entry retrieved from the cache via streaming, in bytes.

This is recorded only on success, and measures the entire stream (not just individual chunks).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`set` metrics

set metrics track non-streamed cache writes (all data is wrtiten in a single request).

`buildbuddy_cache_set_count` (Counter)

Number of cache set requests.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_set_duration_usec` (Histogram)

The time spent writing each entry to the cache, in microseconds.

This is recorded only for successful sets.

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_set_size_bytes` (Histogram)

Size of the value stored in each set operation, in bytes.

This is recorded only for successful sets.

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_set_retries` (Histogram)

Number of retries required to fulfill the set request (an observed value of 0 means the transfer succeeded on the first try).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`write` metrics

write metrics track streamed cache writes.

`buildbuddy_cache_write_count` (Counter)

Number of streamed cache writes started.

This is incremented once for each started stream, not for each chunk in the stream.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_write_duration_usec` (Histogram)

The time spent for each streamed write to the cache, in microseconds.

This is recorded only on success, and measures the entire stream (not just individual chunks).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_write_size_bytes` (Histogram)

Size of each entry written to the cache via streaming, in bytes.

This is recorded only on success, and measures the entire stream (not just individual chunks).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_write_retries` (Histogram)

Number of retries required to write each chunk in the stream (an observed value of 0 means the transfer succeeded on the first try).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

Other cache metrics

`buildbuddy_cache_delete_count` (Counter)

Number of deletes from the cache.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_delete_duration_usec` (Histogram)

Duration of each cache deletion, in microseconds.

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_contains_count` (Counter)

Number of contains(key) requests made to the cache.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_contains_duration_usec` (Histogram)

Duration of each each contains(key) request, in microseconds.

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_contains_retry_count` (Histogram)

Number of retries required to fulfill each contains(key) request to the cache (an observed value of 0 means the request succeeded on the first try).

Labels

tier: Cache tier: memory or cloud. This label can be used to write Prometheus queries that don't break if the cache backend is swapped out for a different backend.
backend: Cache backend: gcs (Google Cloud Storage), aws_s3, or redis.

`buildbuddy_cache_batch_operator_enqueued_digests` (Counter)

The number of digests enqueued in a batch operator, with the outcome of the enqueue operation.

Labels

operator_name:
group_id: Group (organization) ID associated with the request.
status: There are a couple of places where we enqueue RPCs to be batched and sent asynchronously: the atime_updater and the hit_tracker_client. This label tracks the outcome of these enqueue operations. One of "enqueued", "duplicate", "dropped_too_many_updates", or "dropped_too_many_batches"

`buildbuddy_cache_batch_operator_flushed_digests` (Counter)

The number of digests flushed from a batch operator.

Labels

operator_name:
group_id: Group (organization) ID associated with the request.
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_cache_proxy_secondary_sync_write_digests` (Counter)

The number of digests written synchronously to a secondary cache.

Labels

server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

Misc metrics

`buildbuddy_version` (Gauge)

Binary version of the running instance.

Always reports a value of 1 similar to the up metric, but has a label containing the version.

Labels

version: Binary version. Example: v2.0.0.
commit: Binary git commit SHA. Example: 4bd7046417608d785094aa5ec7aa009a9ae53753

`buildbuddy_unexpected_event` (Counter)

Counter for unexpected events.

Labels

name: The name used to identify the type of an unexpected event.

`buildbuddy_health_check_status` (Gauge)

Health check status.

Labels

health_check_name: Name of service the health check is running for (Ex "distributed_cache" or "sql_primary").

`buildbuddy_quota_quota_exceeded_count` (Counter)

Total number of calls banned by quota server

Labels

quota_namespace: The namespace that is protected by quota manager.
quota_key: The key used for quota accounting, either a group ID or an IP address.

`buildbuddy_quota_quota_key_empty_count` (Counter)

Total number of calls with empty quota key

Labels

quota_namespace: The namespace that is protected by quota manager.

`buildbuddy_clickhouse_insert_count` (Counter)

Num of rows inserted into ClickHouse

Labels

clickhouse_table_name: The name of the table in Clickhouse
status: Status of the Clickhouse operation: ok, error.

`buildbuddy_clickhouse_query_duration_usec` (Histogram)

ClickHouse query duration, in microseconds.

Labels

sql_query_template: SQL query before substituting template parameters.

`buildbuddy_clickhouse_error_count` (Counter)

Number of ClickHouse SQL queries that resulted in an error.

Labels

sql_query_template: SQL query before substituting template parameters.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_clickhouse_query_count` (Counter)

Number of ClickHouse SQL queries executed.

Labels

sql_query_template: SQL query before substituting template parameters.

`buildbuddy_compressor_bytes_compressed` (Counter)

The number of decompressed bytes passed into compressors

Labels

compression: Describes the type of compression

`buildbuddy_compressor_bytes_decompressed` (Counter)

The number of decompressed bytes passed out of decompressors

Labels

compression: Describes the type of compression

`buildbuddy_pebble_compression_ratio` (Histogram)

The aggregate compression ratio (compressed / decompressed bytes) for a stream of data

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
compression: Describes the type of compression

`buildbuddy_compressor_compressed_blob_size_write` (Counter)

The number of compressed bytes in all blobs

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
compression: Describes the type of compression

`buildbuddy_compressor_decompressed_blob_size_write` (Counter)

The number of decompressed bytes in all blobs

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
compression: Describes the type of compression

Examples

# Histogram buckets with the count of elements in each compression ratio bucket
# Visualize with the Bar Gauge type
# Legend: {{le}}
# Format: Heatmap
sum(increase(buildbuddy_pebble_compression_ratio_bucket[5m])) by(le)

# Percentage of elements that increased in size when compressed (compression ratio > 1)
# Visualize with the Stat type
(sum(buildbuddy_pebble_compression_ratio_count) - sum(buildbuddy_pebble_compression_ratio_bucket{le="1.0"})) / sum(buildbuddy_pebble_compression_ratio_count)

`buildbuddy_server_upload_size_bytes` (Histogram)

Number of bytes uploaded to the server in each upload.

Use the _sum suffix to get the total uploaded bytes and the _count suffix to get the number of uploaded files.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_server_uncompressed_upload_bytes_count` (Counter)

Number of uncompressed bytes uploaded to the server.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
group_id: Group (organization) ID associated with the request.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_server_download_size_bytes` (Histogram)

Number of bytes downloaded from the server in each download.

Use the _sum suffix to get the total downloaded bytes and the _count suffix to get the number of downloaded files.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
group_id: Group (organization) ID associated with the request.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_server_uncompressed_download_bytes_count` (Counter)

Number of uncompressed bytes downloaded from the server.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
group_id: Group (organization) ID associated with the request.
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_server_digest_upload_size_bytes` (Histogram)

Digest size uploaded to the server in each upload.

This does not always match the actual size uploaded to the server, if the client sends compressed bytes. Use the _sum suffix to get the total uploaded bytes and the _count suffix to get the number of uploaded files.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_server_digest_download_size_bytes` (Histogram)

Digest size downloaded from the server in each download.

This does not always match the actual size downloaded to the server, if the client requests compressed bytes. Use the _sum suffix to get the total downloaded bytes and the _count suffix to get the number of downloaded files.

Labels

cache_type: Cache type: action for action cache, cas for content-addressable storage.
server_name: Describes the name of the server that handles a client request, such as "byte_stream_server" or "cas_server"
tracked: Whether or not billable usage was recorded for this request ("true", "false")

`buildbuddy_logger_log_count` (Counter)

The number of logs

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

Raft cache metrics

`buildbuddy_raft_partition_operations` (Counter)

Number of operations on partitions

Labels

partition_id: The ID of the disk cache partition this event applied to.
op: The type of operation on a partition: "initialize", "soft-delte", "hard-delete".
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_raft_ranges` (Gauge)

Number of raft ranges on each nodehost.

Labels

node_host_id: The ID of a partition.

`buildbuddy_raft_range_replica` (Gauge)

Indicator (always 1) that the labeled nodehost holds a replica of the given range.

Use to verify replica placement across zones and partitions.

Labels

range_id: The range ID of a raft region.
node_host_id: The ID of a partition.
partition_id: The ID of the disk cache partition this event applied to.
zone: The availability zone of a node.

`buildbuddy_raft_leases` (Gauge)

Number of raft leases on each nodehost.

Labels

range_id: The range ID of a raft region.
node_host_id: The ID of a partition.
partition_id: The ID of the disk cache partition this event applied to.
zone: The availability zone of a node.

`buildbuddy_raft_leaders` (Gauge)

Number of raft leaders on each nodehost.

Labels

range_id: The range ID of a raft region.
node_host_id: The ID of a partition.
partition_id: The ID of the disk cache partition this event applied to.
zone: The availability zone of a node.

`buildbuddy_raft_bytes` (Gauge)

Size (in bytes) of each range.

Labels

range_id: The range ID of a raft region.
node_host_id: The ID of a partition.
partition_id: The ID of the disk cache partition this event applied to.
zone: The availability zone of a node.

`buildbuddy_raft_proposals` (Counter)

The total number of statemachine proposals on each range.

Labels

range_id: The range ID of a raft region.
node_host_id: The ID of a partition.
partition_id: The ID of the disk cache partition this event applied to.
zone: The availability zone of a node.

`buildbuddy_raft_reads` (Counter)

The total number of read requests served for each range.

Labels

range_id: The range ID of a raft region.
node_host_id: The ID of a partition.
partition_id: The ID of the disk cache partition this event applied to.
zone: The availability zone of a node.

`buildbuddy_raft_splits` (Counter)

The total number of splits per nodehost.

Labels

node_host_id: The ID of a partition.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_raft_moves` (Counter)

The total number of moves per nodehost.

Labels

node_host_id: The ID of a partition.
move_type: The type of raft move add, or remove.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_raft_rangecache_lookups` (Counter)

The total number of rangecache lookups per nodehost.

Labels

rangecache_event_type: Raft RangeCache event type: hit, miss, or update.

`buildbuddy_raft_split_duration_usec` (Histogram)

The time spent splitting a range in microseconds.

Labels

range_id: The range ID of a raft region.

`buildbuddy_raft_replica_update_duration_usec` (Histogram)

The time spent on replica.Update in microseconds.

Labels

range_id: The range ID of a raft region.

`buildbuddy_raft_eviction_errors` (Counter)

The total number of eviction errors

`buildbuddy_raft_listener_events_dropped` (Counter)

The total number of dropped listener events

Labels

listener_id: The ID of a raft listener
listener_event: Raft Listener Event Type, such as "LeaderUpdated". These are events sent by dragonboat library.

`buildbuddy_raft_store_events` (Gauge)

Number of events in the queue

`buildbuddy_raft_store_event_broadcast_dropped` (Counter)

The total number of dropped events to broadcast

Labels

event_broadcast_source: The source of the event broadcast.
raft_event: Raft Event Type, such as "range-removed", "range-usage-updated"

`buildbuddy_raft_store_event_listener_dropped` (Counter)

The total number of dropped events in store

Labels

listener_id: The ID of a raft listener
raft_event: Raft Event Type, such as "range-removed", "range-usage-updated"

`buildbuddy_raft_lease_action_count` (Counter)

The total number of lease actions

Labels

range_id: The range ID of a raft region.
lease_action: The type of lease action Acquire, Drop.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_raft_lease_action_duration_msec` (Histogram)

The duration of a lease action

Labels

lease_action: The type of lease action Acquire, Drop.

`buildbuddy_raft_zombie_cleanup_tasks` (Gauge)

The total number of pending zombie cleanup tasks

`buildbuddy_raft_zombie_cleanup` (Counter)

The total number of zombie cleanups

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_raft_range_lock_duration_msec` (Histogram)

The duration of the range lock in a client session

Labels

range_id: The range ID of a raft region.

`buildbuddy_raft_sender_range_lock_duration_msec` (Histogram)

Time spent waiting for and holding the sender-side per-range propose lock, which serializes proposes across the full retry lifecycle of a single logical request to keep retries idempotent.

Labels

range_id: The range ID of a raft region.

`buildbuddy_raft_nodehost_method_errors` (Counter)

The total number of nodehost method

Labels

nodehost_method: The type of nodehost method: "SyncPropose", "SyncRead"
dragonboat_error: The error returned by dragonboat library

`buildbuddy_raft_nodehost_method_usec` (Histogram)

The duration of a nodehost method

Labels

nodehost_method: The type of nodehost method: "SyncPropose", "SyncRead"
range_id: The range ID of a raft region.

`buildbuddy_raft_batch_atime_update_usec` (Histogram)

The duration of the batch request to update atime

`buildbuddy_raft_batch_delete_usec` (Histogram)

The duration of the batch request to delete

`buildbuddy_raft_driver_actions` (Counter)

The total number of driver action

Labels

driver_action: Raft Driver Action
driver_requeue_type: Raft Driver Requeue Type
range_id: The range ID of a raft region.

`buildbuddy_raft_intermediate_replicas_count` (Gauge)

The number of intermediate replicas

Labels

replica_state: Raft Replica State: staging, removing
range_id: The range ID of a raft region.

`buildbuddy_raft_txn_record_process_count` (Counter)

The total number of driver action

Labels

txn_record_process_status: The status of processing txn record: "success" or "failure"

`buildbuddy_raft_eviction_samples_chan_size` (Gauge)

Num of items in eviction samples chan

Labels

partition_id: The ID of the disk cache partition this event applied to.

`buildbuddy_raft_eviction_gcs_chan_size` (Gauge)

Num of items in gcs eviction chan

Labels

partition_id: The ID of the disk cache partition this event applied to.

`buildbuddy_raft_gcs_delete_dropped` (Counter)

The total number of dropped gcs deletes

Labels

partition_id: The ID of the disk cache partition this event applied to.

`buildbuddy_raft_atime_update_gcs_count` (Counter)

Count of atime updates to GCS.

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_raft_gcs_eviction_count` (Counter)

Count of evictions from GCS.

Labels

partition_id: The ID of the disk cache partition this event applied to.
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_auth_api_key_lookup_count` (Counter)

Total number of API key lookups.

Labels

status: Whether or not the API Key lookup hit the in memory cache or not: "cache_hit", "cache_miss" or "invalid_key".

`buildbuddy_auth_ip_rules_check_latency_usec` (Histogram)

Latency of IP authorization checks.

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_auth_jwt_verification_count` (Counter)

Total number of JWT verifications by signing-method and outcome.

Labels

method: Signing algorithm used (JWT alg), such as "HS256" or "ES256".
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_encryption_key_refresh_count` (Counter)

Total number of encryption key refresh attempts.

`buildbuddy_encryption_key_refresh_failure_count` (Counter)

Total number of unsuccessful encryption key refresh attempts.

`buildbuddy_encryption_encrypted_block_count` (Counter)

Total number of blocks encrypted.

`buildbuddy_encryption_encrypted_blob_count` (Counter)

Total number of blobs encrypted.

`buildbuddy_encryption_decrypted_block_count` (Counter)

Total number of blocks decrypted.

`buildbuddy_encryption_decrypted_blob_count` (Counter)

Total number of blobs decrypted.

`buildbuddy_encryption_decryption_error_count` (Counter)

Total number of decryption errors.

`buildbuddy_encryption_key_last_encryption_age_msec` (Histogram)

Age of encrypted keys (i.e.

how long it has been since the keys were re-encrypted).

`buildbuddy_remote_cache_pebble_cache_atime_update_count` (Counter)

Count of processed atime updates.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_gcs_operation_count` (Counter)

Count of GCS operations performed by the pebble cache.

Labels

op:
partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_pebble_cache_atime_delta_when_read` (Histogram)

Previous atime of items in the cache when they are read, in msec

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_cas_age_when_atime_updated` (Histogram)

Records (atime - mtime) every time a CAS entry's atime is updated.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_eviction_samples_chan_size` (Gauge)

Num of items in eviction samples chan

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_eviction_resample_latency_usec` (Histogram)

Latency of resampling during a single eviction iteration.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_eviction_evict_latency_usec` (Histogram)

Latency of evicting a single key.

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_eviction_samples` (Counter)

Number of samples produced, by status

Labels

partition_id: The ID of the disk cache partition this event applied to.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_remote_cache_pebble_cache_pebble_compact_count` (Counter)

Number of compactions performed by the underlying Pebble database.

Labels

compaction_type: Pebble DB compaction type.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_compact_estimated_debt_bytes` (Gauge)

Estimated number of bytes that need to be compacted for the LMS to reach a stable state.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_compact_in_progress_bytes` (Gauge)

Number of bytes present in sstables being written by in-progress compactions.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_compact_in_progress` (Gauge)

Number of compactions that are in-progress

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_compact_marked_files` (Gauge)

Count of files that are marked for compaction.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_sublevels` (Gauge)

Number of sublevels within the level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_num_files` (Gauge)

The total number of files in the level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_size_bytes` (Gauge)

The total size in bytes of the files in the level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_score` (Gauge)

The level's compaction score.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_bytes_in_count` (Counter)

The number of incoming bytes from other levels read during compactions.

This excludes bytes moved and bytes ingested. For L0 this is the bytes written to the WAL.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_bytes_ingested_count` (Counter)

The number of bytes ingested.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_bytes_moved_count` (Counter)

The number of bytes moved into the level by a move compaction.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_bytes_read_count` (Counter)

The number of bytes read for compactions at the level.

This includes bytes read from other levels (BytesIn), as well as bytes read for the level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_bytes_compacted_count` (Counter)

The number of bytes written during compactions.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_bytes_flushed_count` (Counter)

The number of bytes written during flushes.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_tables_compacted_count` (Counter)

The number of sstables compacted to this level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_tables_flushed_count` (Counter)

The number of sstables flushed to this level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_tables_ingested_count` (Counter)

The number of sstables ingested into this level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_level_tables_moved_count` (Counter)

The number of sstables ingested into to this level.

Labels

level: Pebble DB level number.
cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_op_count` (Counter)

The number of operations performed against the pebble database.

Labels

pebble_id: Pebble DB ID
pebble_op: Pebble DB operation type.

`buildbuddy_remote_cache_pebble_cache_pebble_op_latency_usec` (Histogram)

The latency of operations performed against the pebble database, in microseconds.

Labels

pebble_id: Pebble DB ID
pebble_op: Pebble DB operation type.

`buildbuddy_remote_cache_pebble_cache_pebble_block_cache_size_bytes` (Gauge)

The total size in pebble's block cache.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_block_cache_requests_count` (Counter)

The number of block cache requests by hit/miss status.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_remote_cache_pebble_cache_pebble_table_cache_requests_count` (Counter)

The number of table cache requests by hit/miss status.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_remote_cache_pebble_cache_pebble_write_stall_count` (Gauge)

The number of write stalls

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_pebble_write_stall_duration_usec` (Histogram)

The duration of write stall in pebble, in microseconds.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_zombie_table_count` (Gauge)

The number of zombie tables

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_zombie_table_size_bytes` (Gauge)

The size of zombie tables in bytes

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_find_missing_digest_count` (Counter)

Count of digests within FindMissing requests.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

`buildbuddy_remote_cache_pebble_cache_find_missing_blob_status_count` (Counter)

Count of blobs checked by FindMissing on this node, by present/absent status and originating purpose.

This is the per-node LOOKUP view: a blob retried across replicas by the distributed cache is counted on each node, so absents are inflated by replication vs the logical rate (see distributed_cache_find_missing_blob_status_count).

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".
purpose: FindMissing purpose: which internal code path originated a FindMissing lookup (see repb.FindMissingBlobsRequest.Purpose), e.g. "ATIME_UPDATE".
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_remote_cache_pebble_cache_presence_cache_entry_count` (Gauge)

Number of entries in the cache.

Labels

cache_name: Cache name: Custom name to describe the cache, like "pebble-cache".

Podman metrics

`buildbuddy_podman_image_pull_latency_msec` (Histogram)

The latency of 'cold' podman pull requests per image, in milliseconds.

'Cold' means the image hasn't been pulled by this executor previously.

Labels

container_image_tag: Container image tag.

Cache Proxy metrics

`buildbuddy_proxy_action_cache_read_requests` (Counter)

The number of ActionCache.GetActionResult requests served by a ActionCacheServerProxy by gRPC status, cache hit/miss status, request type, and result source.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.
result_source: Source used by a cache proxy request to produce a response.

`buildbuddy_proxy_action_cache_write_requests` (Counter)

The number of ActionCache.UpdateActionResult requests served by a ActionCacheServerProxy by gRPC status and cache hit/miss status.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.

`buildbuddy_proxy_action_cache_read_bytes` (Counter)

The number of ActionCache.GetActionResult bytes served by a ActionCacheServerProxy by gRPC status, cache hit/miss status, request type, and result source.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.
result_source: Source used by a cache proxy request to produce a response.

`buildbuddy_proxy_action_cache_write_bytes` (Counter)

The number of ActionCache.UpdateActionResult bytes served by a ActionCacheServerProxy by gRPC status and cache hit/miss status.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.

`buildbuddy_proxy_byte_stream_read_requests` (Counter)

The number of ByteStream.Read requests served by a ByteStreamServerProxy broken down by gRPC status and cache hit/miss status.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.
compression: Describes the type of compression
chunked: Whether the request was handled using chunking

`buildbuddy_proxy_byte_stream_write_requests` (Counter)

The number of ByteStream.Write requests served by a ByteStreamServerProxy broken down by gRPC status and cache hit/miss status.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.
compression: Describes the type of compression
chunked: Whether the request was handled using chunking

`buildbuddy_proxy_byte_stream_read_bytes` (Counter)

The number of bytes read by ByteStream.Read RPCs served by a ByteStreamServerProxy broken down by gRPC status and cache hit/miss status.

Note: this metric tracks bytes sent over the wire, which may be compressed.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.
compression: Describes the type of compression
chunked: Whether the request was handled using chunking
group_id: Group (organization) ID associated with the request.

`buildbuddy_proxy_byte_stream_write_bytes` (Counter)

The number of bytes written to a ByteStream.Write RPCs served by a ByteStreamServerProxy broken down by gRPC status and cache hit/miss status.

Note: this metric tracks bytes sent over the wire, which may be compressed.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
proxy_request_type: Cache proxy request type. If "local_only", indicates the proxy should only use its local cache to fulfil the request. If "default", the proxy should fall back to the remote cache as the source of truth.
compression: Describes the type of compression
chunked: Whether the request was handled using chunking

`buildbuddy_proxy_byte_stream_chunked_write_blob_bytes` (Counter)

Original blob size in bytes for chunked writes.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_write_chunk_bytes_total` (Counter)

Total chunk bytes produced during chunked writes (sum of all chunk sizes).

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_write_chunk_bytes_deduped` (Counter)

Chunk bytes that were deduplicated (already existed on remote) during chunked writes.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_write_chunks_total` (Counter)

Total number of chunks produced during chunked writes.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
group_id: Group (organization) ID associated with the request.

`buildbuddy_proxy_byte_stream_chunked_write_chunks_deduped` (Counter)

Number of chunks that were deduplicated (already existed on remote) during chunked writes.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
group_id: Group (organization) ID associated with the request.

`buildbuddy_proxy_byte_stream_chunked_write_by_group_chunk_bytes_total` (Counter)

Total chunk bytes produced during chunked writes by group ID (sum of all chunk sizes).

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
group_id: Group (organization) ID associated with the request.

`buildbuddy_proxy_byte_stream_chunked_write_by_group_chunk_bytes_deduped` (Counter)

Chunk bytes that were deduplicated (already existed on remote) during chunked writes by group ID.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
group_id: Group (organization) ID associated with the request.

`buildbuddy_proxy_byte_stream_chunked_write_by_action_mnemonic_chunk_bytes_total` (Counter)

Total chunk bytes produced during chunked writes by action mnemonic (sum of all chunk sizes).

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_proxy_byte_stream_chunked_write_by_action_mnemonic_chunk_bytes_deduped` (Counter)

Chunk bytes that were deduplicated (already existed on remote) during chunked writes by action mnemonic.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_proxy_byte_stream_chunked_write_by_action_mnemonic_chunks_total` (Counter)

Total number of chunks produced during chunked writes by action mnemonic.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_proxy_byte_stream_chunked_write_by_action_mnemonic_chunks_deduped` (Counter)

Number of chunks that were deduplicated (already existed on remote) during chunked writes by action mnemonic.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression
action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_cache_client_chunked_upload_chunk_bytes_total` (Counter)

Total uncompressed chunk bytes produced during chunked uploads.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_cache_client_chunked_upload_chunk_bytes_deduped` (Counter)

Uncompressed chunk bytes deduplicated (already existed on remote) during chunked uploads.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_cache_client_chunked_upload_chunks_total` (Counter)

Total number of chunks produced during chunked uploads.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_cache_client_chunked_upload_chunks_deduped` (Counter)

Number of chunks deduplicated (already existed on remote) during chunked uploads.

Labels

group_id: Group (organization) ID associated with the request.

`buildbuddy_cache_client_chunked_upload_by_action_mnemonic_chunk_bytes_total` (Counter)

Total uncompressed chunk bytes produced during chunked uploads by action mnemonic.

Labels

action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_cache_client_chunked_upload_by_action_mnemonic_chunk_bytes_deduped` (Counter)

Uncompressed chunk bytes deduplicated (already existed on remote) during chunked uploads by action mnemonic.

Labels

action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_cache_client_chunked_upload_by_action_mnemonic_chunks_total` (Counter)

Total number of chunks produced during chunked uploads by action mnemonic.

Labels

action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_cache_client_chunked_upload_by_action_mnemonic_chunks_deduped` (Counter)

Number of chunks deduplicated (already existed on remote) during chunked uploads by action mnemonic.

Labels

action_mnemonic: Short action name, such as "GoCompile".

`buildbuddy_cache_client_chunked_download_chunk_bytes_total` (Counter)

Total uncompressed chunk bytes read during chunked downloads.

`buildbuddy_cache_client_chunked_download_chunk_bytes_local` (Counter)

Uncompressed chunk bytes served from a local file during chunked downloads.

`buildbuddy_cache_client_chunked_download_chunks_total` (Counter)

Total number of chunks read during chunked downloads.

`buildbuddy_cache_client_chunked_download_chunks_local` (Counter)

Number of chunks served from a local file during chunked downloads.

`buildbuddy_proxy_byte_stream_chunked_write_duration_usec` (Histogram)

Total duration of chunked write operations, in microseconds.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_write_chunking_duration_usec` (Histogram)

Duration of the chunking phase (receive, decompress, chunk, compress, local write), in microseconds.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_write_remote_duration_usec` (Histogram)

Duration of the remote phase (FindMissingBlobs, upload, SpliceBlob), in microseconds.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_write_upload_size_bytes` (Histogram)

Compressed upload size for remote chunk uploads during chunked writes.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_proxy_byte_stream_chunked_read_requests` (Counter)

Total number of successful chunked read requests.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_blob_bytes` (Counter)

Original blob size in bytes for chunked reads.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_chunks_total` (Counter)

Total number of chunks read during chunked reads.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_chunks_local` (Counter)

Number of chunks served from local cache during chunked reads.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_chunks_remote` (Counter)

Number of chunks fetched from remote during chunked reads.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_bytes_local` (Counter)

Bytes served from local cache to the client during chunked reads.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_bytes_remote` (Counter)

Bytes fetched from remote and served to the client during chunked reads.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
compression: Describes the type of compression

`buildbuddy_proxy_byte_stream_chunked_read_fast_path_attempts` (Counter)

Number of chunked reads where the CDC read fast path was attempted, by outcome.

Labels

outcome: Outcome of a cache proxy fast path attempt.

`buildbuddy_proxy_byte_stream_chunked_read_local_manifest_store_attempts` (Counter)

Number of attempts to store a CDC manifest in the proxy local cache after a successful remote SplitBlob.

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_proxy_byte_stream_chunked_read_local_write_back_failures` (Counter)

Number of failures while writing remotely fetched chunks back to the local cache during chunked reads.

Labels

status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_proxy_byte_stream_chunked_read_failures` (Counter)

Number of chunked read attempts that failed, by failure reason.

Labels

reason:
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".

`buildbuddy_remote_cache_byte_stream_chunked_read_failures` (Counter)

Number of chunked read attempts that failed on the backend, by failure reason.

Labels

reason:
status: Status code as defined by grpc/codes in human-readable format, such as "OK" or "NotFound".
offset_read: Whether the read request had a non-zero offset (e.g. Bazel retry resuming a partial read).

`buildbuddy_proxy_capabilities_requests` (Counter)

The number of requests served by a CapabilitiesServerProxy by gRPC status and cache hit/miss status.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_proxy_capabilities_bytes` (Counter)

The number of bytes served by a CapabilitiesServerProxy by gRPC status and cache hit/miss status.

Labels

status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_proxy_content_addressable_storage_requests` (Counter)

The number of requests served by a ContentAddressableStorageServerProxy by CAS operation and cache hit/miss status.

Labels

op: ContentAddressableStorage Server operation: "FindMissingBlobs", "BatchUpdateBlobs", "BatchReadBlobs", or "GetTree".
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)

`buildbuddy_proxy_content_addressable_storage_digests` (Counter)

The number of digests served by a ContentAddressableStorageServerProxy by CAS operation and cache hit/miss status.

Labels

op: ContentAddressableStorage Server operation: "FindMissingBlobs", "BatchUpdateBlobs", "BatchReadBlobs", or "GetTree".
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
compression: Describes the type of compression

`buildbuddy_proxy_content_addressable_storage_bytes` (Counter)

The number of bytes served by a ContentAddressableStorageServerProxy by CAS operation and cache hit/miss status.

Labels

op: ContentAddressableStorage Server operation: "FindMissingBlobs", "BatchUpdateBlobs", "BatchReadBlobs", or "GetTree".
cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
compression: Describes the type of compression

`buildbuddy_proxy_find_missing_blobs_cache_lookups` (Counter)

The number of digests looked up in the Cache Proxy's local FindMissingBlobs cache, by lookup status, whether the digests are CDC chunks, and result source.

'hit': served from the local cache. 'miss': the remote reported the digest present, so the lookup could have been a hit. 'uncacheable': the remote reported the digest missing; absence is never cached, so these lookups always require a remote check.

Labels

cache_status: Cache lookup result - One of: - "hit" - "miss" - "partial" (for batched RPCs where part of a request was cached) - Or "uncacheable" (for e.g. encrypted resources)
chunked: Whether the request was handled using chunking
result_source: Source used by a cache proxy request to produce a response.

`buildbuddy_proxy_remote_atime_updates` (Counter)

The number of remote atime updates enqueued, with the outcome of the enqueue operation.

Labels

group_id: Group (organization) ID associated with the request.
status: There are a couple of places where we enqueue RPCs to be batched and sent asynchronously: the atime_updater and the hit_tracker_client. This label tracks the outcome of these enqueue operations. One of "enqueued", "duplicate", "dropped_too_many_updates", or "dropped_too_many_batches"

`buildbuddy_proxy_remote_atime_update_requests` (Counter)

The number of FindMissingBlobRequests sent to the remote cache to update remote blob atimes.

Labels

group_id: Group (organization) ID associated with the request.
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_proxy_remote_hit_tracker_updates` (Counter)

The number of remote hit-tracker updates enqueued, with the outcome of the enqueue operation.

Labels

group_id: Group (organization) ID associated with the request.
status: There are a couple of places where we enqueue RPCs to be batched and sent asynchronously: the atime_updater and the hit_tracker_client. This label tracks the outcome of these enqueue operations. One of "enqueued", "duplicate", "dropped_too_many_updates", or "dropped_too_many_batches"

`buildbuddy_proxy_remote_hit_tracker_requests` (Histogram)

The number of HitTrackerService.Track RPCs sent to the remote hit-tracker service to record cache proxy cache hits.

Histogram values are the number of hit-updates per request.

Labels

group_id: Group (organization) ID associated with the request.
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.

`buildbuddy_ociregistry_cache_download_size_bytes` (Histogram)

Number of bytes downloaded from the cache by the OCI registry mirror

Labels

oci_resource_type:

`buildbuddy_ociregistry_cache_events` (Counter)

Number of cache events handled.

Labels

oci_resource_type:
cache_event_type: Cache event type: hit, miss, or upload.

`buildbuddy_ociregistry_egress_size_bytes` (Counter)

The number of bytes served, broken down by destination provider/region inferred from the peer IP.

Labels

provider: Destination provider inferred from the remote IP range: a cloud or hosting provider (aws, gcp, azure, github, digitalocean, oracle, macstadium, phoenixnap, hetzner, ovh, latitude.sh, namespace), one of our own networks (metal, psc, privatelink, internal, loopback), or other for any public IP outside every known range. See enterprise/server/util/trafficstats.
remote_region: Destination region inferred from the remote IP range, or unknown if no known cloud range matches.

`buildbuddy_ocifetcher_request_count` (Counter)

Number of OCIFetcher requests by method, role, and status.

Labels

method:
role:
status:

`buildbuddy_ocifetcher_request_duration_usec` (Histogram)

Duration of OCIFetcher requests by method and role, in microseconds.

Labels

method:
role:

`buildbuddy_remote_execution_input_tree_setup_op_latency_usec` (Histogram)

Latency of individual operations used for setting up input trees.

Labels

`buildbuddy_disk_file_writer_in_progress_ops` (Gauge)

Number of started, but not yet finished, FileWriter operations.

This number includes operations that are blocked on the concurrency limiter.

`buildbuddy_disk_file_writer_tmp_file_bytes` (Gauge)

Total size, in bytes, of temporary files currently staging writes for in-progress FileWriter operations.

Incremented as bytes are written to the temp file, and decremented when the temp file is committed to its final path or deleted.

Container image fetch metrics

`buildbuddy_remote_execution_image_fetch_duration_usec` (Histogram)

Duration of container image fetch attempts on executors, in microseconds.

Use the _count suffix for fetch counts.

Labels

isolation: Effective workload isolation type used for an executed task, such as "docker", "podman", "firecracker", or "none".
registry: Label name for the eTLD+1 of the container image registry.
status: Status code as defined by grpc/codes. This is a numeric value; any non-zero code indicates an error.
on_disk: Label name for whether the image was already on disk on the executor.
has_creds: Label name for whether credentials were provided for the image fetch.
trigger: Label name for what triggered the image fetch.
use_oci_fetcher: Label name for whether the OCI fetcher service was used for the image fetch.

`buildbuddy_grpc_client_rpcs_per_connection` (Gauge)

A gauge measuring the number of pending RPCs per gRPC client connection, broken down by target, connection pool, and gRPC method.

Labels

target:
pool_id: Unique identifier for a gRPC client connection pool. For disambiguating between multiple pools connecting to the same target.
grpc_method:
connection_id: The index of a gRPC connection in a gRPC connection pool

`buildbuddy_grpc_client_connection_count` (Gauge)

Number of client gRPC connections (channelz sockets) observed per target, sampled from channelz.

Labels

target:

`buildbuddy_grpc_client_flow_control_blocked_connections` (Gauge)

Number of client gRPC connections whose remote HTTP/2 flow-control (send) window is 0, per target.

A nonzero value means connections are send-blocked behind the peer's flow control, which head-of-line-blocks every stream on the connection.

Labels

target:

`buildbuddy_grpc_client_flow_control_window_bytes` (Histogram)

Distribution of HTTP/2 flow-control window sizes across client gRPC connections, sampled periodically from channelz.

direction=remote is the window for data we send (0 means send-blocked); direction=local is the window for data we receive.

Labels

target:
direction: The direction of an HTTP/2 flow-control window relative to the local endpoint: remote (the window for data we send) or local (the window for data we receive).

`buildbuddy_grpc_client_connection_open_streams` (Histogram)

Distribution of open HTTP/2 streams across client gRPC connections, sampled periodically from channelz.

Values near the peer's max-concurrent-streams limit (commonly 100) indicate connections at capacity.

Labels

target:

`buildbuddy_grpc_server_egress_bytes` (Counter)

The number of gRPC server response bytes sent over the wire, broken down by group, peer type, and destination provider/region inferred from the peer IP.

Bytes forwarded by another BuildBuddy server are counted there too; filter peer_type=external to count each byte once. Note: this metric tracks gRPC payload bytes, which may be compressed, and does not include HTTP/2 framing or response headers.

Labels

group_id: Group (organization) ID associated with the request.
peer_type: Verified client identity of the peer (e.g. cache-proxy, executor), or external when it presented none.
provider: Destination provider inferred from the remote IP range: a cloud or hosting provider (aws, gcp, azure, github, digitalocean, oracle, macstadium, phoenixnap, hetzner, ovh, latitude.sh, namespace), one of our own networks (metal, psc, privatelink, internal, loopback), or other for any public IP outside every known range. See enterprise/server/util/trafficstats.
remote_region: Destination region inferred from the remote IP range, or unknown if no known cloud range matches.

`buildbuddy_grpc_server_ingress_bytes` (Counter)

The number of gRPC server request bytes received over the wire, broken down by group, peer type, and source provider/region inferred from the peer IP.

Bytes relayed by another BuildBuddy server were counted there too; filter peer_type=external to count each byte once. Note: this metric tracks gRPC payload bytes, which may be compressed, and does not include HTTP/2 framing or request headers.

Labels

group_id: Group (organization) ID associated with the request.
peer_type: Verified client identity of the peer (e.g. cache-proxy, executor), or external when it presented none.
provider: Destination provider inferred from the remote IP range: a cloud or hosting provider (aws, gcp, azure, github, digitalocean, oracle, macstadium, phoenixnap, hetzner, ovh, latitude.sh, namespace), one of our own networks (metal, psc, privatelink, internal, loopback), or other for any public IP outside every known range. See enterprise/server/util/trafficstats.
remote_region: Destination region inferred from the remote IP range, or unknown if no known cloud range matches.

`buildbuddy_grpc_server_requests` (Counter)

The number of finished gRPC server requests, broken down by group, peer type, and source provider/region inferred from the peer IP.

Requests forwarded by another BuildBuddy server were counted there too; filter peer_type=external to count each request once.

Labels

group_id: Group (organization) ID associated with the request.
peer_type: Verified client identity of the peer (e.g. cache-proxy, executor), or external when it presented none.
provider: Destination provider inferred from the remote IP range: a cloud or hosting provider (aws, gcp, azure, github, digitalocean, oracle, macstadium, phoenixnap, hetzner, ovh, latitude.sh, namespace), one of our own networks (metal, psc, privatelink, internal, loopback), or other for any public IP outside every known range. See enterprise/server/util/trafficstats.
remote_region: Destination region inferred from the remote IP range, or unknown if no known cloud range matches.

DNS server metrics

`buildbuddy_dns_server_request_count` (Counter)

The total number of DNS queries handled, by record type, response code, and recursive resolver provider.

Labels

record_type: The DNS query (record) type, such as "A", "AAAA", "CNAME", or "MX". Unrecognized types are bucketed as "OTHER" to bound cardinality, since the type is client-controlled.
rcode: The DNS response code, such as "NOERROR", "NXDOMAIN", or "FORMERR".
resolver_provider: The provider operating the recursive resolver that sent the query, inferred from the transport peer's ASN. Values are bounded by the DNS server.

Examples

# DNS queries per second by record type
sum by (record_type) (rate(buildbuddy_dns_server_request_count[5m]))

# DNS queries per second by recursive resolver provider
sum by (resolver_provider) (rate(buildbuddy_dns_server_request_count[5m]))

# NXDOMAIN rate
sum(rate(buildbuddy_dns_server_request_count{rcode="NXDOMAIN"}[5m]))
/
sum(rate(buildbuddy_dns_server_request_count[5m]))

`buildbuddy_dns_server_handler_duration_usec` (Histogram)

Time to handle a DNS query, in microseconds, by record type.

Labels

record_type: The DNS query (record) type, such as "A", "AAAA", "CNAME", or "MX". Unrecognized types are bucketed as "OTHER" to bound cardinality, since the type is client-controlled.

Examples

# Median DNS handler latency in the past 5 minutes
histogram_quantile(
0.5,
sum(rate(buildbuddy_dns_server_handler_duration_usec_bucket[5m])) by (le)
)

`buildbuddy_dns_server_zone_serial` (Gauge)

The SOA serial of each DNS zone currently served, by zone apex.

A zone that stops being served drops off the metric.

Labels

dns_zone: The apex of a served DNS zone, such as "buildbuddy.io.". Zones come from operator-controlled zone files, so cardinality is bounded. Named "dns_zone" because ZoneLabel ("zone") is the availability zone of a node.

Examples

# Zones currently served, with their versions
max by (dns_zone) (buildbuddy_dns_server_zone_serial)

# Zones where replicas disagree on the served version
max by (dns_zone) (buildbuddy_dns_server_zone_serial)
!=
min by (dns_zone) (buildbuddy_dns_server_zone_serial)

SSL metrics
- buildbuddy_ssl_certificate_reload_failures_total (Counter)
Invocation build event metrics
Remote cache metrics
Remote execution metrics
Blobstore metrics
database/sql metrics
HTTP metrics
Redis client metrics
- buildbuddy_redis_client_dial_errors (Counter)
Internal metrics
Podman metrics
- buildbuddy_podman_image_pull_latency_msec (Histogram)
Cache Proxy metrics
Container image fetch metrics
DNS server metrics

SSL metrics​

buildbuddy_ssl_certificate_reload_failures_total (Counter)​

Invocation build event metrics​

buildbuddy_invocation_count (Counter)​

Labels​

Examples​

buildbuddy_invocation_discarded_anonymous_build_event_stream_count (Counter)​

buildbuddy_invocation_log_live_chunk_written_bytes (Counter)​

buildbuddy_invocation_duration_usec (Histogram)​

Labels​

Examples​

buildbuddy_invocation_open_streams (Gauge)​

buildbuddy_invocation_build_event_count (Counter)​

Labels​

Examples​

buildbuddy_invocation_stats_recorder_workers (Gauge)​

buildbuddy_invocation_stats_recorder_duration_usec (Histogram)​

buildbuddy_invocation_webhook_invocation_lookup_workers (Gauge)​

buildbuddy_invocation_webhook_invocation_lookup_duration_usec (Histogram)​

buildbuddy_invocation_webhook_invocation_payload_size_bytes (Histogram)​

buildbuddy_invocation_webhook_notify_workers (Gauge)​

buildbuddy_invocation_webhook_notify_duration_usec (Histogram)​

Remote cache metrics​

buildbuddy_remote_cache_events (Counter)​

Labels​

buildbuddy_remote_cache_download_size_bytes (Histogram)​

Labels​

Examples​

buildbuddy_remote_cache_download_duration_usec (Histogram)​

Labels​

Examples​

buildbuddy_remote_cache_upload_size_bytes (Histogram)​

Labels​

Examples​

buildbuddy_remote_cache_upload_duration_usec (Histogram)​

Labels​

Examples​

buildbuddy_remote_cache_method_handled_total (Counter)​

Labels​

buildbuddy_remote_cache_method_handling_usec (Histogram)​

Labels​

buildbuddy_remote_cache_disk_cache_last_eviction_age_usec (Gauge)​

Labels​

buildbuddy_remote_cache_disk_cache_eviction_age_msec (Histogram)​

Labels​

buildbuddy_remote_cache_disk_cache_num_evictions (Counter)​

Labels​

buildbuddy_remote_cache_disk_cache_partition_size_bytes_evicted (Counter)​

Labels​

buildbuddy_remote_cache_disk_cache_partition_size_bytes (Gauge)​

Labels​

buildbuddy_remote_cache_disk_cache_sampled_partition_group_size_bytes (Gauge)​

Labels​

buildbuddy_remote_cache_disk_cache_partition_capacity_bytes (Gauge)​

Labels​

buildbuddy_remote_cache_disk_cache_partition_num_items (Gauge)​

Labels​

buildbuddy_remote_cache_disk_cache_duplicate_writes (Counter)​

Labels​

buildbuddy_remote_cache_disk_cache_added_file_size_bytes (Histogram)​

Labels​

buildbuddy_remote_cache_disk_cache_filesystem_total_bytes (Gauge)​

Labels​

buildbuddy_remote_cache_disk_cache_filesystem_avail_bytes (Gauge)​

Labels​

Examples​

buildbuddy_remote_cache_disk_cache_duplicate_writes_bytes (Counter)​

Labels​

buildbuddy_remote_cache_distributed_cache_peer_lookups (Histogram)​

Labels​

buildbuddy_remote_cache_distributed_cache_find_missing_blob_status_count (Counter)​

Labels​

buildbuddy_remote_cache_distributed_cache_backfill_count (Counter)​

Labels​

buildbuddy_remote_cache_distributed_cache_backfill_size_bytes (Counter)​

Labels​

buildbuddy_remote_cache_distributed_cache_backfill_latency_usec (Histogram)​

Labels​

buildbuddy_remote_cache_distributed_cache_read_count (Counter)​

Labels​

SSL metrics

`buildbuddy_ssl_certificate_reload_failures_total` (Counter)

Invocation build event metrics

`buildbuddy_invocation_count` (Counter)

Labels

Examples

`buildbuddy_invocation_discarded_anonymous_build_event_stream_count` (Counter)

`buildbuddy_invocation_log_live_chunk_written_bytes` (Counter)

`buildbuddy_invocation_duration_usec` (Histogram)

Labels

Examples

`buildbuddy_invocation_open_streams` (Gauge)

`buildbuddy_invocation_build_event_count` (Counter)

Labels

Examples

`buildbuddy_invocation_stats_recorder_workers` (Gauge)

`buildbuddy_invocation_stats_recorder_duration_usec` (Histogram)

`buildbuddy_invocation_webhook_invocation_lookup_workers` (Gauge)

`buildbuddy_invocation_webhook_invocation_lookup_duration_usec` (Histogram)

`buildbuddy_invocation_webhook_invocation_payload_size_bytes` (Histogram)

`buildbuddy_invocation_webhook_notify_workers` (Gauge)

`buildbuddy_invocation_webhook_notify_duration_usec` (Histogram)

Remote cache metrics

`buildbuddy_remote_cache_events` (Counter)

Labels

`buildbuddy_remote_cache_download_size_bytes` (Histogram)

Labels

Examples

`buildbuddy_remote_cache_download_duration_usec` (Histogram)

Labels

Examples

`buildbuddy_remote_cache_upload_size_bytes` (Histogram)

Labels

Examples

`buildbuddy_remote_cache_upload_duration_usec` (Histogram)

Labels

Examples

`buildbuddy_remote_cache_method_handled_total` (Counter)

Labels

`buildbuddy_remote_cache_method_handling_usec` (Histogram)

Labels

`buildbuddy_remote_cache_disk_cache_last_eviction_age_usec` (Gauge)

Labels

`buildbuddy_remote_cache_disk_cache_eviction_age_msec` (Histogram)

Labels

`buildbuddy_remote_cache_disk_cache_num_evictions` (Counter)

Labels

`buildbuddy_remote_cache_disk_cache_partition_size_bytes_evicted` (Counter)

Labels

`buildbuddy_remote_cache_disk_cache_partition_size_bytes` (Gauge)

Labels

`buildbuddy_remote_cache_disk_cache_sampled_partition_group_size_bytes` (Gauge)

Labels

`buildbuddy_remote_cache_disk_cache_partition_capacity_bytes` (Gauge)

Labels

`buildbuddy_remote_cache_disk_cache_partition_num_items` (Gauge)

Labels

`buildbuddy_remote_cache_disk_cache_duplicate_writes` (Counter)

Labels

`buildbuddy_remote_cache_disk_cache_added_file_size_bytes` (Histogram)

Labels

`buildbuddy_remote_cache_disk_cache_filesystem_total_bytes` (Gauge)

Labels

`buildbuddy_remote_cache_disk_cache_filesystem_avail_bytes` (Gauge)

Labels

Examples

`buildbuddy_remote_cache_disk_cache_duplicate_writes_bytes` (Counter)

Labels

`buildbuddy_remote_cache_distributed_cache_peer_lookups` (Histogram)

Labels

`buildbuddy_remote_cache_distributed_cache_find_missing_blob_status_count` (Counter)

Labels

`buildbuddy_remote_cache_distributed_cache_backfill_count` (Counter)

Labels

`buildbuddy_remote_cache_distributed_cache_backfill_size_bytes` (Counter)

Labels

`buildbuddy_remote_cache_distributed_cache_backfill_latency_usec` (Histogram)

Labels

`buildbuddy_remote_cache_distributed_cache_read_count` (Counter)

Labels