perf(tests): parallelize kokoro periodic perf tests across 3 VMs by PranjalC100 · Pull Request #4618 · GoogleCloudPlatform/gcsfuse · GitHub

PranjalC100 · 2026-04-14T17:40:35Z

Description

This change refactors the periodic performance tests to run in parallel across multiple Kokoro VMs, significantly reducing the overall wall-clock execution time.

The monolithic execution in build.sh has been updated to act as a router based on the BENCHMARK_TYPE environment variable. The execution is now split into three concurrent child jobs:

read_distributed: Dedicated entirely to distributed read micro-benchmarks.
write_distributed_and_local_tests: Runs the distributed write benchmarks in parallel, alongside the remaining local flat/HNS bucket tests and rename benchmarks.
zonal_distributed: Adds scaffolding for the upcoming zonal performance tests.

Link to the issue in case of a bug fix.

https://b.corp.google.com/issues/502497832

Testing details

Manual - Verified parallel execution and routing logic via a cloned QA Kokoro codelab environment.
Unit tests - NA
Integration tests - NA

Any backward incompatible change? If so, please explain.

No.

gemini-code-assist · 2026-04-21T19:57:06Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request optimizes the periodic performance testing pipeline by moving from a single, sequential execution model to a parallelized approach. By leveraging multiple Kokoro VMs and introducing a routing mechanism in the build script, the overall duration of the performance test suite is significantly reduced. The changes include new configuration files for each test category and improved script robustness with better error handling and execution time tracking.

Highlights

Parallelization of Performance Tests: Refactored the monolithic periodic performance test execution into three parallelizable jobs across multiple Kokoro VMs to reduce total wall-clock time.
Router Logic Implementation: Updated build.sh to act as a router based on the BENCHMARK_TYPE environment variable, enabling targeted execution of read, write/local, and zonal benchmarks.
Configuration Additions: Added new Kokoro configuration files (read_distributed.cfg, write_distributed_and_local_tests.cfg, and zonal_distributed.cfg) to support the new parallelized execution model.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

codecov · 2026-04-21T19:57:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (e76ad16) to head (d81be43).
⚠️ Report is 13 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #4618       +/-   ##
==========================================
- Coverage   83.67%       0   -83.68%     
==========================================
  Files         164       0      -164     
  Lines       20216       0    -20216     
==========================================
- Hits        16916       0    -16916     
+ Misses       2663       0     -2663     
+ Partials      637       0      -637

Flag	Coverage Δ
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gemini-code-assist

Code Review

This pull request refactors the performance testing infrastructure by modularizing the build.sh script to support different benchmark types via the BENCHMARK_TYPE environment variable. It introduces specific execution paths for distributed read, distributed write with local tests, and zonal benchmarks, along with corresponding Kokoro configuration files. The feedback focuses on improving test coverage by ensuring that local performance tests are not skipped if distributed benchmarks fail, and removing redundant exit statements that make trailing code unreachable.

gemini-code-assist · 2026-04-21T20:00:32Z

+  if [ $PERF_BENCHMARKS_FAILED -ne 0 ]; then
+    echo "Distributed WRITE benchmarks have failed."
+    exit 1
+  fi


This early exit prevents the local performance tests (flat bucket, HNS bucket, and rename benchmarks) from running if the distributed write benchmark fails. In the original script, these tests were independent, and a failure in the distributed part did not block the execution of local tests. To maintain maximum test coverage and visibility into all components, consider removing this check and performing it at the end of the block instead.

Suggested change

if [ $PERF_BENCHMARKS_FAILED -ne 0 ]; then

echo "Distributed WRITE benchmarks have failed."

exit 1

fi

# Removed early exit to allow local tests to run even if distributed write fails.

gemini-code-assist · 2026-04-21T20:00:32Z

+  print_duration "Rename Benchmark" "$RENAME_START"
+  exit 0


The exit 0 here (and at lines 67 and 192) is redundant because bash will naturally skip the remaining elif/else blocks once a condition is met. More importantly, it makes any code following the fi (like the TODO at line 198) unreachable. If you moved the distributed write failure check as suggested earlier, you should place it here.

Suggested change

print_duration "Rename Benchmark" "$RENAME_START"

exit 0

print_duration "Rename Benchmark" "$RENAME_START"

if [ $PERF_BENCHMARKS_FAILED -ne 0 ]; then

echo "Distributed WRITE benchmarks have failed."

exit 1

fi

github-actions · 2026-04-23T04:30:23Z