'test_scale_bigtable' flakes on AssertionError

[tseaver]

From this Kokoro build:

_____________________________ test_scale_bigtable ______________________________

instance = <google.cloud.bigtable.instance.Instance object at 0x7f3490cb2490>

    def test_scale_bigtable(instance):
        bigtable_client = bigtable.Client(admin=True)

        instance = bigtable_client.instance(BIGTABLE_INSTANCE)
        instance.reload()

        cluster = instance.cluster(BIGTABLE_INSTANCE)
        cluster.reload()
        original_node_count = cluster.serve_nodes

        scale_bigtable(BIGTABLE_INSTANCE, BIGTABLE_INSTANCE, True)

        for n in range(10):
            time.sleep(10)
            cluster.reload()
            new_node_count = cluster.serve_nodes
            try:
>               assert (new_node_count == (original_node_count + SIZE_CHANGE_STEP))
E               assert 1 == (1 + 3)

metricscaler_test.py:122: AssertionError
----------------------------- Captured stderr call -----------------------------
Scaled up from 1 to 4 nodes.
------------------------------ Captured log call -------------------------------
INFO     bigtable.metricscaler:metricscaler.py:121 Scaled up from 1 to 4 nodes.

[HemangChothani]

@tseaver I think it's old and resolved, I have tried to reproduce it many time but not succeed, also found here https://flaky-dashboard.web.app/org/googleapis/python-bigtable that it doesn't failed since long time.

[tseaver]

The approach I outlined in #159 (using test_utils.retry.RetryInstanceState to do backoff when checking node counts) would fix this issue as well.