OS-Faults + RallyΒΆ

Combination of OS-Faults and Rally gives a powerful tool to test OpenStack high availability and fail-over under the load.

Fault injection is implemented with help of Rally Fault Injection Hook. Following is an example of Rally scenario performing Keystone authentication with restart of one of Memcached services:

---
  Authenticate.keystone:
    -
      runner:
        type: "constant_for_duration"
        duration: 30
        concurrency: 5
      context:
        users:
          tenants: 1
          users_per_tenant: 1
      hooks:
        -
          name: fault_injection
          args:
            action: restart memcached service on one node
          trigger:
            name: event
            args:
              unit: iteration
              at: [100]

The moment of fault injection can be specified as iteration number or in time relative to the beginning of the test:

trigger:
  name: event
  args:
    unit: time
    at: [10]

Parameter action contains fault specification in human-friendly format, see Human API for details.

More on reliability testing of OpenStack: