OS-Faults + Rally¶
Combination of OS-Faults and Rally gives a powerful tool to test OpenStack high availability and fail-over under the load.
Fault injection is implemented with help of Rally Fault Injection Hook. Following is an example of Rally scenario performing Keystone authentication with restart of one of Memcached services:
--- Authenticate.keystone: - runner: type: "constant_for_duration" duration: 30 concurrency: 5 context: users: tenants: 1 users_per_tenant: 1 hooks: - name: fault_injection args: action: restart memcached service on one node trigger: name: event args: unit: iteration at: 
The moment of fault injection can be specified as iteration number or in time relative to the beginning of the test:
trigger: name: event args: unit: time at: 
Parameter action contains fault specification in human-friendly format, see Human API for details.
More on reliability testing of OpenStack:
- Reliability Test Plan in OpenStack performance documentation
- Keystone authentication with restart memcached report collected in OpenStack deployed by Fuel
- Introduction into reliability metrics video cast