Online robustness testing of distributed embedded systems: An industrial approach

Having robust systems that behave properly even in presence of faults is becoming increasingly important. This is the case of the system we investigate in this paper, which is an embedded distributed system consisting of components that communicate with each other via messages exchange in the RBS (Radio Based Station) at Ericsson AB in Gothenburg, Sweden. Specifically, this paper describes a novel fault injection approach for testing the robustness of distributed embedded systems with very limited computation power. The new approach is inspired by Netflix's ChaosMonkey, a fault injection approach that has been developed for testing distributed systems hosted in the cloud. However, ChaosMonkey cannot be used in the context of RBS since the latter consists of small-embedded components with specific requirements of performance, programming language, and communication paradigm. This paper reports about the approach called Postmonkey we developed, illustrates the results of applying it to RBS, and discusses the potential of utilizing fault injection to test complex, embedded, and distributed systems. The approach and tool are now adopted by Ericsson.