I remember once in an interview I was asked in how could I improve the system that there were developing in terms of security and performance.
The answer was quite fast, I would not reinvent the wheel but rather use Doctor Monkey from the Simian Army.
I was quite surprise that actually they did not know what was the Simian Army and how Netflix was using it to assure Amazon services and the cloud services were actually working. THEY DIDN’T KNOW! HOW COME RIGHT?
Basically the tool was I talking about use several external tools for health checks in the instances.
The tools are an analogy to a simian being released from a cage, this analogy is specially interesting as the Chaos Monkey, this tools is very interesting. It actually disable some random instances to verify how the system would react!!
Netflix thanks for that!!