Using stress test results to drive performance modeling: A case study in Gray-Box vendor analysis
Teletraffic Engineering in the Internet EraProceedings of the International Teletraffic Congress - ITC-I7, volume 4 of Teletraffic Science and Engineering, Elsevier, (2001)DOI: http://dx.doi.org/10.1016/S1388-3437(01)80191-8
Abstract
In this case study, we present the results of analysis of a Java-based vendor product that performs dynamic Web page construction in a distributed environment. Since the vendor's scripts are proprietary, we must rely on gray-box stress test results to identify performance bottlenecks and determine system capacity. We present a methodology for reverse-engineering stress test results to build performance models of the system internals, and use those models to see inside the box. This approach provided significant insights into the performance of the vendor's proprietary code, and afforded us better leverage in the vendor management process. In particular, the approach identified significant bottlenecks in the Java code that prevented the system from fully utilizing the hardware resources. The approach also exposed the system's behavior under overload, and revealed the need for overload controls. Based on our success, we recommend that stress testing be incorporated into the software development process, especially when development consists largely of integrating external vendor components. Furthermore, in the case of vendor-provided software, this approach offers perhaps the only way to see inside the code, and in the case of new programming languages such as Java, exposes anomalies that could not be predicted from past experience.