%0 Generic %1 talk-Eismann-HotCloudPerf2018 %A Eismann, Simon %D 2018 %K DML Statistical_estimation_and_machine_learning se2_talk Performance Simulation t_talk Prediction descartes Automated_model_learning Meta-models Analytical_and_simulation-based_analysis %T Flexible Performance Predictions at Run-time %X Performance modeling is a powerful tool to predict the performance of a software system, especially for complex software systems deployed in cloud enviroments. At run-time, these predictions can be used for a variety of purposes. A performance engineer can analyze them to find potential bottlenecks or investigate solutions to an existing performance issue. Alternatively, a design space exploration tool can utilize performance predictions to find an optimal system configuration for a given scenario. A relatively novel use case for performance predictions is proactive auto-scaling. Proactive auto-scaling uses load forecasts to predict load spikes and proactively adapt the system accordingly. Here performance predictions are required to find a suitable system configuration for the incoming load spike. However, all of these use cases have different requirements for the performance predictions in terms of time-to-result, accuracy, and the required metrics. For example during design space exploration accurate predictions of both utilization and response time are required, without major time limitations. An auto-scaler, on the other hand, requires only rough utilization predictions, but with strict time-to-result requirements. A performance engineer requires a balance of accuracy and time-to-result. He is however often only interested in specific metrics, not the performance of the whole system. While a vast number of performance prediction approaches has been proposed, there is as of yet no approach flexible enough to fit all use cases. In this talk, we present our vision for flexible performance predictions at run-time. We envision the usage of one model describing the architecture and performance relevant properties of the system. This model can then be transformed to a number of solution formalisms, such as Queuing Networks (QNs), Layered Queuing Networks (LQNs) or stochastic process models. For these solution formalisms, a series of solution approaches exist, ranging from simple approximations, analytical approaches to full-scale simulations. This ensures that for each use case a suitable solution approach is available without having to maintain multiple models of the system. However, manual selection requires expert knowledge and manual intervention. Therefore, we aim to design an intelligent selection algorithm. This algorithm selects a suitable solution approach for every performance query. We define a performance query as a collection of requested performance metrics, a lower limit for the prediction accuracy and an upper limit for the time-to-result. In order to select a suitable solution approach for a query we first filter out all approaches that are not capable of predicting the requested performance metrics. For example, many analytical approaches are not capable of predicting response time distributions. In a next step, the selection algorithm needs to predict the accuracy and time-to-result for every solution approach. To predict the accuracy we plan to combine expert knowledge about the solvers with smart transformations, that can estimate how well the initial model is represented in the transformation output. For each simplification a transformation makes a number of penalty points, depending the level of simplification is awarded. For example, if a transformation represents loops in a probabilistic fashion only two penalty points per loop would be awarded, while a transformation that ignores passive resources would award a larger number of penalty points for each passive resource in the original model. Additional penalty points are awarded in case the applied solution approach calculates approximations instead of exact results. These penalty scores are determined once using expert knowledge. The resulting penalty score represents an indicator for the expected accuracy of a solution approach. We plan to utilize machine learning approaches to estimate the time-to-result. In general, estimating the time-to-result for performance predictions is challenging. However, we do not need to predict the time-to-result for a previously unseen model. After the inception of the performance model, each solution approach can be benchmarked to generate training data. From this training data, the machine learning approaches can then estimate the time-to-result for a specific request. The accuracy and time-to-result estimations will enable the design of an automated selection algorithm, that selects a suitable solution approach for each performance query. This allows for flexible performance predictions using a single model.

@misc{talk-Eismann-HotCloudPerf2018, abstract = {Performance modeling is a powerful tool to predict the performance of a software system, especially for complex software systems deployed in cloud enviroments. At run-time, these predictions can be used for a variety of purposes. A performance engineer can analyze them to find potential bottlenecks or investigate solutions to an existing performance issue. Alternatively, a design space exploration tool can utilize performance predictions to find an optimal system configuration for a given scenario. A relatively novel use case for performance predictions is proactive auto-scaling. Proactive auto-scaling uses load forecasts to predict load spikes and proactively adapt the system accordingly. Here performance predictions are required to find a suitable system configuration for the incoming load spike. However, all of these use cases have different requirements for the performance predictions in terms of time-to-result, accuracy, and the required metrics. For example during design space exploration accurate predictions of both utilization and response time are required, without major time limitations. An auto-scaler, on the other hand, requires only rough utilization predictions, but with strict time-to-result requirements. A performance engineer requires a balance of accuracy and time-to-result. He is however often only interested in specific metrics, not the performance of the whole system. While a vast number of performance prediction approaches has been proposed, there is as of yet no approach flexible enough to fit all use cases. In this talk, we present our vision for flexible performance predictions at run-time. We envision the usage of one model describing the architecture and performance relevant properties of the system. This model can then be transformed to a number of solution formalisms, such as Queuing Networks (QNs), Layered Queuing Networks (LQNs) or stochastic process models. For these solution formalisms, a series of solution approaches exist, ranging from simple approximations, analytical approaches to full-scale simulations. This ensures that for each use case a suitable solution approach is available without having to maintain multiple models of the system. However, manual selection requires expert knowledge and manual intervention. Therefore, we aim to design an intelligent selection algorithm. This algorithm selects a suitable solution approach for every performance query. We define a performance query as a collection of requested performance metrics, a lower limit for the prediction accuracy and an upper limit for the time-to-result. In order to select a suitable solution approach for a query we first filter out all approaches that are not capable of predicting the requested performance metrics. For example, many analytical approaches are not capable of predicting response time distributions. In a next step, the selection algorithm needs to predict the accuracy and time-to-result for every solution approach. To predict the accuracy we plan to combine expert knowledge about the solvers with smart transformations, that can estimate how well the initial model is represented in the transformation output. For each simplification a transformation makes a number of penalty points, depending the level of simplification is awarded. For example, if a transformation represents loops in a probabilistic fashion only two penalty points per loop would be awarded, while a transformation that ignores passive resources would award a larger number of penalty points for each passive resource in the original model. Additional penalty points are awarded in case the applied solution approach calculates approximations instead of exact results. These penalty scores are determined once using expert knowledge. The resulting penalty score represents an indicator for the expected accuracy of a solution approach. We plan to utilize machine learning approaches to estimate the time-to-result. In general, estimating the time-to-result for performance predictions is challenging. However, we do not need to predict the time-to-result for a previously unseen model. After the inception of the performance model, each solution approach can be benchmarked to generate training data. From this training data, the machine learning approaches can then estimate the time-to-result for a specific request. The accuracy and time-to-result estimations will enable the design of an automated selection algorithm, that selects a suitable solution approach for each performance query. This allows for flexible performance predictions using a single model.}, added-at = {2020-04-06T11:29:57.000+0200}, author = {Eismann, Simon}, biburl = {https://www.bibsonomy.org/bibtex/2350aa52924153d374f79e8638eeedf8a/se-group}, howpublished = {HotCloudperf 2018}, interhash = {5d45798cfd7d81982e760d7c4942642c}, intrahash = {350aa52924153d374f79e8638eeedf8a}, keywords = {DML Statistical_estimation_and_machine_learning se2_talk Performance Simulation t_talk Prediction descartes Automated_model_learning Meta-models Analytical_and_simulation-based_analysis}, month = {April}, timestamp = {2020-10-05T16:26:45.000+0200}, title = {{Flexible Performance Predictions at Run-time}}, year = 2018 }