This project is aimed at developing a decision-theoretic technique and a set of programming tools for performance evaluation and control of complex intelligent systems. The goal is to automate three interrelated aspects of the performance of complex intelligent systems, namely, performance analysis, optimal integration of components, and optimal monitoring of the system. The key advantage of our approach is that performance evaluation does not end with summarizing the performance of each module or of the complete system. We have shown that when performance information is represented using conditional performance profiles, it can be used to introduce and exploit run-time tradeoffs between computational resources and overall performance. Using advanced monitoring strategies, performance information can be used to control and optimize the operation of complex systems.

Performance evaluation of intelligent real-time systems is hard since the value of the information that they produce depends on two factors: (1) The objective quality of the solution to the initial problem conditions that can be measured by its certainty, accuracy or specificity; and (2) The time at which the solution becomes available and the extent of change in the environment that may reduce its applicability. In such areas as situation assessment, information gathering, and automated diagnosis and repair, the system must trade-off decision quality for computational costs. Therefore such systems must be evaluated with respect to a particular run-time monitoring scheme that can exploit the tradeoff between quality and time to maximize overall performance. Using recent techniques in the area of anytime computation, we are developing a new approach to evaluate intelligent systems whose advantages include modularity, clear analytical foundation, robustness and scalability.

Related Publications

• Handling Duration Uncertainty in Meta-Level Control of Progressive Processing
  A. I. Mouaddib and S. Zilberstein. Proceedings of the 15th International Joint Conference on Artificial Intelligence, Nagoya, Japan, 1997.

• An Anytime Approach to Analyzing Software Systems
  D. Rubenstein, L. Osterweil, and S. Zilberstein. Proceedings of the 10th International Florida Artificial Intelligence Research Symposium, pp. 386-391, Daytona Beach, Florida, 1997.

• Using Anytime Algorithms in Intelligent Systems
  S. Zilberstein. AI Magazine, 17(3):73-83, 1996.

• Monitoring the Progress of Anytime Problem-Solving
  E. A. Hansen and S. Zilberstein. Proceedings of the 13th National Conference on Artificial Intelligence, pp. 1229-1234, Portland, Oregon, 1996.

• Anytime Algorithm Development Tools
  J. Grass and S. Zilberstein. In M. Pittarelli (Ed.), SIGART Bulletin Special Issue on Anytime Algorithms and Deliberation Scheduling, 7(2):20-27, 1996.

• Optimal Composition of Real-Time Systems
  S. Zilberstein and S. J. Russell. Artificial Intelligence, 82(1-2):181-213, 1996.

• Optimizing Decision Quality with Contract Algorithms
  S. Zilberstein. Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 1576-1582, Montreal, Canada, 1995.

• On the Utility of Planning
  S. Zilberstein. In M. Pollack (Ed.), SIGART Bulletin Special Issue on Evaluating Plans, Planners, and Planning Systems, 6(1):42-47, 1995.

• Meta-level Control of Approximate Reasoning: A Decision Theoretic Approach
  S. Zilberstein. Proceedings of the Eighth International Symposium on Methodologies for Intelligent Systems, pp. 114-123, Charlotte, North Carolina, 1994.

• Operational Rationality through Compilation of Anytime Algorithms
  S. Zilberstein. Ph.D. dissertation, Computer Science Division, University of California at Berkeley, 1993.