[Back]The Checklist Approach The Checklist Approach
Typically, checklists are organized into categories of criteria. Most checklists have five such categories: Tasks, Domain Expert, Payoff, User and Management. Checklist have a history of focusing primarily on technical aspects such as, the nature of the task, desirable and undesirable characteristics of the task and the sources and availability of domain (business) expertise. However, the practical factors such as organizational setting, management support, potential payoff, designer competence and user concerns often determine a project’s success.
The checklist described here not only addresses technical issues in great detail, but equally emphasizes the practical aspects of evaluating a potential application. These checklist expand the criteria in the five categories and includes a sixth category - the system designer. The checklist developed here was first seen in the February 1991 issue of A. I. Expert magazine. This criteria is based on expert system applications developed at the Internal Revenue Service (IRS) AI Lab and interviews conducted with expert system project developers in government, business and academia. Appropriate weighting of individual criteria and categories greatly improves the accuracy of the evaluation process.
The six (6) categories in the checklist and their relative weights, are task (25), payoff (20), customer management (20), system designer (15), domain expert (10) and user (10). Checklist 1 through 6 at the end of this article, summarize the relevant criteria for each category along with weights (the numbers in parentheses) that reflect the relative importance of each criterion.
Two categories, task and payoff are essential for success, while the other four categories contribute to lesser degrees. Thus for an application to be considered promising, it must score at least 50 % in the task category (13 points) and payoff category (10 points). If either scores less than 50 %, you should choose another task. An application must also score at least 50 % overall (50 points) to qualify; scores below 50 % in any of the later four categories indicates potential difficulties. Keep in mind that these 50 % pass/fail scores are simply rough measures of success.
Desirable Task (Checklist #1)
The nature of the task is the most critical category for assessing the value of applying expert system technology. Not only must the task be technically feasible, but use of expert systems must also be necessary and appropriate. The task should involve significant symbolic processing, complexity, judgment, and uncertainty, and it should not be solvable using conventional programming methods or too difficult requiring AI applied research. If the proposed application scores less than 50 % in the task category, either the task can be solved using conventional programming techniques, or the task is inappropriate or too difficult for expert system technology.
Desirable Task
(2) 1. Task is primarily cognitive, requiring analysis, synthesis, or decision making rather than perception or action
(2) 2. Involves primarily symbolic knowledge and reasoning
(2) 3. Is complex, involving many parameters
(1) 4. Involves chains of reasoning on multiple levels of knowledge
(2) 5. Uses heuristics or rules of thumb and requires judgment or reasoning about subjective factors
(1) 6. Can’t be solved using conventional computing methods
(2) 7. Often must be solved with incomplete or inaccurate data
(2) 8. Often requires explanation, justification of results, or reasoning
(1) 9. Is at an intermediate stage of knowledge formalization that uses heuristics and classification rather than search or algorithms.
(1) 10. Task knowledge is confined to a narrow domain
(1) 11. Task knowledge is stable
(1) 12. Incremental progress is possible; task can be subdivided
(1) 13. Doesn’t require reasoning about time or space
(1) 14. Isn’t natural-language intensive
(1) 15. Requires little or no common sense or general-world knowledge
(1) 16. Doesn’t require the system to learn from experience
(1) 17. Is similar to one in an existing expert system
(1) 18. Data and case studies are available
(1) 19. System performance can be accurately and easily measured
___ =Points Earned = ____ score
25 Points Possible
Payoff (Checklist #2)
The need for the system must be translatable into benefits relevant to the user management. Expert systems can provide several benefits which include reduced costs, improved quality, increased revenues, captured expertise, easily distributed expertise, raised barriers to market entry and a training effect on users. Compared to conventional data processing systems, expert systems often incur additional expenses, the extent of which depends on the complexity of the proposed application. These additional expenses include: additional hardware, software, development cost of eliciting knowledge from domain experts and system maintenance costs.
If score for the payoff category is less than 50 % (10 points), another task should probably be chosen. If the benefits are too low (criteria 1-7), the task probably does not provide sufficient value to the user organization. If the system costs too much to deliver (criteria 8-11), either the system needs to be developed in a simpler shell, custom coded in a language such as “C” or reduced in functionality.
Payoff
(4) 1. Senior management is willing to commit significant funding
(2) 2. Willing to commit significant staffing resources to develop and deploy the system
(1) 3. Supportive, enthusiastic, and has appointed a project coordinator
(1) 4. Receptive to innovation and new technologies
(3) 5. Site management has committed staffing resources for acquiring knowledge, preparing test cases and validating the system
(1) 6. Is supportive, enthusiastic, and has appointed a project contact from management
(2) 7. Management accepts primary responsibility for maintaining the installed system
(1) 8. Use of he system will not be politically sensitive
(1) 9. Either system requires minimal changes to existing procedures, or if it requires substantial changes, management agrees to changes and recognizes the need for user training on the system
(2) 10. Management understands that estimates for resources and deadlines are difficult to estimate and probably will not be met
(2) 11. Management realizes the system will make mistakes and may perform no better than a moderately proficient user
____ = Points earned = ____ score
20 Points possible
Customer Management (Checklist #3)
In most checklist the management category has often been overlooked. This checklist addresses the issues for lower and upper level executives. I have found, from speaking with management at several major corporations, that lower level managers have been willing to support expert system projects by supplying domain experts, serving as test sites for expert system prototypes and by preparing test cases and transcribing data needed for system testing. However, without the support of upper level executives, expert system applications face an uphill battle for implementation. Although technical feasibility may have been demonstrated for many expert system projects, funding of hardware purchases for deployment has been difficult to obtain. Thus, further development and deployment of some applications have been delayed significantly or suspended indefinitely.
Customer Management
(2) 1. System would significantly increase revenues
(2) 2. Reduce costs
(2) 3. Improves quality
(2) 4. Capture undocumented expertise or expertise that is perishable or in short supply
(1) 5. Distribute accessible expertise to novice users
(1) 6. Provide training effect on users through usage
(1) 7. Raise barriers to future market entrants
(1) 8. Require no or minimal more data entry than current system
(2) 9. Be developed using commercial shells; little customized coding needed - Will become quite costly if customization is needed
(1) 10. System maintenance would be low
(2) 11. System would be delivered on an affordable desktop computer or workstation or using existing hardware and platforms
(1) 12. Could be phased in; partial completion would still be useful
(2) 13. Would result in benefit/cost ratio of at least 10:1- Ideal
____ = Points earned = _____ score
20 Points possible
System Designer (Checklist #4)
One aspect of project selection that has been ignored is desirable traits for the system designer. The Knowledge Engineer or AI consultant may also be considered as system designers. The technical success of the system depends chiefly on the designer and secondarily on the domain expert. If the designer has an insufficient background in AI or lacks familiarity with the expert system software that will be used in developing the application, the project may fail. Major project delays and revisions often result from low scores in this category. Although not as critical as the task, payoff, or management categories, a score of less than 50 % (7 points) can pinpoint potential problems.
System Designer
(2) 1. Designer has experience in designing and developing expert systems
(1) 2. Knows how to use a development tool appropriate for the system and has used the chosen tool or shell
(1) 3. Is experienced in acquiring and eliciting knowledge from written sources and domain experts
(2) 4. Has AI background to recognize which techniques will be useful in developing the system
(1) 5. Understands cognitive psychology
(2) 6. Has managed and developed more traditional computing applications - Able to understand the impact the expert system will have on the overall environment
(2) 7. Is knowledgeable or an expert in the domain - Knows the business problem to be solved
(1) 8. Has hardware and software available for development
(2) 9. Can commit at least six months of full-time effort for developing testing and implementing the system
____ = Points earned = _____ score
15 Points possible
Domain (Business) Expert (Checklist #5)
The system must be constructed from a source of expertise, which could consist of formal, written knowledge or informal heuristics (such as rules of thumb) not documented elsewhere. Heuristics must be obtained by interviewing domain experts or observing their actions. If substantial domain knowledge is lacking, methods other than expert systems, a more conventional approach will be appropriate. Therefore, heuristic expertise is crucial to the success of expert systems.
Domain (Business) Expert
(2) 1. Recognized experts exist
(1) 2. Expert performance is probably better than that of amateurs
(1) 3. Task is routinely taught to beginners
(1) 4. Experts are accessible for extended periods of time
(1) 5. Are cooperative
(1) 6. Communicate well
(2) 7. Available to develop test cases and help evaluate the system
___ = Points earned = ____ score
10 Points possible
User (Checklist #6)
Considering the needs and preferences of users is essential to system success. Users should feel a strong need for assistance, yet feel comfortable with the system. Thus, the role played by the expert system is crucial to user acceptance. Most often, systems should be designed as assistants so the user remains in control of the task. If the system automates the task, it is essential that users have other more desirable tasks to replace the automated task. Users and experts will resist all proposed expert system applications if users are displaced, replaced, or loose prestige.
The User category represents 10 out of the 100 points possible on all the checklist categories combined. Although not as critical as the other categories, a score of less than 50 % (5 points) can foreshadow difficulties during and after implementation.
User
(2) 1. Users feel a strong need for the system
(2) 2. Won’t be deskilled or unfavorably displaced as a result of implementing the system
(1) 3. Want to be involved in the system’s development
(2) 4. Don’t have unrealistically high expectations
(3) 5. Have roughly the same level of expertise
___ = Points earned = _____ score
10 Points possible
[Back]