DOE-EM-STD-5505-96, Operations Assessments

TS NOT MEASUREMENT SENSITIVE DOE-EM-STD-5505-96 May 1996 DOE LIMITED STANDARD OPERATIONS ASSESSMENTS U.S. Department of Energy AREA OPER Washington, D.C. 20585 DISTRIBUTION STATEMENT A. Approved for public release; distribution is unlimited. WELCOME This Portable Document Format (PDF) file contains bookmarks, thumbnails, and hyperlinks to help you navigate through the document. All items listed in the Contents are linked to the corresponding sections. In addition, if you click on a section heading while you are reading the text, you will return to the Contents. Click on the DOE seal below to move to the Contents page. This document has been reproduced directly from the best available copy. Available to DOE and DOE contractors from the Office of Scientific and Technical Information, P.O. Box 62, Oak Ridge, TN 37831; (423) 576-8401. Available to the public from the U.S. Department of Commerce, Technology Administration, National Technical Information Service, Springfield, VA 22161; (703) 487-4650. Order No. DE96009462 DOE-EM-STD-5505-96 iii FOREWORD 1. This Department of Energy limited standard is approved for use by all DOE-EM field components. 2. Beneficial comments (recommendations, additions, deletions) and any pertinent data that may improve this document should be sent to the office of Nuclear Safety Policy (EH-31), U.S. Department of Energy, Washington, D.C. 20585, by letter or by sending the self- addressed Standardization Document Improvement Proposal (DOE F 1300.3) appearing at the end of this document. 3. This Department of Energy limited standard was prepared by the Environmental Management Office of Safety and Health (EM-4) with the assistance of Digital Systems Research, Inc. DOE-EM-STD-5505-96 iv Intentionally Blank CONTENTS Section Page DOE-EM-STD-5505-96 v 1. INTRODUCTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. DEFINITIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.2 Concern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.3 Finding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. USING THIS LIMITED TECHNICAL STANDARD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. OPERATIONS ASSESSMENT PROGRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4.1 Operations Assessment Program Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4.2 Operations Assessment Program Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.2.1 Operations assessments conducted in accordance with a documented schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.2.2 Documentation of programmatic breakdowns or widespread problems (concerns) supported by individual items that do not meet requirements (findings) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.2.3 Communication of concerns and findings to contractor management . . . . . 6

4.2.4 Follow-up to verify concerns are corrected . . . . . . . . . . . . . . . . . . . . . . . . 6 4.2.5 Utilization of assessment results as a factor in determining contractor monetary incentives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 5. PERFORMING AN OPERATIONS ASSESSMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 5.1 Operations Assessment Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 5.2 Assessment Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 5.2.1 Resource Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 CONTENTS Section Page DOE-EM-STD-5505-96 vi 5.3 Assessment Team Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.3.1 Team Leader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5.3.2 Team Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.4. Assessment Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.4.1 Pre-assessment Information Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.4.1.1 Graded-Approach Document . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.4.1.2 Activity-Specific Information . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.4.1.3 Prior Assessments, Reports, etc. . . . . . . . . . . . . . . . . . . . . . . . . 11 5.4.2 Assessment Plan Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.4.2.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.4.2.2 Content and Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.5 Activity Briefing and Tour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.6 Assessment Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.7 Assessment In-brief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.8 Assessment Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.8.1 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.8.1.1 What to Observe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.8.1.2 How to Observe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.8.1.3 Where and When to Observe . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.8.1.4 Stop Work Authority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.8.2 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.8.2.1 How to Prepare for Interviews . . . . . . . . . . . . . . . . . . . . . . . . . 18 5.8.2.2 Conducting Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5.8.2.3 Where and When to Conduct Interviews . . . . . . . . . . . . . . . . . . 20 5.8.3 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.8.3.1 What Documents to Review . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

CONTENTS Section Page DOE-EM-STD-5505-96 vii 5.8.3.2 Reviewing Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.8.3.3 When and Where to Review Documents . . . . . . . . . . . . . . . . . . 22 5.8.4. Pursuing leads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5.9 Daily Team Meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.9.1 Daily Meetings with Activity Management . . . . . . . . . . . . . . . . . . . . . . . . 27 5.10 Findings and Concerns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.11 Developing Concerns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.12 Report Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.12.1 Report Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.12.2 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.13 Assessment Out-brief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 6. GUIDELINES FOR ASSESSING OPERATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.1 Assessing Operations Organization and Administration (OPS O&A) . . . . . . . . . . . 31 6.1.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.1.2 Preparing to Assess OPS O&A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 6.1.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6.1.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 6.1.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 6.2 Assessing Shift Routines and Operating Practices . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.2.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.2.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.2.2 Preparing to Assess Shift Routines and Operating Practices . . . . . . . . . . . 37 6.2.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 6.2.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.2.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 CONTENTS Section Page DOE-EM-STD-5505-96 viii 6.3 Assessing Control Area Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.3.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.3.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.3.2 Preparing to Assess Control Area Activities . . . . . . . . . . . . . . . . . . . . . . . 44 6.3.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.3.4. Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

6.3.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.4 Assessing Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.4.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.4.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.4.2 Preparing to Assess Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.4.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.4.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.4.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.5 Assessing Control of On-Shift Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.5.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.5.2 Preparing to Assess On-Shift Training . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 6.5.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 6.5.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 6.5.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.6 Assessing Investigation of Abnormal Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.6.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.6.2 Preparing to Assess Investigation of Abnormal Events . . . . . . . . . . . . . . . 58 6.6.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 6.6.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.6.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 CONTENTS Section Page DOE-EM-STD-5505-96 ix 6.7 Assessing Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6.7.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6.7.2 Preparing to Assess Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.7.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.7.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 6.7.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 6.8 Assessing Control of Equipment and System Status . . . . . . . . . . . . . . . . . . . . . . . . 65 6.8.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.8.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.8.2 Preparing to Assess Control of Equipment and System Status . . . . . . . . . 65 6.8.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

6.8.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 6.8.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 6.9 Assessing Lockouts and Tagouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.9.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.9.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 6.9.2 Preparing to Assess Lockouts and Tagouts . . . . . . . . . . . . . . . . . . . . . . . 71 6.9.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 6.9.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.9.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.10 Assessing Independent Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.10.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.10.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.10.2 Preparing to Assess Independent Verification . . . . . . . . . . . . . . . . . . . . . 77 6.10.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 6.10.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 CONTENTS Section Page DOE-EM-STD-5505-96 x 6.10.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 6.11 Assessing Logkeeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.11.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.11.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.11.2 Preparing to Assess Logkeeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.11.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.11.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 6.11.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.12 Assessing Operations Turnover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.12.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.12.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.12.2 Preparing to Assess Operations Turnover . . . . . . . . . . . . . . . . . . . . . . . . 86 6.12.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.12.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.12.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.13 Assessing Operations Aspects of Facility Chemistry and Unique Processes. . . . . . 90

6.13.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.13.1.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.13.2 Preparing to Assess Operations Aspects of Facility Chemistry and Unique Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.13.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.13.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 6.13.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 6.14 Assessing Required Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 6.14.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 6.14.2 Preparing to Assess Required Reading . . . . . . . . . . . . . . . . . . . . . . . . . . 96 CONTENTS Section Page DOE-EM-STD-5505-96 xi 6.14.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.14.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.14.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.15 Assessing Timely Orders to Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.15.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.15.2 Preparing to Assess Timely Orders to Operators . . . . . . . . . . . . . . . . . 100 6.15.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 6.15.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 6.15.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 6.16 Assessing Operations Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.16.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.16.2 Preparing to Assess Operations Procedures. . . . . . . . . . . . . . . . . . . . . . 104 6.16.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.16.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 6.16.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 6.17 Assessing Operator Aid Postings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.17.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.17.2 Preparing to Assess Operator Aid Postings . . . . . . . . . . . . . . . . . . . . . 111 6.17.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 6.17.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6.17.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

6.18 Assessing Equipment and Piping Labeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 6.18.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 6.18.2 Preparing to Assess Equipment and Piping Labeling . . . . . . . . . . . . . . . 114 6.18.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 6.18.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 CONTENTS Section Page DOE-EM-STD-5505-96 xii 6.18.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 6.19 Assessing Operations-Related Radiological Control . . . . . . . . . . . . . . . . . . . . . . 117 6.19.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 6.19.2 Preparing to Assess Operations-Related Radiological Control . . . . . . . 118 6.19.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6.19.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 6.19.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.20 Assessing Operations-Related Training and Qualification . . . . . . . . . . . . . . . . . . 125 6.20.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.20.2 Preparing to Assess Operations-Related Training and Qualification . . . 125 6.20.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.20.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.20.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 6.21 Assessing Drills. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.21.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.21.2 Preparing to Assess Drills. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.21.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.21.4 Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6.21.5 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 7. APPENDIXES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Appendix A. Sample Operations Assessment Program Schedule . . . . . . . . . . . . . . . . 135 Appendix B. Operations Assessment Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 136 Appendix C. Sample Operations Assessment Preparation Checklist . . . . . . . . . . . . . . 137 Appendix D. Sample Graded-Approach Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Appendix E. Sample Operations Assessment Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Appendix F. Sample Schedule for an Operations Assessment . . . . . . . . . . . . . . . . . . . 144 CONTENTS Section Page DOE-EM-STD-5505-96 xiii

Appendix G. Sample Operations Assessment Report . . . . . . . . . . . . . . . . . . . . . . . . . 145 8. INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 DOE-EM-STD-5505-96 xiv Intentionally Blank DOE-EM-STD-5505-96 1 1. INTRODUCTION Conduct of operations includes those attitudes, processes, and precautions taken in the interest of safety. The exact features of a system of conduct of operations may be different at different activities. The common feature is a formality of operations which will vary in form and degree depending on the conditions and hazards of the activity. The most intensive application of conduct of operations would be found at the most hazardous activities subject to repetitive types of evolutions. Operational formality is a structured and systematic way of performing work. It is not simply a listing of functional areas, but rather a mind set, or way of doing business. A comprehensive program of operational formality should provide detailed guidance for performing essential elements of operations, such as: shift routines, communications, control of equipment, lockouts and tagouts, logkeeping, operations procedures, and equipment labeling. DOE Directives which require implementation of conduct of operations throughout the DOE complex are based on well-developed industrial operations practices. The guidelines in those DOE Directives were written to be flexible so that conduct of operations could be implemented at any Environmental Management (EM) activity. To ensure that DOE contractors have appropriately implemented conduct of operations, contractor operations must be assessed periodically. The purpose of this Department of Energy (DOE) limited technical standard is to (1) provide DOE Field Element assessors with a guide for conducting operations assessments, and (2) provide DOE Field Element managers with the criteria of the EM Operations Assessment Program. DOE-EM-STD-5505-96 2 This limited technical standard does not establish any new requirements, but rather incorporates and supersedes the existing criteria from: � Assistant Secretary of Energy for Environmental Restoration and Waste Management “Operations Assessment of Environmental Restoration and Waste Management (EM) Activities” memorandum of May 19, 1992; � Assistant Secretary of Energy for Environmental Restoration and Waste Management “Implementation of the Guidelines for the Conduct of Operations at DOE Facilities” memorandum of July 20, 1992; � Assistant Secretary of Energy for Environmental Restoration and Waste Management “Conduct of Operations at EM Support Services Facilities” memorandum of July 20, 1993; and � Assistant Secretary of Energy for Environmental Management “Operations Assessment Program Reviews” memorandum of July 7, 1994. This limited standard also updates information contained in the Operations Assessment Field Handbook dated June 24, 1994. Therefore, this limited standard supersedes the above noted memoranda and the Operations Assessment Field Handbook which can be discarded at this time. 2. DEFINITIONS 2.1 Activity. Any operational process, system, structure, equipment, or group that fulfills a programmatic purpose. Examples include, but are not limited to, storage areas, radioactive waste disposal and processing systems, burial grounds, environmental restoration projects, tank farms,

characterization and decontamination projects, and analytical laboratories. 2.2 Concern. A determination of a programmatic breakdown or widespread problem supported by one or more findings. DOE-EM-STD-5505-96 3 2.3 Finding. An individual item which does not meet a requirement. 3. USING THIS LIMITED TECHNICAL STANDARD The limited technical standard is divided into sections that describe: � The EM Operations Assessment Program (Section 4), � The overall operations assessment process including the application of assessment techniques, and how to conduct and report the results of an assessment (Section 5), � The process and techniques to be used to assess each element of conduct of operations, operations-related radiological control, activity drills, and operations-related training and qualification (Section 6). Section 4 which provides the criteria of the EM Operations Assessment Program is intended to be used by DOE Field Element managers who are responsible for EM activities. These managers should also be familiar with section 5 to ensure that personnel are applying effective techniques when planning and conducting operations assessments. Sections 5 and 6 are intended to be used as guidance when planning, conducting, and reporting operations assessments at EM funded activities. This limited technical standard provides a set of tools for field assessors conducting operations assessments. Although Sections 6.1 to 6.21 provide examples of how to assess specific areas, the general techniques of operations assessments (Section 5) may be applied to other areas of health and safety (e.g. fire protection, criticality safety , quality assurance, occupational safety, etc.). 4. OPERATIONS ASSESSMENT PROGRAM 4.1 Operations Assessment Program Objective. The objective of the EM Operations Assessment Program is to provide for consistent and standard assessments of activities, in order DOE-EM-STD-5505-96 4 to: � Establish responsibility and accountability for assessment of contractor operations and for corrective action as required; � Ensure that hazardous facilities and activities are competently operated in a manner providing proper assurance of worker and equipment safety, and protection of the environment; � Enable management to determine the operational condition of all activities on a comparable basis; � Provide the basis for management actions to improve activity operations; � Provide an objective methodology to track performance of an activity in conduct of operations; and � Provide an objective basis for contractor monetary incentive determination and for assignment of penalties, when appropriate. 4.2 Operations Assessment Program Criteria. DOE Field Element managers are responsible for carrying out the EM Operations Assessment Program which shall include: � Operations assessments conducted in accordance with a documented schedule � Documentation of programmatic breakdowns or widespread problems (concerns) supported by individual items that do not meet requirements (findings) � Communication of concerns and findings to contractor management � Follow-up to verify concerns are corrected � Utilization of operations assessment results as a factor in determining contractor monetary incentives 4.2.1 Operations assessments conducted in accordance with a documented schedule. Field Element managers shall ensure full operations assessments are conducted at least once every two

years and partial operations assessments are conducted at least once every six months for each EM activity. A full operations assessment shall cover all applicable elements of conduct of DOE-EM-STD-5505-96 5 operations, operations-related radiological control, activity drills, and operations-related training and qualification. A partial operations assessment shall cover one or more applicable elements of conduct of operations, operations-related radiological control, activity drills, or operations-related training and qualification. Since other DOE Directives require assessments of areas besides conduct of operations, operations assessments may be performed in conjunction with these other assessments. Field Element managers shall maintain a documented operations assessment schedule which covers a two year period. Field Element managers should update the schedules semiannually and provide copies of the schedules to DOE Headquarters Program Managers and the Office of Safety and Health (EM-4). The sample operations assessment schedule in Appendix A illustrates an acceptable approach to scheduling operations assessments at the required periodicity. Scheduling operations assessments at the wastewater treatment plant and hazardous waste storage area is rather straightforward because operations always take place at the same physical location. However, scheduling operations assessments of the decontamination & decommissioning activity and pilot well monitoring activity are more challenging because these operations may occur at various locations across the site over the course of a year. By treating the decontamination & decommissioning activity or the pilot well monitoring activity as a discrete operational process that fulfills a programmatic purpose, operations assessments may be scheduled on the respective activities regardless of the location of their operations. In this manner, the intent of the operations assessment program will be fulfilled. Field Element managers shall also ensure that the assessment teams are conducting operations assessments through the use of observations, interviews, and document reviews. Sections 5 and 6 describe how to perform an operations assessment and provide guidelines for assessing the elements of conduct of operations, operations-related radiological control, drills, and operations- DOE-EM-STD-5505-96 6 related training and qualification, respectively. 4.2.2 Documentation of programmatic breakdowns or widespread problems (concerns) supported by individual items that do not meet requirements (findings). Field Element managers shall ensure that the concerns resulting from operations assessments are documented by the assessment team and supported by one or more findings. Identification of concerns will require that assessment teams go beyond merely determining deviations from requirements to understanding the status of the programs. Therefore, assessment teams should aggressively evaluate the findings for underlying concerns. The process of developing concerns is described in section 5.11. 4.2.3 Communication of concerns and findings to contractor management. Field Element managers shall communicate the concerns and findings to contractor management. The Field Element managers should ensure that the contractor receives the results of the assessments in a timely manner so that prompt corrective action may be taken. Field Element managers may

deliver the results of an assessment to contractor management either verbally, in writing, or both. 4.2.4 Follow-up to verify concerns are corrected. After completion of an operations assessment, Field Element managers shall follow up to verify concerns are corrected. Field Element managers may verify that concerns are effectively corrected during subsequent assessments. 4.2.5 Utilization of assessment results as a factor in determining contractor monetary incentives. Field Element managers shall utilize the results of operations assessments as a factor in determining contractor monetary incentives. When determining contractor monetary incentives, Field Element managers should consider the severity of concerns, completion of corrective actions, and overall contractor progress. DOE-EM-STD-5505-96 7 5. PERFORMING AN OPERATIONS ASSESSMENT 5.1 Operations Assessment Overview. An operations assessment analyzes and evaluates operations through the use of observations, interviews, and document reviews. The methodology used for this type of assessment is characterized by the following general process: � Assessment planning � Observations of routine operations, maintenance, special activities, and activity drills � Interviews with operations, maintenance, training, and radiological control organization personnel � Review of activity policies and procedures, logs, round/tour inspection sheets, radiological control records, etc. � Comparing the information gathered by the techniques above with the requirements contained in activity procedures, policies, and applicable DOE Directives to determine where deviations from requirements appear to exist � Further investigation using additional observations, interviews, and document reviews to clarify these deviations from requirements and determine if programmatic breakdowns or widespread problems exist The operations assessment process does not rely on formulating checklists to ensure that every aspect of activity operations is evaluated against applicable requirements; rather, it provides a methodology to sample a wide variety of operations to develop a picture of how safely and efficiently the activity is being operated. A flow diagram that depicts the process of developing findings and concerns is contained in Appendix B. 5.2 Assessment Preparation. Early and consistent communications between the Operations Office and the activity to be assessed are vital to a successful assessment. The activity to be assessed should designate a point of contact to coordinate with the Operations Office in preparing for the assessment. A checklist that will aid the Operations Office when making DOE-EM-STD-5505-96 8 arrangements for an assessment is contained in Appendix C. 5.2.1 Resource Requirements. There are specific resource requirements necessary to support an assessment which should be coordinated through the site point of contact. These resources include a conference room and reference material. The conference room should provide enough space for scheduled interviews, document reviews, and daily team meetings. If possible, the conference room should be within walking distance from the activity being assessed to facilitate the daily assessment team meetings and debriefs. Upon arrival at the activity, the assessment team should verify that all support documentation requested for the assessment is available and staged for the assessment team's use. If not readily

available, the location of the required documents should be determined. 5.3 Assessment Team Composition. Early identification of the assessment team leader and assessment team coordinator is important. The coordinator should arrange site-specific training (if necessary), take care of security requirements, and handle other assessment preparations. Assessment team members should be selected based on their technical ability and expertise in specific functional areas of the activity to be assessed, as well as assessment experience. Assessment teams should be comprised of personnel such as facility representatives, project managers, and technical experts. 5.3.1 Team Leader. The operations assessment team leader has responsibility for the overall performance of the assessment team. The team leader also serves as the spokesperson for any communication between the assessment team and management. During the preparation for and performance of an assessment, the team leader is the focal point for communicating observations, findings, and concerns to management. Assessment team leaders should be federal employees. The team leader must be capable of representing DOE to senior contractor management in a creditable way. Each field office should determine the training and DOE-EM-STD-5505-96 9 qualification required for team leaders. Prior to an assessment, the team leader should identify the members of the assessment team and prepare an assessment plan. Assessment plan development is discussed in section 5.4. 5.3.2 Team Members. The entire assessment team should be identified in advance of an assessment to allow adequate preparation and to enable the team to make the most efficient use of time spent on the site. Conduct of operations assessable areas may be logically grouped and assigned to team members. For example, one team member may be responsible for assessing Shift Routines and Operating Practices, Control Area Activities, Operations Turnover, and Logkeeping, while another team member may focus on Lockouts and Tagouts, Independent Verification, and Control of Equipment and System Status. Team members should inform the team leader of the day-to-day assessment activities and of any difficulties in performing the assessment. Each field office should determine the training and qualification required for Team Members. 5.4. Assessment Planning. Research and planning prior to each assessment improves efficiency and produces the most effective use of limited assessment time and resources. Preparation prior to the visit will identify potential performance or technical problems, and it will also result in an assessment plan focused on the most important areas of activity operations. Assessment planning involves two steps: a pre-assessment information review and the development of an assessment plan. 5.4.1 Pre-assessment Information Review. The documents to be reviewed prior to the start of the assessment can be categorized generally as either documents that describe how the activity is designed and operated or documents that form the basis for DOE acceptance of operation. The documents listed in sections 5.4.1.2 and 5.4.1.3 may not be all inclusive. Assessors must decide which documents should be reviewed during assessment planning based on the scope of the assessment and the operations performed by the activity. DOE-EM-STD-5505-96 10 5.4.1.1 Graded-Approach Document. A good way to start the information review process is

to review the activity-specific graded-approach document (plan, matrix, etc.) for conduct of operations. The graded-approach document is approved by DOE and is used by activity management to document conformance with requirements. The graded-approach document indicates whether a specific guideline applies to an activity, tells where and how each of the elements of conduct of operations is applied within the contractor's existing policies and procedures, and identifies any deviations or exemptions from the guidelines. In other words, an approved graded-approach document is an agreement between DOE and the contractor that defines the basis for activity operations and provides the requirements to which the activity will be assessed. Appendix D contains a sample graded-approach matrix. During the pre-assessment information review, assessors should review the graded-approach document for conduct of operations and determine which guidelines apply to the activity and which activity policies and procedures implement these requirements. Assessors should then review these policies and procedures to familiarize themselves with how the guidelines are applied at the activity. Assessors will normally evaluate operations against activity-specific procedures. If there are no activity procedures, assessors should use site procedures. If site procedures are not available, the assessment should be based directly on the guidelines of applicable Directives and Regulations related to conduct of operations. Also, if there is no approved graded-approach document, then assessors should evaluate operations against all applicable guidelines for conduct of operations. 5.4.1.2 Activity-Specific Information. In planning for an assessment, it is important that the assessment team be familiar with the activity management structure, the functions performed at the activity, the processes and hardware, and the importance and function of critical systems. Typically, the following documents provide this information: � Safety Analysis Reports DOE-EM-STD-5505-96 11 � Key administrative procedures � Organizational charts � Operational Safety Requirements/Technical Safety Requirements � Activity layout diagrams � System drawings - Wiring Diagrams - Schematics -Piping & Instrument Drawings � Lists of activity maintenance requests � Requests for Action � Order Compliance Schedule Agreements � Activity mission statement � Basis for Interim Operations � Health and Safety Plans � Operations Procedures � Quality Assurance Plan � Emergency Plan 5.4.1.3 Prior Assessments, Reports, etc. Of equal importance to understanding activity operations is the historical performance of the activity. Emphasis should be placed on the identification of repetitive or chronic performance problems, events, equipment failures, or potential or previous programmatic deficiencies. Documents typically containing historical performance information include the following: � Occurrence reports � Performance indicators and trends � Self-assessment reports � External assessment reports � Safety issues previously identified DOE-EM-STD-5505-96 12 � Technical Safety Appraisals � Tiger Team assessment reports � Award fee appraisals By reviewing the above documents the assessment team can focus the assessment plan on the areas of interest at the activity and identify the specific documents to review, personnel to interview, and evolutions to observe.

5.4.2 Assessment Plan Development. Since an assessment is a "sampling" exercise, assessment team members must plan how to start the assessment and target key areas to look at, bearing in mind that following leads or observations will take up a large portion of assessment time. In preparing for an assessment, team members develop an assessment plan. The development of this plan is an individualized process in that the formality of the plan will vary from team to team, and some assessors will need to develop more detailed plans than others. Depending on the level of knowledge, skill, and style of the assessor, varying degrees of planning are necessary. It is important that leads developed during the assessment, not the plan, drive the conduct of the assessment. This plan functions primarily as a list of starting points, not as a checklist, and must be flexible enough to respond to unique or non-routine operations at the site. 5.4.2.1 Approach. The assessment team should develop a plan that lists each area that has been chosen by the field element manager and the approach to be used for each area. The general process for developing the plan involves three steps: (1) The team leader should assign the chosen assessment areas to the assessment team members; (2) Assessors should identify the initial observations, interviews, and document reviews necessary to assess the guidelines and concepts for their assigned element of conduct of operations, operations-related radiological control, activity drills and operations-related training and qualification; and (3) Specific DOE-EM-STD-5505-96 13 interview questions may be developed by assessors for each area based on the level of knowledge, skill, and style of the assessor. 5.4.2.2 Content and Format. Appendix E contains a sample assessment plan. The assessment plan is detailed and establishes the objectives, criteria, observations, interviews, and document reviews that will be conducted to begin the assessment. Furthermore, a great deal of time and effort should not be expended in developing the plan. Normally, an adequate assessment plan can be developed in 30 minutes to two hours. 5.5 Activity Briefing and Tour. Introductory and safety briefs from the DOE Facility Representative and activity management should be required by the assessment team. As a minimum, these briefs should cover the graded-approach document for conduct of operations, a description of restricted or hazardous areas, and the status of any ongoing operations. Arranging a brief tour of the activity for those team members unfamiliar with the activity can also be beneficial. In addition to familiarizing the team members with the general layout of the activity, the tour provides orientation for security access, radiological protection requirements, and communications procedures. When possible, the activity tour should occur prior to the beginning of the assessment to allow for optimum use of time while actually conducting the assessment. 5.6 Assessment Schedule. The final step in the assessment preparation phase is the development of an assessment schedule. This schedule is developed by the assessment team, after the assessment plan has been completed, to inform activity management of the key interviews, observations, and drills that the team requires to conduct the assessment. The activity plan of the day and plan of the week are useful tools in developing a schedule for

observable operational and maintenance-related events and activities. This schedule of key events should indicate the time of the interview or observation, the person being interviewed or event being observed, and the team member requesting the interview or observation. The DOE-EM-STD-5505-96 14 schedule should be provided to activity management at the in-brief for approval and is subject to change and revision as the assessment progresses. A sample assessment schedule is provided in Appendix F. 5.7 Assessment In-brief. A formal activity in-brief by the DOE Facility Representative, Activity Manager, and team leader should start each assessment. All assessment team members, contractor management, and available activity employees should attend this meeting. The meeting should be lead by the team leader and generally performs the following functions: � Provides an opportunity for contractor management to describe the activity, operating status, and programs related to the assessment. The status of any formal investigations and corrective actions for occurrences should also be described. � Allows the team leader to describe the overall objective of the assessment and the concept that the assessment is a snapshot of the activity operations and procedures in place. As a result, draft procedures and other planned activities not currently in place will not be assessed or commented on in the assessment report. � Allows the team leader to outline the assessment plan and schedule of observations, interviews, and document reviews. � Allows the team leader to emphasize the working relationship between team members and activity personnel. Specifically, team members are not part of the activity management, and as such, team members should not direct any actions of the activity. � Allows the team leader to address stop work authority in the context of the Operations Office policy. � Allows team members to be introduced to activity personnel and functional area counterparts. The ten to fifteen minutes spent in an entrance meeting can save many hours of work and prevent misunderstandings between assessment team members and activity personnel. DOE-EM-STD-5505-96 15 5.8 Assessment Techniques In order to identify and validate deviations from requirements and determine if programmatic breakdowns or widespread problems exist, an assessor must develop leads through the use of observations, interviews, and document reviews and pursue leads to obtain more detailed information. The use of these techniques is described in the following sections. 5.8.1 Observation. Observation is usually the most effective technique used during an operations assessment. Observing activities as they are performed provides information directly related to the effectiveness of operations. Observations should be combined with interviews and document reviews to paint an accurate picture of activity operations. 5.8.1.1 What to Observe. In planning for the assessment, determine what initial observations should be conducted for each area of responsibility. Some activities that can be observed include: � Management control, communication, and command; � Personnel using procedures; � Personnel following rules, obeying signs, using safety equipment; � Operators performing inspection tours; � Control area activities including response to indications, access control, monitoring of control panels, control of routine operations and special activities;

� Testing of emergency communication systems and the use of communications equipment such as radios; � On-shift training during routine operations, special activities, and drills; � Authorization, placement, verification, and removal of lockouts and tagouts during maintenance operations; � Independent verification of component and system alignment before start up or testing activities; DOE-EM-STD-5505-96 16 � Shift turnover between operators, maintenance personnel, shift supervisors or foremen, and personnel in the control area; � The recording of data in logbooks or on round sheets by operators, maintenance personnel, control area operators, and shift supervisors or foremen; � Process water sampling and analysis; � The control and use of activity procedures during routine operations, activities, emergency situations, and drills; � Placement, control, and removal of jumpers and temporary equipment in use at the activity; � Activity drills and exercises conducted during the assessment; � Operator aids posted throughout the activity; � Number and reason alarms actuated; � The labeling of equipment and systems in the activity. 5.8.1.2 How to Observe. In preparing for and performing observations, there are several things to keep in mind: � Be familiar with the operations � Be familiar with the guidelines and use them as an expectation baseline � Pay close attention to ensure every detail of the observation is taken in and recorded � Follow your intuition. If something does not look or seem right, check further. Assume your intuition is correct until your research proves otherwise � Take copious notes. This will make it easier to follow up on your observations � Data gathered from observations should be verified and cross-checked by the following methods: observe the same operation being performed by different personnel, observe different operations on the same shift, or observe different shifts performing similar operations These principles can be applied to all observations conducted during an assessment. For example, while observing maintenance on a contaminated valve at a waste-water processing DOE-EM-STD-5505-96 17 facility, the assessor may review the work documentation and applicable facility operating and radiological control procedures before the observation. This will allow the assessor to become familiar with the guidelines to be used as an expectation baseline for the observation. While observing the operation, the assessor should watch the performance of as many operations personnel as possible and take notes on what is observed. Finally, the assessor should review these notes to identify any apparent deviations from requirements and determine the follow up actions necessary to validate or disprove them. 5.8.1.3 Where and When to Observe. Observations can be conducted the entire time that the assessor is at an activity. Observations can take place during scheduled activities (drills, maintenance operations, system alignments, etc.) or can be conducted during activity walkthroughs and tours. The activity plan of the day or plan of the week should aid in identifying the time and place for observing scheduled activities. 5.8.1.4 Stop Work Authority. In general, assessors should be familiar with and abide by the Operations Office's policy concerning stop work authority. While observing an operation, if an unsafe situation arises in which operators fail to recognize a potential safety concern, the assessor

has a responsibility to point this out to the operators. Having the safety concern pointed out, the operators should correct the problem. If they do not, the next step might be to contact the work supervisor so that he or she can put a stop to the unsafe work. If the situation requires immediate action to prevent personal injury, the assessor must be the authority in stopping the unsafe work. An assessor who stops unsafe work should notify the team leader immediately. The team leader will take the necessary steps to inform activity management. 5.8.2 Interview. During any assessment, the interview process will be an important information-gathering tool. Interviews are an effective method of determining level of knowledge and familiarity with activity policies and procedures. An interview can be the key step in following up on leads. To ensure that all relevant information is obtained efficiently and DOE-EM-STD-5505-96 18 accurately, each interview should be planned and organized. 5.8.2.1 How to Prepare for Interviews. Planning is critical to the success of any interview. It enables the assessor to maximize the use of interview time. The planning process does not have to be formal; it may be simply a matter of determining what information is being sought. The assessor should determine which interviews can be conducted during walkthroughs, special activities, routine operations at the activity, and which interviews require coordination and scheduling with activity management. In planning the interview, the assessor should identify the goals for the interview and the items to be discussed so that a logical sequence of questions can be developed beforehand. It may also be appropriate to bring relevant reference documentation for review during interviews. For an interview with the Radiological Control Manager, it may be useful to have survey records and the pertinent radiological control manual readily available. 5.8.2.2 Conducting Interviews. Two types of questions are routinely used during the interview process: open-ended and closed-ended. A good mix of these two types of questions should provide the assessor with enough information. An open-ended question places the burden of conversation on the interviewee and gives the assessor time to analyze what the interviewee is saying. It reduces the total number of questions asked. An open-ended question elicits more than a yes or no response and is very useful when starting a line of questioning in a new subject area. For example, asking an operator to "explain" or "describe" an event is an open-ended question. The advantage of open-ended questions is that they usually provide a large amount of information about the topic of interest to the assessor. The interviewee does most of the talking in responding to them, and the information provided is generally volunteered. A closed-ended question is a specific question that is often answered with only one or two words. Examples are "what," "when," "who," and "where" questions. Closed-ended questions DOE-EM-STD-5505-96 19 place the burden of conversation on the assessor in that the assessor spends much more time thinking of and stating the question than it takes the interviewee to answer it. The advantages of closed-ended questions are that they allow the interviewer to obtain specific information about a topic of interest, are less time-consuming to answer, and are easier to record. Assessors should

avoid using only closed-ended questions as they can elicit insufficient information, and require a lot of follow-up. The proper combination of open-ended and closed-ended questions will provide the most efficient use of time for both the assessor and the activity. To prevent the interviewee from digressing from the topic of interest, the assessor must direct the interview. The assessor’s use of leading questions should be minimized. In a leading question, the assessor gives the interviewee a partial answer and expects the interviewee to complete it, or the assessor gives the complete answer and expects the interviewee to agree or disagree with it. During the interview, the interviewer should continuously evaluate the information being gathered, and ask follow up questions when necessary. This may lead to additional paths of discussion that are necessary to clarify the topic. Before concluding the interview, the assessor should attempt to summarize the information received from the interviewee. This will ensure that the assessor has correctly interpreted and recorded the information provided during the interview. After completion of an interview, the assessor should decipher, analyze, and evaluate the notes taken and identify areas that require follow up. The assessor should decide whether follow-up observations, document reviews, or interviews are required to clarify the information gathered during the interview. An important point to remember is that the information gathered during an interview, observation, or document review should be corroborated with other information. In order to validate apparent deviations from requirements and identify programmatic breakdowns DOE-EM-STD-5505-96 20 or widespread problems, more than one assessment technique should normally be used. 5.8.2.3 Where and When to Conduct Interviews. During an operations assessment, most interviews should be conducted while performing observations and document reviews. If necessary, interviews can also be scheduled and conducted separately. For example, operators and shift supervisors can be interviewed during rounds and while conducting operations, as time permits. However, the assessor must ensure that the operators are not distracted from performing their duties while being interviewed during operational activities. If it is necessary, these personnel can also be interviewed in a more formal setting such as an office or conference room. In general, operators and shift supervisors can be interviewed at their work location. Conversely, it is usually best to interview activity managers, operations supervisors, and staff personnel in their offices or in another convenient location. Assessors should coordinate with the interviewee to establish the time and location of scheduled interviews. To minimize the impact on activity operations, assessors should always plan to arrive at the interview location on time, or reschedule the interview. 5.8.3 Document Review. The assessment team should review applicable documents both prior to and during the assessment. In general, document reviews are conducted for the following reasons: � To gain familiarity with the requirements contained in activity policies, procedures, etc. � To validate or disprove apparent deviations from requirements and identify programmatic breakdowns or widespread problems � To investigate apparent inconsistencies between activity policies and procedures, and DOE

Directives DOE-EM-STD-5505-96 21 � To refresh the assessors knowledge of the elements of conduct of operations. Applicable DOE standards and guidelines should be reviewed prior to the assessment. There are DOE Standards for each element of conduct of operations. � To identify equipment or shift-duty performance problems Since operations assessments should be focused on operations, assessors should not spend the majority of their time conducting document reviews. Document reviews should be used by assessors to establish their expectations concerning activity operations and when pursuing leads developed from observations and interviews. 5.8.3.1 What Documents to Review. As discussed in section 5.4.1, assessors should review selected documents when planning an assessment. Additional documents that should be reviewed during the assessment include: � Activity program and operations procedures and policy manuals � Lockout and Tagout procedure, logs, and "Danger" and "Caution" tags � Round/tour inspection sheets � Training records and databases � Notification lists and documentation � Equipment and system alignment records � Equipment deficiency logs and identification tags � Maintenance requests and work packages � Temporary modification control records � Operating logs � Shift turnover checklists � Chemistry sampling and analysis records � Required reading files and indexes � Equipment and system operating procedures � Timely orders to operators DOE-EM-STD-5505-96 22 � Operator aids posted in the activity � Radiological control survey logs � Radiological postings � Equipment calibration records � Approved lists of abbreviations and acronyms � Emergency communication system test records 5.8.3.2 Reviewing Documents. In general, an assessor should keep the following guidance in mind when conducting document reviews: � Identify questions for interviews and key steps or sections of operating procedures to observe. � Identify administrative and technical source documents and higher-level procedures to check. � Take specific and accurate notes to allow efficient follow up, if necessary. Documents can be reviewed before, during, or after observations and interviews. To start an assessment of chemistry and unique processes, for example, the assessor could decide to review operating documents such as activity logs and cross-check them for consistency with each other and the governing procedure. For instance, if a sample was taken, analyzed, and reported, do the sample logs and the discharge logs agree? Was the sample taken, analyzed, and the results reported prior to the discharge occurring? Is this required? If discrepancies are found, additional observations, interviews, and document reviews should be conducted. The assessor could also start the assessment of this area by reviewing activity and site procedures. Once these procedures are reviewed, an observation of the sampling process could be conducted to determine if deviations from requirements exist. 5.8.3.3 When and Where to Review Documents. Document reviews can be conducted in conjunction with observations and interviews, as well as separately. For example, while DOE-EM-STD-5505-96 23 following an operator conducting rounds throughout the activity, an assessor can normally review the following documents: � Operator round/tour inspection sheets and other narrative logs � Activity operating procedures at various work stations � Radiological control postings and survey logs

� Control area operating logs and other documents that may be kept in the control room such as the Lockout and Tagout log, Material Deficiency log, Occurrence Notification lists, etc. � Operator aids log As stated above, documents can also be reviewed as a separate activity. Complicated documents such as program procedures and policy manuals may require dedicated, in-depth review by assessors. These reviews can be done at the activity (in a meeting room, for example), or at the assessment team's operating base. 5.8.4. Pursuing leads. "Pursuing leads" is the process of following up on an apparent deviation from requirements using observations, interviews, and document reviews. A lead is considered a deviation from expectations or an apparent deviation from requirements that is developed as a result of observations, interviews, or document reviews. The process of pursuing leads should continue until all leads have been validated, and any programmatic breakdowns or widespread problems are identified, or until sufficient evidence exists to disprove the apparent discrepancy. The ability to identify leads and pursue leads depends on the intuition, experience, and tenacity of the assessor. To effectively identify leads, an assessor must: � Trust intuition. If something does not seem right, keep looking until verified right or wrong. � Do not assume more than evidence supports. � Maintain a questioning attitude about all that is observed during the assessment. � Be familiar with the requirements of each area of conduct of operations and activity policies and procedures. DOE-EM-STD-5505-96 24 Furthermore, to pursue leads effectively, an assessor must: � Seek supporting information and evidence through follow up observations, interviews, and document reviews � Investigate the programmatic aspects (Is there a programmatic breakdown or widespread problem, or is this an isolated case?) � Be tenacious and persistent until satisfied with the answers obtained � Use good judgement when determining how much time is to be spent on one issue (If pursuit of a lead will result in time constraints on the assessment, discuss the issue with the team leader.) Pursuing leads is a vital part of the operations assessment process. It is not sufficient to just identify apparent deviations from requirements. In order to develop well-supported findings and concerns, the assessment team must confirm all findings and develop concerns that direct activity management's attention to the underlying causes. To illustrate the process of pursuing leads, consider an operator pumping a wastewater retention basin to the environment through a permitted outfall via a natural creek flowing next to the site boundary. During the assessment, an assessor notes that one operator bypasses the pH meter when starting the discharge to the creek. This action is not stated in the procedure. This is a deviation from the expectation that operators should follow a procedure and should not take important action which is not covered in a procedure. The assessor needs to determine why the operator took an action which was not covered in the procedure. Bypassing alarms, warning devices and instrumentation is a serious operational matter. For instance, the assessor could decide to interview the operator during the basin pumping operation. During the interview, the operator states that she always bypasses the pH

meter on starting the pump because the shift supervisor has instructed her to do so. Now the situation has changed in that it is not necessarily one operator who is not following procedures; it DOE-EM-STD-5505-96 25 is the shift supervisor disregarding written procedures and instructing the shift operators accordingly. So far, the assessor has been limited to observing only one shift. Therefore, an observation of an operator on a different shift might be appropriate. During this observation, the assessor notes that the second operator also bypasses the pH meter in the same way. When questioned, the operator states that the shift supervisor provided the same instruction since it reduces the alarms that are received in the control room. The assessor now can conclude that this practice which is not allowed by procedure is not limited to one operator or shift. There are several directions in which the assessor could proceed. For example, the assessor could interview personnel from the engineering or maintenance or training organization. The questioning may indicate that supervisors and operators are living with system deficiencies because of a design or maintenance problem with the pH circuit or defective cell system. The logical questions that the assessor should ask are: Have the problem alarms been identified? Has engineering evaluated the problems? Is operations compensating for poor design or performance? How was the determination made that the alarms received are false? What was the basis and who authorized bypassing an NPDES instrument? Is bypassing the cell approved? Why are there no compensatory measures in the procedure to ensure that the cell is not bypassed during the discharge? There are some areas of concern that may arise from this scenario. The fundamental issue of concern related to conduct of operations is the bypassing of a safety or monitoring system. Another area of potential concern is that there may be a reportable issue to the state or federal environmental protection agency because the monitoring system is part of the site environmental permit. An additional area of concern might be why is there a bypass at all. More generally, why do trained personnel, who should be knowledgeable about alarms, permits, bypassing systems, instrumentation, procedural protocol, not challenge the verbal direction to bypass DOE-EM-STD-5505-96 26 systems without extraordinary compensatory measures. Other areas that the assessor might investigate to pursue leads further include: � Determining if the procedure change system is too cumbersome for operators to use for known problems � Investigating to see if the technical engineering group submits procedure changes in response to design problems � Analyzing if the procedure writer's group has been tasked with higher priority projects than making changes to the retention basin procedure � Exploring activity management involvement in assuring that procedures are followed according to management expectation and that they remain current 5.9 Daily Team Meetings. The time and location for a daily meeting between all team members should be established. The meeting should be chaired by the team leader and should last approximately one hour. To ensure that every team member participates, the meeting should be structured in a relatively informal manner with enough opportunity for open discussion.

Typically, each team member provides a short summary of any leads, potential findings or concerns that are developing, and any areas of inquiry and issues to be pursued the next day. All team members should have the opportunity to question observations and comment on potential similarities or overlaps with their activities. However, team members should not go into lengthy discussions describing an accounting of their days activities. During the day, team members may discover information that is the responsibility of other team members. For large teams, it is helpful to designate a written method for passing information from member to member. It may be advantageous to utilize a 3x5 index card listing the following information: the lead discovered, the name of the individual who discovered it, and the group to whom it is being passed. This information should be passed as soon as possible but not DOE-EM-STD-5505-96 27 later than the daily team meeting. During the discussion by each team member, the team leader should take notes summarizing leads, findings, and concerns. The information could be recorded using a flip chart or overhead transparency, to allow the team leader to track the resolution of each item as the assessment progresses. This record also provides the team leader with a list of topics to be discussed with activity management, normally accomplished by a separate daily meeting between the team leader and the managers of the activity. 5.9.1 Daily Meetings with Activity Management. The management of the activity should be provided with periodic information regarding the status of the assessment, findings, and areas in which management attention could facilitate the conduct of the assessment. The team leader can either allow contractor management to attend the daily assessment team meeting or can meet separately with contractor management immediately following the daily team meeting. Any safety issues that require corrective action should be communicated immediately to activity management. Open communication is imperative. Feedback from the activity can prevent inaccuracies from being included in the findings. The most effective way to resolve misunderstandings is to promptly contact the management of the activity and request their assistance. This request should be made by the team leader. By coordinating all communication with activity management through the team leader, the team leader is in a position to monitor the assessment and ensure it progresses smoothly. 5.10 Findings and Concerns. The end product of the assessment process is the identification and reporting of findings and concerns. The assessment report provides activity management with areas for improvement, as reported through these concerns. Findings and concerns are defined as follows: DOE-EM-STD-5505-96 28 � A finding is an individual item that does not meet requirements. � A concern is a determination of a programmatic breakdown or widespread problem supported by one or more findings. While findings are the “facts” and are observable evidence of deviations from policy, procedure, or requirements in a DOE Order or activity policy or procedure, concerns are the issues that management must address in order to eliminate the recurrence of findings. For example, an operator is observed performing an incomplete self survey before leaving a posted Contamination

Area. This is a finding that could be written as follows: “An operator exited a Contamination Area without conducting self-monitoring as required by the posted procedure.” An individual finding that indicates a programmatic breakdown or widespread problem can also rise to the level of a concern. For instance, an assessor reviews the graded-approach document and realizes that it has not been approved by DOE. This is a concern that can be written as follows: “There is no approved graded-approach document for implementing conduct of operations at the activity .” Concerns can result from an individual finding, as discussed above, or also from several findings in one or more areas of the assessment. For instance, an assessor discovers that 7 of 10 Control Area operators are not aware of maintenance activities in progress, do not control routine operations, and do not participate in the shift turnover process as required by activity procedures. These findings indicate that there are widespread programmatic breakdowns regarding the control of activity equipment and systems. This is a concern that can be written as follows: “Waste- treatment control room operators are not aware of all operations in progress and do not adequately control the waste treatment process. They are normally not briefed at shift turnover.” This example illustrates how findings from two areas of an assessment, Control Area Activities and Shift Turnover, can result in a concern. The flow diagram in Appendix B shows the process used to develop findings and concerns. DOE-EM-STD-5505-96 29 5.11 Developing Concerns. The final team meeting should focus on development of concerns based on all findings uncovered during the assessment. The daily team meetings should have familiarized each team member with the findings developed by other assessors. This familiarity enables the team to consider all findings while developing activity-wide concerns. Prior to the concerns generation meeting, it is helpful to post the findings identified by each assessor on the walls of the meeting room so they are readily available for reference. Each assessor should develop any preliminary concerns based on the entire team's findings. The concerns may focus on the areas examined by the individual assessor or they could be from another area. Preparation will ensure that the meeting progresses smoothly. The team leader should facilitate the session and start by choosing one assessor to enumerate any concerns that they have developed. The first assessor should present concerns one at a time, explaining the evidence supporting them. The team should discuss each concern as it is presented, either changing the wording and scope as necessary to make the concern accurately reflect conditions in the activity or discarding it altogether if there is not enough evidence to support it. This process should continue until all assessors have presented their concerns and the team has reached consensus on the final list. The team leader should decide in those cases where there is a split opinion. 5.12 Report Generation. When the assessment is completed, all team members must have completed their investigations and have sufficient documentation to support all findings and concerns. The out-brief should provide activity management with the findings and concerns from the current assessment. 5.12.1 Report Format. The final report should be in the format illustrated in Appendix G and

contain the following elements: DOE-EM-STD-5505-96 30 � Introduction . The introduction should provide a brief overview of the purpose and scope of the assessment and should address the status of the activity's graded-approach document for conduct of operations. �� Concerns. Concerns should be organized into the following five areas, if applicable: Activity Management, Control of Operations, Maintenance, Training, and Radiological Control. � Findings. Findings should be grouped by the applicable Element of Conduct of Operations, followed by findings in the areas of Radiological Controls, Drills, and Training, as appropriate. � Assessment Team Members. A complete listing of assessment team members and their organizational affiliation should be included as an attachment to the report. 5.12.2 Distribution. Copies of the draft report should be provided for information to activity management and assessment team members at the exit meeting. Additionally, a copy of each operations assessment report shall be submitted by the assessment team to the senior EM executive at the Operations Office and the responsible Headquarters offices. 5.13 Assessment Out-brief. The assessment out-brief, conducted by the team leader, should provide immediate, organized feedback to the activity regarding the assessment team's conclusions. As with the activity in-brief, the DOE Facility Representative, team members, and contractor management should attend this meeting. The team leader should prepare opening remarks followed by a clear presentation of all concerns developed by the team. Questions by activity management on specifics of the concerns should be encouraged by the team leader. Team members most knowledgeable in the findings that support individual concerns should be prepared to assist the team leader in answering questions when needed. DOE-EM-STD-5505-96 31 6. GUIDELINES FOR ASSESSING OPERATIONS 6.1 Assessing Operations Organization and Administration (OPS O&A). 6.1.1 Discussion. The organization and administration of operations should ensure that a high level of performance in DOE activity operations is achieved through effective implementation and control of operational activities. A high level of performance in DOE operations is accomplished by establishing high operating standards, by communicating operating standards to the working level, by providing sufficient resources to the operations department, by ensuring personnel are well trained, by closely monitoring performance in operations, and by holding workers and their supervisors accountable for their performance in conducting evolutions. 6.1.2 Preparing to Assess OPS O&A. To develop an assessment plan for OPS O&A, assessors should first review the activity-specific graded-approach document and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. OPS O&A provides the philosophical underpinning of Conduct of Operations. As such, OPS O&A provides the framework for the management controls contained in sections 6.2 through 6.18. Because it is an all-encompassing section, OPS O&A should be continuously evaluated by all assessors. All assessors should measure how well management has defined and communicated

their policies, how well working-level personnel understand and execute management's expectations, and how management monitors operations. Assessors should continuously ask themselves three questions: � Do workers know what is expected of them? DOE-EM-STD-5505-96 32 � Has management provided guidance about their expectations? � Are workers properly trained? There are two complementary approaches to assessing OPS O&A. The first is a top-down approach that emphasizes interviews with each level of management to determine their expectations and how they monitor operations performance throughout the organization, down to the operating level. The second approach emphasizes observing and interviewing operators to determine their level of knowledge and then moving up the organization to determine management expectations. Assessors should decide whether to use one or a combination of these approaches based on the activity being assessed and their particular assessment experience and style. The nature of this area of conduct of operations necessitates the assessor's use of interviews. However, assessors must actually go out and verify that activity performance matches management guidance. 6.1.3 Observation. Observation can be an important tool in determining how effectively management has communicated its policies to workers and how well workers execute management expectations. There are numerous observations an assessor can make to determine if policies are being communicated and implemented. Specific examples of activities that an assessor might observe include: � Activity walkthroughs � Activity management conducting scheduled walkthroughs � Pre-job briefings � Postings of management safety policies and goals � Graphs and charts of operations output and performance � Postings of safety goals and actual worker performance DOE-EM-STD-5505-96 33 As an example of how to conduct an observation of OPS O&A, consider the first two examples from the above list. First, walk through the activity to identify apparent operations and material deficiencies. Once this has been completed, accompany a manager or supervisor while they conduct a scheduled management walkthrough. During the observation, note what actions the manager takes when deficiencies are identified, specifically: � Are deficiencies documented? � Did the manager identify the deficiencies that you previously identified? The observation may lead to an unscheduled interview with the manager. Some questions related to this example include: � How do you monitor activity performance? � How often do you perform activity walkthroughs? � How do you address deficiencies that are identified during walkthroughs? Once the observation has been completed, compare what was observed to the requirements contained in activity polices and procedures to identify any apparent deviations from requirements. Pursue leads using additional observations, interviews and document reviews to confirm or disprove those leads and determine if programmatic breakdowns or widespread problems exist. The above is just one example of an observation that could be performed while assessing OPS O&A. 6.1.4 Interview. Depending on the leads developed during observations, interviews related to the organization and administration of the activity are an effective method of determining where potential deficiencies exist. One of the primary objectives of interviews is to determine how well

goals are communicated from top management down through the organization's front-line workers. As stated earlier, OPS O&A affects all levels of the organization. As a result, it is imperative that assessors talk to personnel at all levels of the organization. Interviews with managers, supervisors, and a sampling of front-line workers will give an indication of how well DOE-EM-STD-5505-96 34 goals are communicated and understood, in addition to how management monitors activity operations. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Scheduled interviews should also be conducted to develop and clarify leads. Some interview goals include determining whether: � Operations policies exist and are communicated to workers � Goals are communicated to and from each level of the organization � Resources are sufficient to accomplish assigned tasks � Activity management takes an active role in activity operations by monitoring operating performance � Training is formalized for supervisory and management positions � Activity management effectively plans for safety Some example interview questions for managers at the activity include: � What are your operating goals and how do you communicate them to workers? How are these goals achieved and tracked? � How do you monitor operations and safety performance and improvement? � How do you plan for safety? � What is your involvement in developing training for your workers? � How do your supervisors and operators know what their responsibilities are, and those of their immediate subordinates? How do they know the scope of their authority? � What is your perspective of safety versus production? What are your worker's impressions of safety versus production? � What type of supervisory training do you and your managers/ supervisors receive? � How does management support job planning? How are workers included in the job planning process? DOE-EM-STD-5505-96 35 Interview questions for operators include: � What are management's goals? � How often do you see management in the activity? � How do you plan for safety? � What is your impression of safety versus production? Finally, a question that could be asked of training, maintenance, and radiological control organization personnel is: � What is the working relationship between operations and support organizations (Training, Maintenance, Radiological Control)? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by conducting one or more of the following: � Observations of and interviews with operators � Tours of the activity and interviews with shift supervisors or foremen � Interviews with shift supervisors, activity managers, and operations supervisors � Reviews of operating and program procedures � Interviews with training, maintenance and radiological control organization personnel 6.1.5 Document Review. Activity policies may be reviewed during the assessment to become familiar with activity operations and management expectations concerning goals and authority. However, since document review only describes administrative performance, actual performance should be verified through interviews and observations. Examples of documents that could be reviewed include: � Management safety policies and goals

� Performance indicators � Operations resources and staffing plans DOE-EM-STD-5505-96 36 � Policies and procedures related to monitoring operations performance, safety goal performance, and management training � Pre-job or safety checklists � Documented management self-assessments � Management training records When reviewing documents, assessors may decide to review the activity's goals. There are three questions the assessor should ask concerning goals: � Do goals exist? � Are goals measurable, challenging, and auditable? � Do all personnel understand the goals and have familiarity with the actions needed to achieve them? The existence of goals is only part of the answer, however. While goals may be posted on bulletin boards throughout the activity, this in itself does not indicate that workers understand how management expects those posted goals to be executed. After becoming familiar with activity goals, assessors may want to conduct informal interviews with managers, supervisors, and operators to determine their understanding of goals. 6.2 Assessing Shift Routines and Operating Practices. 6.2.1 Discussion. The purpose of Shift Routines and Operating Practices is to set standards for the professional conduct and level of performance expected of operations personnel to ensure facilities are operated safely and efficiently. The guidelines establish the overall framework for single and multi-shift operations and apply to routine operations as well as to the detection of and response to abnormal conditions. 6.2.1.1 Approach. Since the observations, interviews, and document reviews conducted to DOE-EM-STD-5505-96 37 assess Shift Routines and Operating Practices will normally provide leads in other elements of conduct of operations (Logkeeping, for example), the assessment of shift routines can be combined with other elements of conduct of operations such as Shift Turnover, Control Area Activities, Communications, Operations Aspects of Facility Chemistry and Unique Processes, and Logkeeping. This combined approach is presented in the following sections. 6.2.2 Preparing to Assess Shift Routines and Operating Practices. To develop an assessment plan for Shift Routines and Operating Practices, assessors should first review the activity-specific graded-approach document and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.2.3 Observation. The activity plan of the day and plan of the week should be reviewed to determine the time and place for observing work in progress and drills. Discussion and coordination with the shift supervisor or person in charge is very useful to establish opportunity for observations. The assessor should be sure the person in charge knows the reason for the observation. Observations of shift routines should be conducted during walkthroughs and tours with operators and shift supervisors, and during the conduct of special operations and drills. The following situations are good candidates for conducting observations: � Maintenance activities that require interface or coordination with roving operators, control area operators, and shift supervisors � Startup and shutdown of systems and equipment

� Process or rate change operations � Activity drills � Routine operations of equipment � Control Room operations DOE-EM-STD-5505-96 38 � Operator and Supervisor inspection tours There are several aspects of Shift Routines and other related elements of conduct of operations that can be assessed during these observations. For example, while observing the startup or shutdown of a system or a component, the following items might be observed: � Operator response to indications and alarms as valves are operated and system components are energized or deenergized � Operator adherence to posted personnel protection requirements, use of monitoring instruments, and adherence to radiation and safe work permits � Resetting of protective devices, if trips occur � Knowledge, skill and level of qualification of operations personnel � Notification of changes in system status by operators and shift supervisors � Communication between shift positions Observations during routine and abnormal events may present the opportunity to observe the following: � Workers conducting assigned tasks expeditiously should not interfere with good watchstanding practices � Operators notifying the shift supervisor (or cognizant manager for research and test facilities) of any unexpected situations � Operations personnel adhering to the requirements of the facility industrial safety program. Proper hearing, eye, head, foot, and respiratory protection should be worn in designated areas to reduce the potential for injury. � Ladders or other approved means are being used to access equipment located in the overhead when permanent steps or catwalks are not available. � Personnel exercising appropriate precautions when entering or working in or around energized panels or equipment. For example, ensure that electrical panel closures are securely fastened prior to making the breakers operable to energize equipment. DOE-EM-STD-5505-96 39 � Workers conducting a thorough tour of all areas within his/her responsibility, making appropriate equipment inspections at designated times at least once per shift (inspected less frequently because of adverse radiological or equivalent personnel safety conditions, or more frequently if problems have been encountered.) During walkthroughs and inspections by operators and supervisors, the following aspects of Shift Routines and other related elements of conduct of operations can normally be evaluated: � The scope, detail, and periodicity of operator inspection tours � Adherence to personnel protection practices, postings, radiation and safe work permits, etc. � Response to instrument readings, indications, and abnormal conditions � Use and adequacy of the notification process regarding changes in status, abnormalities, and difficulties encountered during operations � Communication between shift positions � The use of narrative operating logs and round/tour inspection sheets � The adequacy of shift turnover documentation and the process used � Workers checking components, such as electrical panels, alarm panels, auto-start standby equipment, and breakers for unusual conditions. � Workers inspecting all areas for which they are responsible � Supervisory personnel periodically monitoring operator rounds. Operators, maintenance personnel, and shift supervisors from each shift should be observed during the assessment. If possible, the observation of a particular shift position should start at

shift turnover since vital information about the operating status of equipment and systems, and operations and maintenance in progress, is passed between shifts at this time. To observe an operator performing rounds the following methodology could be used: � Observe shift turnover conducted between the on-coming and off-going operators. Pay attention to the information passed verbally between the operators concerning operations DOE-EM-STD-5505-96 40 planned or in progress, status of activity systems and equipment, and any abnormal conditions that exist in the activity. Review any narrative logs, round/tour inspection sheets, and any other documentation used in the turnover process. � Attend the shift crew briefing. Observe the process used and the information exchanged. Compare the information disseminated at the briefing with the information that was passed between the operators during their turnover. � Accompany the operator as he performs his duties. Observe the items listed in section 6.2.3. If the activity has a control area, observe the performance of the control area operator and the interface between shift positions. � At the end of the shift, observe the turnover process between this operator and the oncoming operator. Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. A similar process can be used to observe control area operators and shift supervisors during operations and drills conducted by the activity. 6.2.4 Interview. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and validate leads. The following personnel might be interviewed during the assessment: � Operators and maintenance department personnel � Shift supervisors or foremen � Activity managers and operations supervisors � Radiological control technicians � Activity operations staff DOE-EM-STD-5505-96 41 � Training department personnel Most of the interviews accomplished to assess Shift Routines and Operating Practices should be conducted with operators, maintenance personnel, and shift supervisors. Whenever possible, these interviews should occur during observations of operations, and activity walkthroughs and tours. To pursue leads, scheduled interviews with the activity manager, and training, maintenance, and radiological control organization personnel may also be necessary. For example, while observing a roving operator beginning at shift turnover, the following types of questions could be used to assess level of knowledge about shift routines: � What are your duties and responsibilities during the shift? � What things do you normally look for during an activity tour? How often do you conduct tours? What areas of the activity are you responsible for? Describe the process that you use to document, report, and correct deficiencies discovered during tours. � Do you use a round/tour inspection sheet to document your tours? What types of information do you record? How do you document abnormal readings? What do you do to report and correct the condition?

� What are your responsibilities regarding the activity industrial safety program? Are there any areas in the activity that have special safety requirements? How do you know where they are and what safety equipment is required? � Are there any Controlled Areas in the activity? How do you know what personnel protective equipment (PPE) must be used to enter the area? Have you been trained in the use of PPE? � How do you normally communicate with the control area operator and shift supervisor? How do they communicate urgent information to you? Do you ever use radios to communicate? Are there any areas in the activity that radios can't be used? If so, where are they? � How do you determine what information should be passed at turnover to the on-coming operator? Are you required to use a turnover checklist to aid the process? What guidance have you received from the shift supervisor or activity manager concerning how to conduct DOE-EM-STD-5505-96 42 shift turnover? Do you feel that the process used at turnover supports your needs? What changes, if any, should be made to improve the process? Many of the questions asked during a walkthrough with an operator may be prompted by observing the work area for which the operator is responsible. After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Tours with several more operators � Observations of and interviews with control area operators � Tours of the activity and interviews with shift supervisors � Interviews with radiological control technicians � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures � Interviews with training and maintenance department personnel 6.2.5 Document Review. Examples of documents that should be reviewed include: � Operating logs and round/tour inspection sheets � Alarm status log � Radiation and safe work permits � Safety and radiological postings � Shift turnover checklists � Health and safety plans Most document reviews necessary to assess Shift Routines and Operating Practices can be conducted in conjunction with observations. For example, while touring the activity with an operator, a review of documents such as operating logs, round/tour inspection sheets, operations DOE-EM-STD-5505-96 43 procedures, safety and radiological control postings, and radiation and safe work permits, can be accomplished. To illustrate how to perform a document review, consider a round/tour inspection sheet. Various items on the sheet can serve as a lead to develop interview questions and conduct other observations. For example, look for equipment parameters that are out of specification or are approaching an out of specification condition. If found, ask the operator what actions should be taken if that parameter goes out of specification, or make up a hypothetical situation and ask the same type of question. Look at the round sheets for vague checkoffs such as "check sump level." A follow on question for this may be "What do you look for when you perform this check?" and then the operator's response can be compared with the answers of other operators. Another example might be, the recording of key equipment parameters during tours to provide a

record of equipment performance and can be used to reconstruct events leading up to unusual occurrences or system malfunctions. Is there evidence that these records are used to improve equipment performance or maintenance? Is there evidence that the round sheets are used for short-term trending by operators and other technical support organizations? Review round sheets (those which are in use and completed) for technical content such as: � Round/Tour Inspection Sheets Round inspection sheets should where appropriate include maximum/minimum equipment parameters or expected operating ranges to enable operators to recognize abnormal readings quickly. � Safety limits derived from Technical Specifications or Operational Safety Requirements should be highlighted. � Equipment should be listed on round sheets in the same order that it would be encountered during a normal tour of the operating station, and the round sheets should include a narrative section. This technique of using document review as a source for interview questions and observations can be an extremely effective tool for an assessor. DOE-EM-STD-5505-96 44 6.3 Assessing Control Area Activities. 6.3.1 Discussion. Because most control areas or control rooms function as the coordination point for important activity work and operations, activities in the control area or room must be formalized and conducted in a professional, businesslike manner. Control Area Activities addresses important elements that are necessary to support safe and efficient activity operation. 6.3.1.1 Approach. As discussed in Assessing Shift Routines and Operating Practices, control area activities are an integral part of shift routines. This "combined" approach of assessing activities in the control area with shift routines is presented in that section. Since it may be necessary to assess activities in the control room separately, the following sections discuss how to assess Control Area Activities only. 6.3.2 Preparing to Assess Control Area Activities. To develop an assessment plan for Control Area Activities, assessors should first review the activity-specific graded-approach document and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.3.3 Observation. Control Area Activities is an observation-intensive area of conduct of operations to assess. Observations of activities in the control area should be conducted during activity walkthroughs and tours, and during the conduct of operations and drills. The activity plan of the day and plan of the week should be reviewed to determine the time and place for observing operations and drills. The following situations are good candidates for conducting observations: � Startup and shutdown of systems and equipment � Process or rate change operations DOE-EM-STD-5505-96 45 � Activity drills � Maintenance and special operations controlled from the control area � Excessive and extraneous communication to and from the control room, for example persons calling the control room to have persons paged, phone calls not related to official business. There are several aspects of activities in the control area that can be assessed during these

situations. For example, while observing the startup of a system or component the following might be observed: � Control of personnel access into the control area and the "at-the-controls" area � Behavior and atmosphere in the control area by operators on shift and other personnel in the area � Operator response to indications as valves are operated and system components are energized or deenergized � Operator response to alarms and abnormal conditions � Operation of control area equipment by operators � Use of status boards in the control area � Ancillary duties performed by control area operators � Control area operators should not be overburdened with administrative responsibilities, and control area access should be limited so that operators will not be distracted from properly monitoring facility parameters. During the assessment, control area operators from each shift should be observed to get a broad picture of the level of knowledge, skill, and qualification of operators. If possible, these observations should start at shift turnover since vital information about the operating status of equipment and systems, maintenance and operations in progress is passed between shifts during turnover. DOE-EM-STD-5505-96 46 To observe a control area operator, the following methodology could be used: � Observe the shift turnover conducted between the on-coming and off-going control area operators. Pay attention to the information passed verbally between the operators concerning operations planned or in progress, status of activity systems and equipment, and any abnormal conditions that exist in the activity. � Attend the shift crew briefing. Compare the information disseminated at the briefing with the information that was passed between the operators during their turnover. � Observe the control area operator as duties are performed. � At the end of the shift, observe the turnover process between this operator and the on-coming operator. Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursuing these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. 6.3.4. Interview. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and validate leads. The following personnel might be interviewed during the assessment: � Control area operators � Shift supervisors or foremen � Activity managers and operations supervisors � Activity operations staff � Training department personnel Most of the interviews accomplished to assess Control Area Activities should be conducted with DOE-EM-STD-5505-96 47 operators, and shift supervisors. Whenever possible, these interviews should occur during observations of operations, and activity walkthroughs and tours. To pursue leads, scheduled interviews with the activity manager and training department personnel may also be necessary. For example, while observing a control area operator beginning at shift turnover, the following types of questions could be used to assess his level of knowledge: � What are your duties and responsibilities during the shift?

� How do you control access to this area? How do you decide what personnel should be granted access? Is there an official access list for the control area? Is there a limit to the number of people allowed in this area? � Who normally operates the equipment in this area? How do you know who is authorized to operate this equipment? Under what circumstances do personnel other than yourself operate this equipment during your shift? � Do you normally perform ancillary duties while on shift? What types of ancillary duties do you perform? How do you know what ancillary duties are authorized during your shift? � What types of operations, and maintenance activities do you control from this area? Describe the process for controlling these operations. � What is the policy concerning reading newspapers or magazines in the control room? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by conducting one or more of the following: � Observations of and interviews with other control area operators � Interviews with shift supervisors � Interviews with activity managers and operations supervisors � Reviews of program procedures A similar methodology to the one described above can be used to observe specific activity operations and drills, considering that the list of interview candidates and sample questions DOE-EM-STD-5505-96 48 provided may not be all inclusive. 6.3.5 Document Review. Examples of documents that should be reviewed when assessing activities in the control area include: � Program and operating procedures � Control area access lists � Alarm status log Most document reviews necessary to assess control area activities can be conducted in conjunction with observations. For example, while observing an operator in the control area, review control area access lists and other operational postings. If necessary, reviews of program procedures and other documents that are not routinely used during activity operations can be fit in between scheduled interviews and observations. 6.4 Assessing Communications. 6.4.1 Discussion. Highly reliable communication provides accurate transmission of information within an activity. Since accurate communication is essential for the safe and efficient operation of facilities, guidance should be provided by activity management regarding the use of all forms of audible communication. This includes oral (face-to-face), telephone, radio, public address, and special (horns, sirens, and bells) communication equipment and methods used during operations. 6.4.1.1 Approach. As discussed in Assessing Shift Routines and Operating Practices, communication is an integral part of shift routines. This "combined" approach of assessing communications with shift routines is presented in that section. Since it may be necessary to assess communications separately, the following sections discuss how to assess this area of conduct of operations only. DOE-EM-STD-5505-96 49 6.4.2 Preparing to Assess Communications. To develop an assessment plan for Communications, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and

procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.4.3 Observation. Observations of activity communications should be conducted during walkthroughs and tours with operations department personnel, and during the conduct of special operations and drills. The activity plan of the day and plan of the week should be reviewed to determine the time and place for observing operations and drills. The following situations are good candidates for observing communication: � Periodic testing of emergency communication systems and equipment. Emergency communications systems should be periodically tested to ensure that they are functional. Control areas should have the capability of overriding other users of the public address system for emergency announcements. � Maintenance and other activities that require communications between roving operators, control area operators, and shift supervisors � Startup and shutdown of systems and equipment � Activity drills � Occurrence notifications � Shift turnover and shift crew briefings � On-shift training activities � Monitoring radio channels used by operations There are several aspects of communication that can be assessed during these situations. For example, while observing a fire drill, the following items might be observed: � Operation of emergency communication systems and equipment DOE-EM-STD-5505-96 50 � Use of the activity public address system. � Use of portable radios by various shift positions � Use of abbreviations, acronyms, and "repeat backs" during communications � Use of standardized terminology and phonetic alphabet � Communications during the occurrence notification process � Communications between drill team controllers Observing the activities above should be supplemented with observations of routine shift communications conducted during walkthroughs and tours of the activity and control area. Operators, maintenance personnel, and shift supervisors from each shift should be observed to get a broad picture of communication accuracy, reliability, and effectiveness. If possible, the observations of a particular shift position should start at shift turnover since vital information about the operations, maintenance, and special operations in progress at the activity, is passed at this time. To observe communications while touring the activity with an operator, the following methodology could be used: � Observe the shift turnover conducted between the on-coming and off-going operators. Pay attention to the information passed verbally between the operators concerning operations planned or in progress, status of activity systems and equipment, and any abnormal conditions that exist at the activity. Review any documentation used to communicate written information in the turnover process. � Attend the shift crew briefing. Observe the information exchanged. � Accompany the operator as he performs his duties. Observe the use of portable radios, the public address system, telephones and face-to-face communications between the operator and other shift positions. If the activity has a control area, observe communications between the control area operator and other shift positions. DOE-EM-STD-5505-96 51 � At the end of the shift, observe the turnover process between this operator and the on-coming operator.

Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. A similar process can be used to observe communications between shift supervisors, maintenance personnel, and other operations department personnel when conduction operations at the activity. 6.4.4 Interview. Interviews of on shift personnel should occur during observations and document reviews. Additional scheduled interviews might also be necessary to follow up and validate leads. The following personnel might be interviewed during the assessment: � Operators and maintenance department personnel � Shift supervisors or foremen � Activity managers and operations supervisors � Radiological control technicians � Activity operations staff � Training department personnel Most of the interviews accomplished to assess Communications should be conducted with operators, maintenance personnel, and shift supervisors. Whenever possible, these interviews should occur during observations of operations and activity walkthroughs and tours. To pursue leads, scheduled interviews with the activity manager and training department personnel may also be necessary. For example, while observing a roving operator beginning at shift turnover, the following types of questions could be used to assess his level of knowledge about the use of communication equipment and methods: DOE-EM-STD-5505-96 52 � How do you determine what information should be communicated at turnover to the on- coming operator? Are you required to use a turnover check sheet to document this information? What guidance have you received from the shift supervisor or activity manager regarding the type of information to be passed at turnover? � What equipment do you normally use to communicate with other shift positions? Do you ever use radios to communicate? If so, are there any areas in the activity that radios can't be used? Where are they? � What type of information is passed on the activity public address system? Who normally uses this equipment? How frequently is it used? � Is there a procedure that addresses how to communicate? If so, what type of information is contained in the procedure? Is there an approved list of abbreviations and acronyms? When is it to be used? Are there any circumstances that require the use of "repeat backs" Can you give an example of one of these circumstances? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Additional walkthroughs and tours of the activity � Observations of and interviews with roving operators, control area operators, and maintenance operators � Tours of the activity and interviews with shift supervisors � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures � Interviews with training department personnel A similar methodology to the one described above can be used to observe other shift positions during operations and drills, considering that the list of interview candidates and sample questions

provided may not be all inclusive. DOE-EM-STD-5505-96 53 6.4.5 Document Review. Examples of documents and other written information that can be reviewed to assess Communications include: � Approved lists of abbreviations and acronyms � Records of periodic emergency communication equipment testing � Portable radio postings � Turnover checklists � Program and operating procedures � Emergency plan Most document reviews necessary to assess activity communications can be conducted in conjunction with observations. For example, while touring the activity with an operator, an assessor should look for and review the approved list of abbreviations and acronyms, portable radio postings, and turnover checklists. After reviewing these documents, observe how and when they are used by operators. If necessary, reviews of program procedures and other documents that are not routinely used during activity operations can be fit in between scheduled interviews and observations. 6.5 Assessing Control of On-Shift Training. 6.5.1 Discussion. An effective On-Shift Training Program ensures that unqualified personnel are properly supervised and controlled to avoid operational errors and to maximize use of the trainee's time. On -shift training should be conducted so that the trainee satisfactorily completes all the required training objectives and receives maximum learning benefit from this experience. Additionally, the program should be structured to ensure that On-the-Job Trainers (OJT) have received the proper training necessary to qualify them as OJT Instructors. On-shift training is not intended to cover all aspects of an effective training and qualification program. Rather, it addresses only the component that includes actual hands-on the equipment DOE-EM-STD-5505-96 54 by the trainee. Additional information regarding training is included in section 6.20. 6.5.2 Preparing to Assess On-Shift Training. To develop an assessment plan for On-Shift Training, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.5.3 Observation. Assessment of on-shift training should be conducted by observing on-shift training in progress. The activity manager, operations supervisor, training manager, or shift supervisor should know the time and location of on-shift training events that will take place during the assessment. Whenever possible, observations should be conducted during each shift. This will provide a good overview of the quality and consistency of the training conducted at the activity. Examples of observations of training evolutions include: � Performing a lockout or tagout � Lining up a system for startup or shutdown � Activity drills � Conducting rounds and tours � Making entries in logbooks and roundsheets � Performing an analysis, taking samples When conducting observations, it is important to note that what is being evaluated is not the student's level of knowledge, but how the training is controlled and supervised. When observing an activity involving on-shift training, start when the trainer first meets with the student and continue until they finish the training session. The following are aspects of

controlling and supervising on-shift training that an assessor should observe: DOE-EM-STD-5505-96 55 � Was communication between the trainer and trainee clear and concise? � Does the trainer explain the objectives to the student prior to starting an operation or task? � Does the trainer fully explain or ensure that the student understands all equipment and processes before allowing the student to execute the desired task? � Does the instructor use a method for verifying the student's understanding of instructions prior to allowing the student to perform the desired task? � Was the number of trainees assigned to a trainer consistent with the on-shift training policy? � Does the trainer provide direction to the trainees on how to stop training if an actual problem condition was encountered? � Does the trainer allow the trainee to exercise their own knowledge without excess coaching, leading, or prompting? � Is the instructor qualified? Does the line organization believe that training and the quality of the trainee performance as a result of the training are the responsibility of the training organization? Once the observation has been completed the assessor should discuss any unclear aspects with the OJT instructor. Clarifications immediately following a training session can often resolve many apparent deviations. For example, an instructor may not have appeared to review procedural prerequisite requirements prior to allowing the student to perform a given task. Through an open discussion of this matter, the OJT trainer may point out that the prerequisites were reviewed the previous day and that the student understood all requirements. The assessor should then pursue leads that have not been resolved using additional observations, interviews, or document reviews and determine if programmatic breakdowns or widespread problems exist at the activity. 6.5.4 Interview. Most of the interviews accomplished to assess on-shift training should be conducted during observations. Depending on the leads developed during observations, additional scheduled interviews with operations and training department personnel may be DOE-EM-STD-5505-96 56 necessary to pursue leads. A sample listing might include the following: � OJT trainers in the operations and maintenance departments � Unqualified operators and maintenance personnel � Operations and maintenance supervisors and managers � Training manager and his instructors � Activity managers and operations supervisors An example line of questioning for the OJT trainer to determine his level of knowledge concerning activity OJT procedures, might proceed as follows: � What training requirements must you fulfill prior to being qualified as an OJT Trainer? � Do you use any forms that are later used to debrief the trainee on their performance? � How do you know whether or not the trainee understands the requirements for the task at hand? � What actions are taken if an unusual situation arises? � How are you evaluated as an OJT trainer? Who does the evaluations? How often are you evaluated? � Do you have any requalification requirements/periodicity? If so, what are they? As evidenced, the line of questioning could go on at great length. While the answers to such questions will lead the assessor to his next question, it is a good idea to jot down several questions prior to starting the interview. After completing an interview, the information gathered should be evaluated to determine if there

are any apparent deviations from activity requirements. These leads can normally be validated or disproved using one or more of the following: � Additional observations of on-shift training activities � Interviews with qualified trainers and unqualified operators � Interviews with training department personnel DOE-EM-STD-5505-96 57 � Interviews with activity managers, operations supervisors, and shift supervisors � Reviews of operating and program procedures 6.5.5 Document Review. Examples of documents that could be reviewed when assessing On- Shift Training include: � Site or activity training procedures and policy � Supervisor, OJT trainer, and trainee qualification cards � Training Implementation Matrix (TIM) Prior to observing OJT training, the assessor may spend some time ensuring that he is familiar with the procedures and policies governing OJT training. For example, by reviewing an activity's OJT training syllabus, an assessor would be able to understand the requirements that the OJT trainer must fulfill during the training session. For example, these requirements could include the following: � Pre-job brief which might include warnings, cautions, prerequisites � Communication requirements between the trainee and the trainer. � Control of the trainee by the OJT trainer The assessor should also become familiar with the procedure(s) being used by the instructor and trainee to perform their tasks. By observing the OJT trainer in action, the assessor will be better able to evaluate what should occur before, during, and after the training session. 6.6 Assessing Investigation of Abnormal Events. 6.6.1 Discussion. In general terms, the abnormal event investigation program is necessary to investigate abnormal events thoroughly, identify the root cause of events, ensure all necessary notifications are completed, comply with DOE requirements, and ensure appropriate corrective action steps are established to minimize the chance of event recurrence. DOE-EM-STD-5505-96 58 6.6.2 Preparing to Assess Investigation of Abnormal Events. To develop an assessment plan for Investigation of Abnormal Events, assessors should first review the activity-specific graded- approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. Assessment of this area presents several challenges in that it might appear difficult to observe the process in action, but with proper planning and coordination between the team leader and activity management, assessors can collect enough data points to determine how effectively the activity has implemented the guidelines of this area of conduct of operations. 6.6.3 Observation. While assessment of investigation of abnormal events requires an emphasis on document reviews and interviews, it is possible to perform an observation of an on-going investigation. However, before allowing an assessor to observe the investigation process, the team leader should receive concurrence from the activity manager in order to minimize any distractions to the investigation team. Observations could include the following: � Management/supervisory response with respect to classification and notification of a pre-

planned drill event � Observation of an on-going investigation of a previously identified and reported event Observation of this area is extremely difficult, but in the absence of an on-going investigation, an assessor can walk through the necessary steps required for an actual investigation. Whether observations are made during an actual investigation or during a walkthrough, the following methodology can be used: � Determine the person responsible for leading the investigation. Review this person's qualifications as the investigation team leader. DOE-EM-STD-5505-96 59 � Observe the person responsible for leading the investigation during the investigative process. � Observe the investigation that is in progress. Determine if it is being conducted in accordance with activity procedures and DOE Directives. Specifically, observe the process used to conduct the investigation (i.e., reconstruction of the event, event analysis and evaluation, root cause analysis, corrective action determination). Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. 6.6.4 Interview. The following personnel may be interviewed to assess Investigation of Abnormal Events: � Selected members of the investigation team � Shift supervisors � Activity managers and operations supervisors � Training department personnel A line of questioning for a member of the investigation team might proceed as follows: � As a member of the investigation team, what are your responsibilities? � Are these responsibilities delineated in writing? � What training have you had in the area of root cause analysis? � Briefly explain the process employed in performing an investigation. � How are lessons learned from investigations used to help prevent recurrence of similar events? � How does management determine the level of investigation required for a particular event? � Discuss how events are reconstructed. DOE-EM-STD-5505-96 60 After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved using one or more of the following: � Interviews with additional operators and shift supervisors � Interviews with additional investigation team members � Additional interviews with the activity manager or operations supervisor � Reviews of additional investigation reports � Reviews of program and policy procedures 6.6.5 Document Review. Documents that can be reviewed to assess this area include: � Site or activity procedures and policies on investigation of abnormal events � Training and qualification requirements for personnel involved in investigating reportable events � Historical event report files � Trends and analysis of previous events � Lessons learned file � Abnormal event corrective action plans The best way to evaluate an investigation program is to observe it in action. Another effective method is to take an investigative report and associated records and review them. Items that an assessor could look for when conducting reviews include:

� The list of personnel contacted during the event. Were all personnel involved in the event contacted to gain a complete picture of the event? � The root cause of the event. Does it make sense? � The overall format and content of the report. Is the report logically written, indicating that the investigation was performed in a systematic fashion? Is there evidence of management review and approval? � The thoroughness of the investigation. Did the investigator exhaust all avenues of potential DOE-EM-STD-5505-96 61 information in the course of the investigation and consider multiple causes? � Indications of investigator bias as a result of preconceived ideas about what the investigation results should be. � The event history. Was a complete and accurate event history constructed as part of the investigation? By reviewing the items listed above, an accurate picture of the capability of one investigator can be drawn. This, however, may or may not be representative of the site. To get a better picture of the program as a whole, a second report done by a different investigator should be reviewed to determine if there is consistency in the overall site or activity program. This type of review can be time consuming, therefore, assessors may want to pursue only a few aspects of this review in detail. 6.7 Assessing Notifications. 6.7.1 Discussion. Notifying DOE personnel following certain events or conditions helps ensure the safety and efficiency of DOE facilities. Timely notification of appropriate DOE personnel and other agencies, when required, is employed to ensure that the facility is responsive to public health and safety concerns. For events that require notification of DOE personnel (and when appropriate, state and local officials), it is essential that information be gathered and transferred in a systematic, controlled method. Procedures that define responsibilities and provide for adequate documentation are used to control the process and ensure that the notification procedure is effective. Following notification, the DOE can trend and analyze the reported conditions in an effort to focus needed resources or direction. Additionally, central analysis allows for the dissemination of lessons learned to other facilities where similar potential exists for occurrence of such events. DOE-EM-STD-5505-96 62 The final purpose of a prompt notification system is to demonstrate responsiveness to the health and safety of the public. Since notifications is one of the first steps following a reportable incident or event, assessment of this chapter can be accomplished in conjunction with Investigation of Abnormal Events. 6.7.2 Preparing to Assess Notifications. To develop an assessment plan to assess Notifications, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.7.3 Observation. Assessment of notifications could be performed in conjunction with a pre- planned activity drill. If, however, a drill is not performed during the course of the assessment, a walkthrough of an activity's notifications procedure could be an adequate substitute.

To observe the notification process, an assessor should observe an activity drill or actual event. During these situations, the following items might be observed: � The use of a procedure or checklist by the person responsible for making the notification � Notification timeliness as required by DOE O 232.1 � The accuracy of the notification contact list Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. DOE-EM-STD-5505-96 63 6.7.4 Interview. In general, activity personnel who are directly or indirectly associated with the notification process should be interviewed to verify their familiarity with the requirements of activity procedures. A list of personnel who might be interviewed includes: � Activity operators � Shift supervisors � Activity managers and operations supervisors � Occurrence Notification Center (ONC) Supervisor An example line of questioning for the activity manager or operations supervisor might proceed as follows: � Briefly describe how the notification process is executed. � As the Operations Supervisor, what role do you play in the notification process? � What training is required to properly execute your responsibilities for notification of abnormal events? � What documentation is required while performing the notification procedure? � Are there procedures that aid you in notifying the appropriate personnel? If so, how detailed is the notifying chain? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Interviews with several more operators � Interviews with shift supervisors � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures 6.7.5 Document Review. Examples of documents that could be reviewed when assessing notifications include: DOE-EM-STD-5505-96 64 � DOE O 232.1 � The activity programs and procedures which spell out the responsibility and authority for making notifications � Activity notification records, to verify the completeness of information as required by activity procedures and DOE O 232.1 � Filed reports, to ensure that all reportable aspects of the event were identified � Activity operating logs and records, to determine if any reportable events were not properly notified � Notification lists � Emergency plan A good starting point for reviewing documents is the site or activity procedure for making notifications related to reportable events. These documents should be checked to ensure that they provide a structured methodology that allows the proper people to be notified in a timely fashion concerning any events that warrant invoking the procedure. Some key elements to look for are: � Specific responsibilities for conducting notifications � A list of events and conditions that require notifications � Identification of who should be notified, priorities, and time requirements to be met � Definition of records that must be kept to document notifications such as reason, time

accomplished, and personnel contacted Using the list of events and conditions requiring notifications to be made, a check of activity logs and records will sometimes reveal occurrences of reportable conditions that were not handled within the notification system. A check may also be made of the list of personnel to be notified to ensure that it is complete and that it contains up-to-date telephone numbers or other instructions for reaching the designated personnel. Finally, there should be a log of past notifications. A review of these documents should indicate whether notifications are DOE-EM-STD-5505-96 65 accomplished consistently and completely. 6.8 Assessing Control of Equipment and System Status. 6.8.1 Discussion. The purpose of control of equipment and system status is to ensure that activity operations personnel know the status of all equipment and systems and maintain appropriate control at all times. This element of conduct of operations provides an overall perspective on control of equipment and system status and discusses the use of a broad range of administrative control systems and programs such as lockouts and tagouts, equipment and system alignments, equipment deficiency identification, and work authorization. 6.8.1.1 Approach. Since the observations, interviews, and document reviews conducted to assess Control of Equipment and System Status will normally provide leads in other elements of conduct of operations, such as Lockouts and Tagouts and Independent Verification, the assessment of equipment control can be combined with these elements of conduct of operations. This "combined" approach is presented in the following sections. 6.8.2 Preparing to Assess Control of Equipment and System Status. To develop an assessment plan for Control of Equipment and System Status, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.8.3 Observation. Observations should be conducted during walkthroughs and tours with operators and shift supervisors, and during the conduct of special activities. The activity plan of the day and plan of the week should be reviewed to determine the time and place for observing DOE-EM-STD-5505-96 66 operations. The following situations are good candidates for conducting observations: � Maintenance operations that require interface or coordination with other shift positions and the use of lockouts and tagouts � Startup and shutdown of systems and equipment that require component alignment � Process or rate change operations � Installation of temporary equipment or system modifications � Equipment and system testing and return to service � Equipment reported and placed out of commission There are several aspects of the program that can be assessed during these operations. For example, while observing the shutdown of a system or a component to conduct preventive maintenance, the following items might be observed: � Performance and documentation of a system alignment upon shutdown, prior to testing, and before return to service

� Authorization, communication, and documentation of status changes from system shutdown to return to service � Preparation, review, and approval of maintenance and testing documentation � The preparation, review, approval, documentation, and placement of locks and tags � Independent verification of locks and tags, and system alignments Observing the activities and operations discussed above should be supplemented with several activity walkthroughs and tours with operators, maintenance personnel, and shift supervisors. During walkthroughs, the following additional aspects of equipment control can normally be evaluated: � Documentation of and compliance with operational limits for equipment and systems � Control of alarm status and actions taken to monitor equipment parameters when alarms are deenergized � Administrative control systems used to review, approve, document, and audit the installation DOE-EM-STD-5505-96 67 of temporary modifications � Distribution and control of equipment and system documents such as procedures, drawings, and specifications � Equipment deficiency identification, documentation, and communication � Authorization and isolation of maintenance Operators, maintenance personnel, and shift supervisors from each shift should be observed during the assessment. If possible, the observation of a particular shift position should start at shift turnover since vital information about the operating status of equipment and systems, and activities and maintenance in progress, is passed between shifts at this time. To observe a control area operator, the following methodology could be used: � Observe the shift turnover conducted between the on-coming and off-going operators. Pay attention to the information passed verbally between the operators concerning activities planned or in progress, status of activity systems and equipment, and any abnormal conditions that exist in the activity. � Attend the shift crew briefing. Observe the information exchanged. � Observe the operator as duties are performed. Observe the items listed in section 6.8.3 as well as the interface between this operator and other shift positions, and how the operational status of activity equipment and systems is controlled. � At the end of the shift, observe the turnover process between this operator and the oncoming operator. Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. A similar process can be used to observe roving operators and shift supervisors, DOE-EM-STD-5505-96 68 as well as activities and drills conducted by the activity. 6.8.4 Interview. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and validate leads. The following personnel might be interviewed during the assessment: � Operators and maintenance department personnel � Shift supervisors or foremen � Activity managers and operations supervisors � Activity operations staff � Training department personnel Most of the interviews accomplished to assess equipment control should be conducted with

operators, maintenance personnel, and shift supervisors. Whenever possible, these interviews should occur during observations of activities, and activity walkthroughs and tours. To pursue leads, scheduled interviews with the activity manager and training department personnel may also be necessary. For example, while observing an operator beginning at shift turnover, the following types of questions could be used to assess level of knowledge: � Describe the process used to identify, document, and communicate equipment deficiencies. Who is responsible for identifying deficiencies? How are they corrected? � What are your operational limits? How do you document compliance? � When and how are equipment and system alignments conducted? How are they documented? � What actions do you take when alarms are temporarily disabled or deenergized? � Describe the process used to review, approve, document, and audit the installation of temporary modifications. � How is a lock and tag placement accomplished? What are your responsibilities? Describe the process used to review, approve, place, and remove locks and tags. When are locks, "Danger," and "Caution" tags used? Why are they different? What information is contained DOE-EM-STD-5505-96 69 in the lockout and tagout log? How are periodic reviews conducted? How do you record the discrepancies that you find? Describe the process used to resolve these discrepancies. How is this documented? What training have you had concerning how to do lockouts and tagouts? How was the training conducted? Have you received any retraining? Describe the lockout and tagout notification process. � How is independent verification of lockouts and tagouts and system alignments accomplished and documented? What training have you received regarding how to do independent verification? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Tours and interviews with operators � Observations of and interviews with control area operators � Tours of the activity and interviews with shift supervisors � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures � Interviews with training and maintenance department personnel A similar methodology to the one described above can be used to observe other activity operations and drills, considering that the list of interview candidates and sample questions provided may not be all inclusive. 6.8.5 Document Review. A list of documents that should be reviewed during the assessment includes: � Program and operating procedures � Equipment deficiency log � Equipment and system alignment checksheets DOE-EM-STD-5505-96 70 � Maintenance work requests and completed work packages � Temporary modification control documentation � Lockout and tagout log � Shift supervisor and operator logs Most document reviews will be conducted to follow up on equipment deficiency or equipment control issues that were observed during a walkthrough or based on information gained through personnel interviews. For example, if a piece of equipment was noted to be shut down during a walkthrough, the following documents could be checked to ensure the status is being tracked in the proper fashion: � Equipment deficiency log

� Lockout and tagout log � Shift turnover checklists � Maintenance work requests, and maintenance work packages 6.9 Assessing Lockouts and Tagouts. 6.9.1 Discussion. The purpose of Lockouts and Tagouts is to provide a method for equipment status control through component tagging or locking which should protect personnel from injury, protect equipment from damage, maintain operability of plant systems, and maintain the integrity of the physical boundaries of plant systems. If there is a potential for equipment damage or to inadvertent activation of equipment, a facility Lockout/Tagout should be established and used. The Lockout/Tagout program should provide for independent verification of the removal from service and the restoration to service of safety-related and other facility equipment. The Lockout/Tagout Program is intended to meet the requirements of 29 CFR 1910, as a minimum. 6.9.1.1 Approach. As discussed in Assessing Control of Equipment and System Status, DOE-EM-STD-5505-96 71 lockouts and tagouts are an integral part of equipment control. This combined approach of assessing Lockouts and Tagouts with Control of Equipment and System Status is presented in that section. Since it may be necessary to assess Lockouts and Tagouts separately, the following sections discuss how to assess this area of conduct of operations only. 6.9.2 Preparing to Assess Lockouts and Tagouts. To develop an assessment plan for Lockouts and Tagouts, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.9.3 Observation. Operations of activities available for assessment can be identified via review of the activity's plan of the day or plan of the week. These documents should list the routine operations, maintenance, and special activities that will take place during the assessment. The following situations are good candidates for observation: � Periodic and corrective maintenance of equipment and systems � System testing � Returning a system to operation � Periodic lockout and tagout reviews and inspections � Use of group lockouts and tagouts � Work performed by outside contractors or other outside service personnel There are several aspects of the program that can be assessed during these operations. For instance, while observing periodic and corrective maintenance, the following items can normally be observed: � Identification of energy sources DOE-EM-STD-5505-96 72 � Use of system prints and other documentation � Physical placement of tags and locks � Lockout and tagout log documentation � Tag entries � Notification process Several lockouts and tagouts should be observed during the assessment. Observing the activities and operations discussed above should be supplemented with activity tours and, if necessary, walkthroughs of the approval, placement, verification, and removal of lockouts and tagouts with operators, maintenance personnel, and shift supervisors. To observe an operation that involves placing and verifying tags, the following methodology could be used:

� Observe the preparation, review, and approval process for a lockout or tagout conducted by the responsible supervisor. � Observe the independent verification of the tagout. � Review the lockout or tagout record sheet, log index, and accompanying tags. � Observe an operator as the required locks or tags are hung. � Verify the personnel placing and verifying tags are properly trained and qualified. � Observe the notification process. Once the observation has been completed, compare what was observed to the requirements of the program procedure to identify any deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. A similar process can be used to observe other aspects of the program. DOE-EM-STD-5505-96 73 6.9.4 Interview. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and validate leads. The following personnel could be interviewed during the assessment: � Operators and maintenance department personnel � Shift supervisors or foremen � Activity managers and operations supervisors � Activity operations department staff � Training department personnel Interviews conducted to assess the Lockout and Tagout Program should be aimed at determining the level of knowledge and skills of: � Personnel who conduct maintenance and those that place, verify, and remove locks and tags, (i.e., operators and maintenance personnel) � Personnel who review, approve, and manage lockouts and tagouts, (i.e., shift supervisors, activity managers, and operations supervisors) � Personnel who work at the activity but do not review, approve, place or remove locks and tags (operations department staff) Based on the information gathered from these interviews, the assessor should determine whether adequate knowledge of the program is demonstrated by personnel at all levels of the activity. The assessor can normally conduct a cursory review of lockout and tagout training. If deficiencies appear to exist at one or more levels, a thorough review of training and other aspects of the program must be conducted to get to the root of the problem. Sample questions for interviews with operators, maintenance personnel, shift supervisors, activity managers, and operations supervisors include: � What is the purpose of the Lockout and Tagout Program? What are your responsibilities DOE-EM-STD-5505-96 74 under the program? � How do you prepare a lockout or tagout? � Describe the review and approval process. � What information is contained in the lockout and tagout log? � How is lock and tag placement accomplished? � How do you review lockouts and tagouts? How do you record the discrepancies that you find? Describe the process used to resolve these discrepancies. How is this documented? � When are locks, "Danger," and "Caution" tags used? Why are they different? � What training have you had concerning how to do lockouts and tagouts? How was the training conducted? Have you received any retraining? � Describe the lockout and tagout notification process. The following questions could be used to assess the level of knowledge of personnel who work at the activity but do not review, approve, place, or remove locks and tags:

� What is the purpose of the Lockout and Tagout Program? What are your responsibilities under the program? � When are locks, "Danger," and "Caution" tags used? Why are they different? What are your responsibilities with regard to locks and tags? � What training have you had concerning lockouts and tagouts? How was the training conducted? Have you received any retraining? Finally, if apparent deficiencies exist in the level of knowledge of activity operations personnel, training department personnel should be interviewed. The following questions could be asked to help evaluate lockout and tagout training: � How is lockout and tagout training accomplished? Is retraining accomplished? What is the periodicity for conducting training? � Describe the process used to determine the level of training required for operations personnel. DOE-EM-STD-5505-96 75 � How do you ensure that the required level of knowledge is obtained by operations personnel as a result of this training? � Are deficiencies discovered during lockout and tagout reviews factored into training? Describe this process. � How is the training documented? � Describe the process used to ensure that the appropriate personnel are retrained. The above list of interview candidates and questions is not all inclusive. Assessors must use additional interviews, observations, and document reviews as necessary to thoroughly investigate any apparent deviations, and determine if programmatic breakdowns or widespread problems exist. Note that an assessor may be assigned to training. If it is necessary to assess lock and tag training, these interviews would be accomplished by the training assessor. 6.9.5 Document Review. Examples of documents that may be reviewed when assessing lockouts and tagouts include: � Lockout and tagout program procedures � Active and completed lockout and tagout record sheets � "Danger" and "Caution" tags � Lockout and tagout authorization log index � Key control documentation, and � Training records Periodic reviews of the lockout and tagout program should be conducted by the activity, and evidence of the process should be apparent in the lockout and tagout program documentation. Checking for evidence of a review program, observing a review in process, or personally performing part or all of a review is a good technique for an assessor to use to obtain an accurate picture of the activity's program. DOE-EM-STD-5505-96 76 The following is a general description of how an activity might conduct a review of the lockout and tagout program. The review should start with the index or log. A check is usually made to verify that there is a tagout sheet for every active entry in the index. If there is not a one to one correlation, then a tagout sheet is missing or was cleared and not removed from the index. Secondly, each tagout sheet should be assigned to an operator qualified to participate in the lockout and tagout program. The operator should walk through every item on the tagout record sheet and verify that all locks or tags are still hanging on the proper device. When the operator completes the review it should be documented on the record sheet. Any discrepancies should be noted along with the corrective action that was taken. When all tagout sheets have been reviewed, an entry should be made in the tagout index to record the review and summarize any discrepancies that were found and then corrected.

The following is an example of how a portion of the review process can be conducted by an assessor. A review of a specific lockout or tagout could be done by starting with either the lockout and tagout log or with the locks and tags that have been placed. The first method is described below: � Select a lockout or tagout record sheet from the log. � Review the record sheet and log index. Check for periodic audits and reviews to determine whether they are conducted, and if the discrepancies noted are documented and corrected. � Walk through the activity and locate each lock or tag indicated on the record sheet. � Check that each lock and tag is properly placed, and that all required information on the tags is legible and complete. Compare the information on the record sheet and the tag. Once the review has been completed, compare what was observed to the requirements of the program procedure to identify any leads. DOE-EM-STD-5505-96 77 6.10 Assessing Independent Verification. 6.10.1 Discussion. Independent Verification is the act of checking that a given operation conforms to established operational criteria, as well as checking a component position independent of activities related to establishing the component's position. A comprehensive Independent Verification Program will identify components to be included in the program, define when Independent Verification is required, and prescribe the methods to be used when performing Independent Verification. If implemented effectively, Independent Verification reduces the chance that human error will cause an operational failure. Independent auditing of operations should confirm that established operational requirements are met. 6.10.1.1 Approach. As discussed in Assessing Control of Equipment and System Status, Independent Verification is an integral part of equipment control. This combined approach of assessing Independent Verification with Control of Equipment and System Status is presented in that section. Since it may be necessary to assess Independent Verification separately, the following sections discuss how this could be accomplished. Independent verifications are conducted in a manner such that each check constitutes an actual identification of the component and a determination of both its required and actual positions. Checks are performed minimizing interaction between the personnel operating components and those performing the independent verifications. 6.10.2 Preparing to Assess Independent Verification. To develop an assessment plan for Independent Verification, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment DOE-EM-STD-5505-96 78 techniques. 6.10.3 Observation. The following special activities and operations could demonstrate Independent Verification: � The alignment of valves and switches in preparation for system startup, maintenance, testing, and prior to return to service � Verifying the status of a system or component which is in doubt Some of these events will be listed in the activity plan of the day or plan of the week. Other

situations that require Independent Verification will be event driven. If an Independent Verification is not scheduled to occur during the assessment, the assessor should ask the activity manager to arrange a demonstration or a walkthrough of the procedure. When observing Independent Verification, the assessor should see some form of time-distance spacing between the two verifiers. Remember that an independent check is required. Time spacing means that the two verifiers should not check a component at the same time. Distance spacing means that neither verifier should be able to see the other check the component. To observe two operators conducting a system alignment that requires Independent Verification, the following methodology could be used: � Accompany the first operator as he aligns the system as listed on the checksheet. Observe the use of the checksheet and the operator's skills and abilities concerning locating and operating system components. � Once the system has been aligned by the first operator, observe any communication between the first and second operators, and follow the second operator as he verifies the position of system components. Pay particular attention to the use of the checksheet, the operator's skills and abilities concerning locating system components and verifying their component position, and the independence of the second check from the original alignment. DOE-EM-STD-5505-96 79 Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. A similar process can be used to observe other activities that require Independent Verification such as the placement of a lockout or tagout. 6.10.4 Interview. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and clarify leads. The following personnel might be interviewed during the assessment: � Operators who are qualified to conduct Independent Verification � Shift supervisors � Activity managers and operations supervisors Most of the interviews accomplished to assess Independent Verification should be conducted with operators, maintenance personnel, and shift supervisors. Whenever possible, these interviews should occur during observations of operations, and activity walkthroughs and tours. To pursue leads, scheduled interviews with the activity manager and training and maintenance department personnel may also be necessary. For example, the following questions could be used to evaluate an operator's level of knowledge regarding the program: � When were you trained on how to conduct Independent Verification? � When are you required to conduct Independent Verification? � How do you perform Independent Verification? � How do you independently verify a throttle valve position? � How are you evaluated on conducting Independent Verification? DOE-EM-STD-5505-96 80 The following question could be used to interview managers: � How do you train operators in Independent Verification? � When do you require Independent Verification be conducted? � How would you independently verify a throttle valve?

� How do you ensure that an operator properly performs Independent Verification? � What systems require Independent Verification? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Tours with several more operators � Tours of the activity and interviews with shift supervisors � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures � Interviews with training and maintenance department personnel 6.10.5 Document Review. A list of documents to be reviewed during the assessment would include: � Independent Verification program procedures � Operating logs and system lineup checksheets � Quality assurance plan Assessors should first determine when and on what systems Independent Verification is required by reviewing the appropriate administrative procedure, and then review activity logs to find events that require Independent Verification. If documentation of Independent Verification is not required, some other technique for checking that verification has been completed will have to be used (i.e., interviews). DOE-EM-STD-5505-96 81 Another area to investigate is Independent Verification training. The activity should have a reference that instructs personnel on how to verify the position of various activity components. In addition, personnel should receive training on how to perform Independent Verification. Training records are usually held by a supervisor or by the training department. Note that an assessor may be assigned to training. If it is necessary to assess Independent Verification training, these interviews may be accomplished by the training assessor. 6.11 Assessing Logkeeping. 6.11.1 Discussion. The purpose of logkeeping is to record activity status and events as required to provide an accurate history of activity operations. A log is defined as "a narrative sequence of events or functions performed at a specific shift position." This limited standard, on the other hand, does not attempt to differentiate between narrative records maintained by various shift positions, and records such as Resource Conservation Recovery Act (RCRA) or operator round/tour inspection sheets, radiological survey records, equipment deficiency logs, and lockout and tagout logs. There are two reasons for this: (1) the guidelines in each area of conduct of operations address fundamental principles such as timeliness of recordings and legibility that should be applied to the maintenance of inspection sheets and other records as well as narrative logs, and, (2) round/tour inspection sheets and other non-narrative records frequently contain a narrative section, making differentiation between logs and inspection sheets unnecessary. 6.11.1.1 Approach. As discussed in Assessing Shift Routines and Operating Practices, Logkeeping is an integral part of shift routines. This combined approach of assessing Logkeeping with shift routines is presented in that section. Since it may be necessary to assess Logkeeping separately, the following sections discuss how to assess this area of conduct of operations only. DOE-EM-STD-5505-96 82 6.11.2 Preparing to Assess Logkeeping. To develop an assessment plan for Logkeeping, assessors should first review the activity-specific graded-approach document for conduct of

operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.11.3 Observation. Observations of Logkeeping should be conducted during walkthroughs and tours of the activity with operators and shift supervisors, and during the conduct of special activities and drills. The following situations are good candidates for conducting observations: � Maintenance operations that require interface or coordination with surveillance operators, control area operators, and shift supervisors � Startup and shutdown of systems and equipment � Process or rate change operations � Activity drills There are several aspects of logkeeping that can be assessed during these operations. For example, while observing an operator as he performs a system or component startup, the following items might be observed: � Timeliness of entries � The type, scope, and format of entries made to document the startup by all shift positions involved in the procedure � Legibility of entries and method used to make corrections � Consistency in the use of logs and information recorded between shift positions � Periodic supervisory review Observing the activities and operations discussed above should be supplemented with several activity walkthroughs and tours with operators and shift supervisors. During walkthroughs, all DOE-EM-STD-5505-96 83 logs maintained by operators, maintenance personnel, and shift supervisors should be assessed. While touring through the activity with an operator, the following methodology could be used: � During the tour, observe the timeliness, legibility, etc. of the entries made in the round/tour inspection sheet. � Review any operational records that are kept by the activity. Examples include the control area and shift supervisor's logs, equipment deficiency logs, the lockout and tagout log, and radiological survey logs and records. Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. 6.11.4 Interview. Most interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and clarify leads. The assessment plan should identify the initial interviews necessary to start the assessment. The following personnel might be interviewed during the assessment: � Operators and maintenance department personnel � Shift supervisors or foremen � Activity managers and operations supervisors � Radiological control technicians, and � Activity operations staff While observing a shift supervisor or worker, the following types of questions could be used to assess his level of knowledge about logkeeping: DOE-EM-STD-5505-96 84 � What types of information do you record in your log? What guidance have you been given on the type, scope, and format of log entries?

� How do you correct log errors? Is this method used by all shift positions? � Do you review any logs that are maintained by other shift positions? If so, how do you conduct and document these reviews? Are logs reviewed by activity management? How are these reviews conducted? � Where are completed logs kept? Describe the process used to retrieve completed logs. The information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Walkthroughs and tours with operators � Observations of and interviews with control area operators � Additional tours of the activity and interviews with shift supervisors � Interviews with radiological control technicians � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures 6.11.5 Document Review. A list of documents to be reviewed during the assessment should be included in the assessment plan. Examples of documents that should be reviewed include: � Program and operating procedures � Shift supervisor and operator logs and round/tour inspection sheets � Radiological survey records � Shift turnover checklists � Lockout and tagout logs � Equipment deficiency logs � Maintenance work request logs � Occurrence reports DOE-EM-STD-5505-96 85 A correlation of information recorded in different places during the same period may prove useful to identify communication or teamwork problems. During assessment planning, the administrative procedure for logkeeping should be reviewed to gain familiarity with site requirements. This background knowledge is a necessary prerequisite to making an intelligent evaluation of the logkeeping program. The remainder of the document reviews necessary to assess logkeeping can be conducted in conjunction with observations. For example, while touring the activity with an operator, a review of documents such as shift supervisor and operator logs, round/tour inspection sheets, operations procedures, radiological surveys, lockout and tagout logs, and equipment deficiency logs can normally be accomplished. The individual logs and round sheets should be examined to ensure that the elements required by administrative procedures are contained in the various documents and that they provide enough information to accurately document the history of activity operations. In another example, a return to an area may be made to make a correlation of several related logs and records to ensure they are properly prepared and meet requirements when taken as a whole. This often yields a larger view of the organization and may point to teamwork or communication problems. For example, groups often log conflicts with or failures to properly work with other groups as a way of covering their own situation. These sorts of entries in the records should prompt follow up. In another example of an assessment technique, a review of a log or record for an extended period of time often gives insight to other types of problems. If it is difficult to get something repaired or corrected, or if there is sufficient support or follow up for deficiencies or inoperable equipment, it is often recorded in the logs where the appeal to make corrections was documented. Such appeals, if repeated, often signal other problems that warrant attention. DOE-EM-STD-5505-96 86

6.12 Assessing Operations Turnover. 6.12.1 Discussion. Operations turnover provides on-coming operators and supervisors with an accurate picture of the overall status of the activity. The turnover process should include a comprehensive review of appropriate written and visual information, be guided by a checklist, and complemented by a discussion between the off-going and on-coming shift personnel. Although the guidelines in this section specifically address operations turnover at facilities that operate 24 hours a day, the spirit and intent of the guidance should be followed by activity management at facilities that operate on a single-shift basis. In many cases, the turnover in a day-to-day operation is more difficult than that encountered in 24-hour shift operations. An example is often observed in activities where the person who normally initiates the daily activities is unavailable and an alternate person feels the need to initiate the activity. This is common when starting up laboratories or laboratory equipment and for sampling or inspection rounds of permitted storage. Breakdowns of turnover information may be of considerable concern in such cases. (In these cases, there is often no continuity of turnover) 6.12.1.1 Approach. As discussed in Assessing Shift Routines and Operating Practices, Operations Turnover is an integral part of shift routines. This combined approach of assessing shift turnover with shift routines is presented in that section. Since it may be necessary to assess Operations Turnover separately, the following sections discuss how to assess this area of conduct of operations only. 6.12.2 Preparing to Assess Operations Turnover. To develop an assessment plan for Operations Turnover, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable DOE-EM-STD-5505-96 87 activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. 6.12.3 Observation. Observations of Operations Turnover should be conducted during walkthroughs and tours of the activity with operators, maintenance personnel, radiological control technicians, and shift supervisors. If possible, personnel from each group or shift should be observed during the assessment. To observe the shift turnover process at an activity, the following methodology could be used: � Observe the shift turnover conducted between an on-coming and off-going operator. Pay attention to the information passed verbally between the operators concerning activities planned or in progress, status of activity systems and equipment, and any abnormal conditions that exist in the activity. Review any narrative logs, round/ tour inspection sheets, and any other documentation used in the turnover process. � Attend the shift crew briefing. Observe the process used and the information exchanged. Compare the information disseminated at the briefing with the information that was passed between the operators during their turnover. � Accompany the operator as he performs his duties. Look for any activities or operations in progress that were not discussed or briefed at turnover. Also pay attention to the status of

equipment and systems to identify any differences between what was briefed and what you observe. � At the end of the shift, observe the turnover process between this operator and the on-coming operator. Once the observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursue these leads using additional observations, interviews, and document DOE-EM-STD-5505-96 88 reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. A similar process can be used to observe control area operators, shift supervisors, maintenance personnel, and radiological control technicians conducting operations turnover. 6.12.4 Interview. Interviews of operations and maintenance department personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and clarify leads. The following personnel might be interviewed during the assessment: � Operators and maintenance department personnel � Shift supervisors or foremen � Activity managers and operations supervisors � Radiological control technicians � Activity operations staff � Training department personnel Most of the interviews accomplished to assess operations turnover should be conducted with operators, maintenance personnel, and shift supervisors. Whenever possible, these interviews should occur during observations of shift turnover, activity operations and activities, and during activity walkthroughs and tours. To pursue leads, scheduled interviews with the activity manager and training department personnel may also be necessary. For example, while observing a roving operator beginning at turnover, the following types of questions could be used to assess the effectiveness of the shift turnover process: � What guidance have you received from the shift supervisor or activity manager concerning how to conduct shift turnover? Do you feel that the process used at turnover supports your needs? � How do you determine what information should be passed at turnover to the on-coming operator? Are you required to use a turnover check sheet to aid the process? If so, what type DOE-EM-STD-5505-96 89 of information is recorded on the check sheet? How is this information used during the shift? � What documents do you review during shift turnover? How does the turnover check sheet aid in the document review process? � Are there any control panels that you walk through during shift turnover? If so, how do you do this and what information does it provide you? � Do you conduct a verbal turnover to the on-coming operator? What type of information do you pass? � Is a shift crew briefing conducted as part of the turnover process? When is it conducted? What type of information is passed during the brief? Who attends the brief? � How do you conduct turnover for reliefs that occur during the shift? What type of information do you pass to your relief? What guidance have you received from activity management concerning reliefs during the shift? The information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Observations of and interviews with several more operators

� Tours of the activity and interviews with shift supervisors � Interviews with radiological control technicians � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures � Interviews with training department personnel A similar methodology to the one described above can be used to observe and interview control area operators, maintenance personnel. radiological control technicians, and shift supervisors. 6.12.5 Document Review. Examples of documents that should be reviewed include: � Program and operating procedures DOE-EM-STD-5505-96 90 � Operating logs and round/tour inspection sheets � Shift turnover checksheets The site or activity administrative procedures should be reviewed as part of pre-assessment preparations to identify the actions which must be completed to conduct a proper operations turnover. The remainder of the document reviews necessary to assess operations turnover can be conducted in conjunction with observations and interviews. For example, while observing a turnover between shift supervisors, a review of documents such as operating logs and turnover checklists can be conducted. 6.13 Assessing Operations Aspects of Facility Chemistry and Unique Processes. 6.13.1 Discussion. This section describes the operational monitoring of activity chemistry or unique process data to ensure that parameters are maintained. The term "unique processes" includes all parameters that require monitoring for process control including chemistry parameters. The guidelines of this section use chemistry as an example. The requirements for activity-specific process control policies may be detailed in one or more administrative procedures. It is expected that operators understand the parameters that are recorded, how process changes affect those parameters, and the actions to be taken to correct process parameters when they approach an out of specification condition. When matrixed organizations, such as chemistry labs, are responsible for measuring process control parameters, operations personnel should ensure that sample results are received in a timely manner so that they can be applied for process control. 6.13.1.1 Approach. As discussed in Assessing Shift Routines and Operating Practices, Operations Aspects of Facility Chemistry and Unique Processes are an integral part of shift routines. This "combined" approach of assessing Operations Aspects of Facility Chemistry and DOE-EM-STD-5505-96 91 Unique Processes with shift routines is presented in that section. Since it may be necessary to assess Operations Aspects of Facility Chemistry and Unique Processes separately, the following sections discuss how to assess Operations Aspects of Facility Chemistry and Unique Processes only. 6.13.2 Preparing to Assess Operations Aspects of Facility Chemistry and Unique Processes. To develop an assessment plan for this area of conduct of operations, assessors should first review the activity-specific graded-approach document for conduct of operations and determine which guidelines are applicable at the activity. This will allow them to determine the scope of the assessment and which applicable activity policies and procedures to review when developing the assessment plan. Sections 5.4 and 5.8 provide a detailed explanation of assessment planning and the use of assessment techniques. Drills are particularly effective tools in the assessment of this area. At least, they provide the

observations and indications from which the assessors can pursue more fundamental information. Activity and site organization as well as the activity's purpose will play a large role in assessment planning. Assessors should determine if and where matrixed organizations interface with operations personnel and how those organizations affect process control. The activity description and tour should provide an assessor with an understanding of the activity process. When the assessment begins, noticing which indications, such as gauges, meters, or sight glasses, are recorded on round sheets will provide a good starting point. 6.13.3 Observation. The activity's plan of the week or plan of the day should be reviewed to determine if there are major processes occurring that will require the monitoring of parameters. The following are sources of possible observations: � An operator recording and acting on process parameters DOE-EM-STD-5505-96 92 � A technician monitoring parameters or taking samples � An operator performing maintenance that will cause an alarm condition Operators and technicians from each shift should be observed during the assessment. If possible, the observation of a particular shift position should start at shift turnover since vital information about the status of activity processes is passed at this time. An effective way to observe operators is to follow them during their inspection tours. If they are taking round sheets, observe what parameters are monitored and the parameter normal, high, and low values recorded on the round sheet. If the actual parameter value is outside the high/low band, observe what actions the operator takes, if any. If a maintenance action or operator will cause an expected alarm, then are the appropriate operations personnel informed prior to receipt of the alarm? Because matrixed technicians are outside the operations organization, observing the communication of sample results or parameter monitoring will be required. Once again, an effective means of conducting the observation is to contact the matrixed organization, and arrange to follow a technician while sampling or monitoring operations. Observe the technician take the sample or monitor the parameter as before. If the sample requires analysis to obtain a parameter, observe if the results are returned to the operations organization in time to have meaningful process control. Once an observation has been completed, compare what was observed to the requirements contained in activity policies and procedures to identify any apparent deviations from requirements. Pursuing these leads using additional observations, interviews, and document reviews to confirm or disprove them and determine if programmatic breakdowns or widespread problems exist. DOE-EM-STD-5505-96 93 6.13.4 Interview. Interviews of operations personnel should occur during observations and document reviews. Additional scheduled interviews should also be conducted to follow up and clarify leads. The following personnel may be subject to interviews when assessing this area of conduct of operations: � Activity operators � Matrixed technicians � Shift supervisors � Matrixed organization supervisors � Activity managers and operations supervisors The primary focus of interviews should be to determine the level of technical knowledge that is required by each area of conduct of operations, and a working knowledge of the administrative

requirements. For example, the following general questions could be asked of operators: � What are the general requirements when a parameter leaves its normal control band? � Have you ever received data from a matrixed organization (such as a chemist) on which you could not have reacted in a timely manner? � Do you conduct trend analysis? (i.e., look at recorded parameter data to understand why it changed and determine if the trend predicts the parameter leaving the normal band). � Have you ever received an alarm that was caused by maintenance? Were you informed prior to receiving that alarm? General questions to ask matrixed technicians are: � Who do you inform when you are taking samples or monitoring parameters at the activity? � How do you ensure that samples taken at multiple locations are not switched? � Are you required to inform anyone when complete sampling? Who? � Are you required to inform the activity of the results? How? � What are you required to do prior to performing work that will cause an alarm condition? Interviews with operators and technicians are most effectively conducted in conjunction with DOE-EM-STD-5505-96 94 observations. Interviews with shift supervisors, activity managers, and operations supervisors could include the following questions: � Can you remember encountering a process problem that could have been prevented if sample results had been received earlier? � Do you expect your operators to conduct trend analysis? � How do you conduct trend analysis? � How do you ensure that operators understand the activity process? � What actions do you expect operators to take when an out of specification parameter is discovered? After completing an interview, the information gathered should be evaluated to determine if there are any apparent deviations from activity requirements. These leads can normally be validated or disproved by using one or more of the following: � Tours with several more operators � Tours of the activity and interviews with shift supervisors � Interviews with technicians � Interviews with activity managers and operations supervisors � Reviews of operating and program procedures � Interviews with training department personnel 6.13.5 Document Review. A list of documents to be reviewed during the assessment should be included in the assessment plan. Some will be reviewed during assessment planning; others will be reviewed only if pursuing leads requires that they be examined. The list of documents includes: � Emergency preparedness documents and safety analyses � Various administrative policies that detail operator responsibilities for trend analysis, out-of- specification-parameter actions, activity-specific process training, etc. � Various key shift position roundsheets and logs DOE-EM-STD-5505-96 95 � Chemistry Laboratory Quality Assurance/Quality Control Plan � Any procedure with hold points for which a matrixed organization is responsible As noted in the previous list, the required documents for this area of conduct of operations may be distributed among many activity and site administrative policies. The documents to be reviewed during assessment planning are the activity or site administrative policies and procedures. Most of the matrixed organization policies, including QA/QC plans, will not be examined unless pursuing leads draws an assessor into those