Presented at the American Society for Engineering Education 2001 Illinois/Indiana Section Annual Conference, West Lafayette, IN
 

When assessing the effectiveness and efficiency of any system, there must be a standard to which it is measured. This standard is often found in the business world, as well as in educational systems, in the form of a mission or vision statement with supporting goals and objectives. Assessments may then be made by determining measurements for the stated goals and objectives that are then quantifiably evaluated. Although this appears to be simplistic logic, it unfortunately can become a difficult task for individuals if it is not clear which goals and objectives are being evaluated. Such is often the case when organizations create functional layers within themselves requiring the various layers to develop their own individual goals and objectives without a clear understanding of the 'big picture' for the overall organization. This results in a myriad of missions, goals and objectives that may or may not be tied together causing, as a worst case, a lack of focus within the organization, or in the best case a difficulty in reporting on the attainment of goals and objectives.

This paper will focus first on defining the purpose for developing mission statements, goals and objectives at various levels in an organization. It will then explore the various requirements for assessment from internal and external sources within a typical university system. Finally, methods used at Purdue University Calumet (PUC) to develop a tie between the goals and objectives at various levels starting with the program levels within their Manufacturing Engineering Technologies and Supervision (METS) department will be presented.

I. Developing missions, goals, and objectives

Missions, goals and objectives are all components of a total quality environment. A total quality environment involves creating an environment within an organization that focuses on customer needs by getting all employees involved in some way in the strategic operation of the organization as well as focusing on the never ending pursuit for excellence. This process is perhaps described most simplistically as the Deming Cycle [1] (see figure 1). The Deming Cycle has been used over and over since the 1950's when Dr. Edwards Deming first starting teaching the Japanese people about the advantages of using a total quality approach in industry. The cycle has four distinct phases that support Dr. Deming's basic philosophy which is known today as his 14 points for quality management. These four phases are: plan, do, check, and act.

The Deming Cycle

(Figure 1)

Plan refers to developing goals and objectives and the initiatives to accomplish them. Do refers to the actual implementation of the initiatives. Check refers to the measurement or assessment of the initiative result to the intended result as stated in the goal. And finally, Act refers to the establishment of permanency of the initiative if it was successful or the appropriate adjustment if the goal was not attained. As can be seen when following through this cycle, the first stage requires developing missions, goals and objectives. This is in direct support of Dr. Deming's first point of quality management, "create constancy of purpose for improvement of product and service with a plan to become competitive and stay in business [2]." The basic premise of Dr. Deming's first point is that unless everyone in the organization is aware of and buys-in to the main purpose of the organization, they will not be able to satisfy the customer in the end. The best way to achieve this buy-in is to make sure that all employees participate in the goal setting process.

When developing missions, goals and objectives, the first requirement is to define relevant terms so that everyone understands them to be the same thing. The University of North Dakota is one institution of higher education that has been through this process and recommends getting terms straight as a first step for faculty and everyone involved in the process. Some of their definitions follow [3]:

Goal development is a fundamental activity for any organization looking to succeed in a world of competition. Companies that have been successful in delivering high quality service are noted for establishing formal goals relating to service quality. Because services are performances, the goals for service delivery usually are set and measured in terms of human or machine performance. For example, American Express, after analyzing customer complaints, found that timeliness, accuracy, and responsiveness were the important outputs to be achieved. Management then identified 180 goals for different aspects of service quality provided to customers. After the formal goal setting, they developed monitoring devices to evaluate the speed with which telephones were answered, complaints handled, bills mailed, and new applications approved. The goals established by American Express illustrate many of the characteristics of effective goals. They were specific and accepted. They cover important job dimensions. They were reviewed with appropriate feedback, were measurable, challenging but realistic, and match individual characteristics [4].

This basic strategic initiative of goal setting is just as applicable in the education industry as it is in other types of service industries. The cry is the same everywhere: "Do more with less." Organizations are all being asked to maintain or even increase productivity in spite of budget or staff cuts [5].

Major contributors to the development of the strategic concept and to the planning process include Professors Andrews, Christensen, and others in the Policy group at the Harvard Business School. A definition by this group is contained in their highly regarded text on the subject:

II. Developing program goals at PUC that support university goals

Sparked by a self-study conducted during preparation for an accreditation visit in 1999 by the Technology Accreditation Commission (TAC) of the Accreditation Board for Engineering and Technology (ABET), the faculty in the METS department at PUC recognized that there were some gaps in their program goals as compared to their university goals. This led to a more extensive analysis of existing relevant goals at all layers of the organization. The university mission should be the starting point to "define the business the company intends to be in [8]," and the program goals should then be measurable attributes that support the mission with follow-on initiatives to be completed resulting in supportable levels of goal attainment. However, what the METS department faculty found was that these definitions of mission, goals, and initiatives did not necessarily apply. There was not even a clearly visible link of the existing program goals to those of the department (which has 4 programs) to those of the school (4 departments) to those of the university (6 schools).

In fact, when looking at the specific items documented as program goals, what was found was an actual "to do" list which more closely resembled initiatives. This was easily noted when trying to assess the items on the "to do" list, as assessment requires measurement. "To do" lists (or initiatives) are, more often than not, either completed or not completed. There is nothing to actually measure. Goals should be measurable.

When the METS faculty then looked to the department goals, they found they were not as defined earlier in this paper by The University of North Dakota as a lofty aim, but rather they were themselves a list of specific measures. As they were very specific, for example "70% of graduates will communicate effectively," they were either achieved or not. If achieved, there were no existing plans for follow-on improvement or increase to the goal.

Armed with new understanding of goals as a lofty aim that is measurable and conducive to continuous improvement within the organization, the METS department faculty undertook the effort of goals setting for their department. They intended to reestablish a list of goals that were both measurable and linked to their school, and ultimately, their university goals.

This process began by listing and numbering the existing goals of the METS department, its school, and the university. The next step was to match each goal, starting at the department level, to a goal it supported at the next level of the organization. The intent being to establish the link between all goals or, for those without a link, reevaluate the appropriateness of that goal to the mission of the university. To facilitate this effort, a simple "goal matching" multi-column table was created. Below is an example showing goals that are related or supportive of each other at the different levels from the METS department up to the university level:

Goal Matching Matrix - Example

Department	School	University
Goal 1	Goal 1	Goal 1
Goal 2	Goal 1	Goal 1
Goal 6	Goal 1	Goal 1

In this case the university goal 1 is "excellence in education." The school goal 1 is "reward, recognize, and encourage excellence in education." The department goal 1 is to "create and continuously improve programs relating to the focus of the department." The department goal 2 is to "promote excellence in teaching within the department." And department goal 6 is to "apply student learning assessment, program/course assessment, outcomes assessment, benchmarking, and interactive quality improvement concepts to strive for excellence." It is only after going through this process of systematically setting measurable goals that tie into higher level organizational goals that assessment showing progress toward attaining those goals may be conducted.

Furthermore, in program accreditation by TAC of ABET, the requirement from the General Criteria for Accrediting Engineering Technology Programs states that "programs must have written goals consistent with overall institutional goals." The goals that must be assessed are for the program. Note that to simplify, all programs within the METS department at PUC have the same goals. Only the initiatives to attain the goals may differ from program to program.

From an accreditation standpoint, it is a lack of clear goals and objectives that was the most frequently cited finding by evaluators in the 1999-2000 program evaluation cycle according to David Baker, [9] former chair of TAC. The existence of written goals is one of a list of items checked by the accreditation team. The team checks off this requirement along with four other related items among many others on an Engineering Technology Program Evaluation Form (T4) which is intended to represent the general ABET criteria. This approach has been called "bean counting." In addition to being part of ABET requirements, regional accreditation of colleges and universities also have many similar requirements regarding goals.

III. Things are changing in accreditation

In the past, accreditation was largely based on following a list of "dos and don’ts" and making sure the institution conformed to a list of preset criteria. Today, this is no longer the case. For example, the Engineering Accreditation Commission (EAC) of ABET has replaced "bean counting" with student outcomes assessment based criteria as the basis for engineering program accreditation in EC2000. The Technology Accreditation Commission (TAC) is following a similar course in moving to TC2K for engineering technology programs. Both accreditation criteria depend heavily on continuous improvement. TC2K will not be fully implemented for all engineering technology visits until fall of 2004 according to the schedule current at the time this paper is being written. However, even current requirements require program goals consistent with those of the institution, a continuous improvement plan, and evidence of program improvements made as a result.

Regional accreditation, for example the North Central Association (NCA), has also added outcomes assessment requirements. Its new Academic Quality Improvement Project (AQIP) moves away from a one to two year preparation period followed by a team visit every ten years. AQIP requires a yearly continuous improvement report with an on-site evaluation as part of an institutional quality review at least every seven years. AQIP is still voluntary at this point but is the direction of the future, so Purdue University Calumet has joined the project.

In general, accrediting agencies expect the institution, school, department or program to set their own goals and objectives. But progress toward meeting these goals must be assessed and continual progress toward improving the process must be documented. Vision, mission, goals, objectives, strategic initiatives, strategy, and tactics must all support each other. Some simplification of vision, mission, and goals becomes advisable at the program level which is the focus of TAC. The fewer administrative levels with goals and objectives, the easier it is to document progress toward meeting them. This is why the METS department at PUC has a mission statement, goals, and initiatives but has done away with vision, objectives, strategy, and tactics. Moreover, all 4 programs have the same goals. Initiatives result naturally from the continuous improvement process. They may be one-time suggestions from the industrial advisory committee or from students, faculty, technical staff, administration, or TAC visitors. Assessment data, workshops, or internal reporting requirements may motivate the initiatives.

4. Assessment: the next step in the continuous improvement process

Assessment data are analogous to the feedback needed by the closed loop control circuit of continuous improvement. Alternately assessment data are the sample inspection portion of a statistical quality control system. TAC has long been requiring survey data from alumni and employers but as TAC chair Frank Hart put it, "TAC is moving to accredit based on the effectiveness of a program’s quality assurance system rather than on past performance." Continuous improvement is the quality assurance system required by TAC [11].

By definition, continuous improvement means that you are continually setting higher goals for yourself [12]. To document and present their continuous improvement program, the METS department faculty added a 4^th column for Assessment Measures and a 5^th column for Initiatives and Results to their "goal matching" matrix. This format may require landscape orientation of the paper, small fonts, and/or ledger size paper.

Another 5-column presentation format sometimes advocated for presenting a continuous improvement plan for TAC accreditation has headings:

1. Expanded Institutional Goal
2. Program Intended Outcomes/Objectives
3. Assessment Criteria and Procedures
4. Assessment Results
5. Use of Results.

V. Conducting Assessments

Just as different goals exist at the various levels within an organization, assessment too may be performed at different levels: institution, program, course and at the individual student level [13]. For beginning faculty, the most benefits are obtained at the individual student level. The feedback from this assessment and the associated intervention provide major benefits to the student learning process. The most useful way to conduct an assessment is either in-class or student-embedded. This method gives a direct payoff and intervention and remedial action could be immediate.

In-class assessment can be performed by asking students to write down the important concepts learned on a given topic and the most difficult concept to understand.

An example of a student embedded assessment is to add a few True/False or multiple choice questions from a certification review course to the final exam. These are evaluated but not counted toward the student’s score on the final. In the capstone senior project course at PUC faculty and staff are asked to evaluate the senior project presentation. Results are used to quantify the learning process and the results tabulated to suggest corrective action, if necessary. The format used [14] stated the goals/objectives for that particular course. Three areas were measured: the knowledge of the discipline, oral communication and computer competency. The results were summarized and presented at a department meeting for review.

The METS department has also developed a typical road map for faculty to achieve a quality assessment program for student learning.

VI. Conclusion

The concept of developing missions, goals and objectives is not new to successful business strategists, however it is new to many faculty members and other personnel at academic institutions. To many it appears to be "just another exercise to add to the already heavy work load." But upon examination of the overall continuous improvement process, it can be seen that proper development of missions, goals and objectives is the first step toward effectively conducting assessment and continually improving any organization. Without proper goals that tie into the purpose of the organization, the assessment processes required for accreditation becomes more difficult and less valuable to the organization.

References

1. Brawner, C., Anderson, T., Aorowski, C., Serow, R., Demery, J., "Closing the Loop: Using Qualitative Assessment in the Continuous Quality Improvement of the SUCCEED Coalition," ASEE Annual Conference and Exposition Proceedings (1999), Session 3431.
2. Ross, J., Total Quality Management, second edition, St Lucie Press, Boca Raton, 1995, pp. 91-93.
3. Newell, J., Newell, H., Owens, T., Erjavec, J., Hasan, R., Sternberg, S., "A Process for Developing and Implementing an Assessment Plan in Chemical Engineering Departments," ASEE Annual Conference and Exposition Proceedings (1999), Session 3513.
4. Van Matre, J., Foundations of TQM: a readings book, The Dryden Press, Orlando, FL, 1995, pp. 243-244.
5. Ref. 2
6. Ref. 2
7. Ref. 2
8. Ref. 2
9. Baker, David and Hart, Frank, "TC2K – New ABET Criteria Update," Engineering Technology Leadership Institute, October 21-24, 2000
10. Jishke, M., From a faculty convocation November 20, 2000. Purdue Chronicle, December 5, 2000, page 3.
11. Ref. 9
12. Chang, R., Continuous Process Improvement, Richard Chang Associates, Inc. Publications Division, Irvine, CA, 1994, pp. 75.
13. Wright, Barbara D., "Assessment for Beginners: Getting Started," AAHE Assessment Conference 2000, Workshop 10, Charlotte, North Carolina, June 14, 2000.
14. Palomba, Catherina A., Eder, Douglas J., Banta Trudy W., "Assessment Essentials - Strategies for Success," The 2000 Assessment Institute in Indianapolis, Nov. 5-7 Indianapolis.

SUSAN SCACHITTI is an Assistant Professor of Industrial Engineering Technology. She holds degrees in Industrial Engineering Technology from the University of Dayton and a Master of Business Administration in Management from North Central College. She has ten years of industrial experience. Her accomplishments include playing key roles in ISO9001 certification and establishing a benchmark for a self-directed workforce. She teaches total quality management and consults in the area of continuous improvement. She is treasurer of the IE Division of ASEE.

GREGORY P. NEFF is an Associate Professor of Mechanical Engineering Technology (MET). He has graduate degrees in mechanical engineering, physics, and mathematics. He is a Registered Professional Engineer, a Certified Manufacturing Engineer, a Certified Manufacturing Technologist, and a Certified Senior Industrial Technologist. He has served as a TAC/ABET MET program accreditation visitor since 1996 and is vice-chair (program) and webmaster for the MET Dept. Heads Committee (METDHC) of the American Society of Mechanical Engineers (ASME). He formerly served as METDHC secretary and as Region VI Representative to the executive committee of the METDHC.

LASH MAPA is an Associate Professor in Industrial Engineering Technology at the Purdue University Calumet. He holds a Ph.D. from The University of Manchester, Manchester, England and actively involved in Engineering Cost Analysis, Statistical Process Control and International Standards. He has twelve years of industrial and management experience in petrochemical and automotive component manufacturing. He is a member of ASQC.

MOHAMMAD A. ZAHRAEE, Professor and Head of the Department of Manufacturing Engineering Technologies and Supervision, received his Ph.D. in Theoretical and Applied Mechanics from the University of Illinois. He is a registered P.E. in Indiana. He has served as the National Chair of the MET Dept. Heads Committee of ASME, on the ASME Board on Engineering Education and is a member of the ASME Council on Education. He has been a TAC (Technology Accreditation Committee) evaluator for ABET since 1992. A member of the Committee on Technology Accreditation of ASME, he was elected TAC alternate for ABET in 1997, a member of TAC in 1998. He is a TAC accreditation team chair. He is chair of the ASME committee preparing MET program criteria for TC2K.