TIME POWER EVALUATION PLAN

This paper was written by Grady McAllister . It was completed as a project for a class in Occupational Technology at the University of Houston. The ideas expressed do not represent the views of the University of Houston or the College of Technology.

This material is a case study. Although this report refers to certain published facts about Day-Timers, Inc., the writer is not connected to that organization in any way.

The training proposals offered in this material are entirely hypothetical. The training evaluation proposed is for a fictionalized company.

Time Power is a real training program. The published details of the program have been adapted to fit this case study.

The Time Power curriculum was developed by Dr. Charles R. Hobbs of Salt Lake City, Utah.

Hobbs received his doctorate from Teacher’s college at Columbia University. In 1974, he left his position as Associate Director of the Teacher Development Program and spent the next eighteen months developing his Time Power system.

Hobbs sees his curriculum as the logical outgrowth of his "life career question": "How can a teacher bring about change in the lives of people through group instruction?"

Hobbs’ time management system relies heavily on the use of the Day-Timer, a time management tool sold by Day-Timers, Inc., in Allentown, Pennsylvania. At the time that he developed his program, Hobbs had no direct connection with Day-Timers. Hobbs’ training company later became a Day-Timers subsidiary. Although Hobbs has retired, Day-Timers still offers his Time Power system.

For over forty years, the Day-Timer planner was available only by mail. Recently, Day-Timers began selling the Day-Timer in regular retail outlets. Time Power time management materials are part of the package sold in stores.

The program was designed for presentation in a classroom setting. Implementation also includes a five hour personal planning session on the day after the last session. There is also a follow-up consultation six weeks after the training sessions. At that time, participants assemble in smaller groups for additional study.

Time Power is delivered by the Sales and Training Resource Division of Day-timers, Inc.

Program Stakeholder:	Needs:	Evaluation Link:
Company leadership	Increased profits	Assure that the training adds more to company revenue than it adds to company expenses.
Sales manager	Increased sales; fewer customer complaints	Assure that the training contributes to sales and customer service goals.
Customers	Follow-up on commitments made by salespeople	Customer satisfaction
Company Training Manager	Assure current effectiveness of the training	Assure that the training contributes to the value of training in the organization.
Day-Timers, Inc.	Day-Timers has developed a newer program which places more focus on work groups and work teams. They would like to compare the effectiveness of Time Power to the new program.	Determine the current effectiveness of Time Power. Evaluate participant reaction, learning, and performance. Compare it to the new program using the same criteria.

At the end of the Time Power Program, the participants will have an understanding of…

The participants are high school and college graduates. All participants have average to above average intellectual capacities.

They have no prior exposure to time management training and are not aware of the value of managing their time.

Total work experience, time on the job, and physical abilities are not factors in the selection of participants.

Indicator:	Criteria:	Measurement Standards:
When participants like the training.	How did the participants feel about the… course materials time management tools quality of presentation asking and answering of questions pace of instruction knowledge of subject How did the participants feel about the course objectives as they relate to… job requirements their willingness to perform the course objectives applicability on the job	Measurement standard: An average approval rating of 80% among at least 80% of participants. Success for the program requires that at least 80% of the students give a "good" (or higher) rating for the program as a whole.
When participants have learned enough of the material to know how to use it on the job.	Learning objectives: Participants are able to write a goal with the company, using the criteria for goal setting. Participants are able to create a prioritized Daily Action List in a Day-Timer, using the proper symbols. Participants are able to record and retrieve information in a Day-Timer.	Test scores of 80% or better for 80% of the students will be necessary to prove a transfer of learning. .
When participants apply the key elements of the program to their jobs.	How many times have the participants applied each of the above learning objectives on the job? Have any factors on the job impeded the application of the Time Power program?	At least 95% of participants have applied learning objectives no. 1. at least one time. At least 95% of the participants have applied learning objectives nos. 2 and 3 on at least 90% of the work days since completion of training.
When participants contribute more to profits as a result of the program.	What expenses were incurred purchasing and delivering the training? What dollar value has the company received as a result of the training?	Return on investment.

Program Goals:	Strategic Goals:
Have increased awareness of the value of more effectively managing time.	Salespeople initiate contacts with more customers, resulting in more sales.
Have in writing clearly defined productivity goals.	Employees devote time on the job to activities which contribute to company profits.
Have a system for writing goals with the company and personal life goals.	Employees achieve goals which support the strategic goals of the company.
Have a system for planning and control in carrying out the goals.	Employees avoid revenue losses that occur when goals are not achieved.
Carry out the goals.	Employees achieve goals which increase company profits.

Skill:	Performance Application:
The ability to establish individual goals with the company that harmonize with company goals and mission.	Employees devote time on the job to actions that contribute to individual and company objectives.
The ability to write a prioritized daily action list during a morning planning period.	Employees devote most time on the job to high priority actions.
The ability to identify time management problems and to write productivity goals.	Employees include three productivity goals on their daily action list.
The ability to record and retrieve information.	Employees avoid delays caused when information is not recorded or can not be located. Customers are favorably impressed by command of detail.
The ability to schedule follow-up with current and potential customers.	Employees follow-up customer contacts at an appropriate time. Customers react favorably to follow-up on commitments.
The ability to write an agenda for a forthcoming meeting.	Employees devote meetings to items of highest priority.
To ability to delegate effectively.	Employees assume increasing levels of responsibility. Employees acquire increased knowledge and range of experience.
The ability to avoid procrastination.	High priority tasks are performed as soon as possible. Salespeople eliminate "call reluctance" when contacting potential customers.

Evaluation Team Actions:	Deadline & Days Allowed for Completion:
Upon conclusion of first pilot class, collect level 1 and 2 evaluation data.	March 25. Three days.
Analyze data.	March 28. Three days.
Upon conclusion of second pilot class, collect level 1 and 2 evaluation data.	April 1. Three days.
Analyze data.	April 5. Three days.
Distribute evaluation report to stakeholders.	April 12. One day.
Collect level 3 data.	May 1. Three days.
Analyze and interpret level 3 data.	May 15. Five days.
Collect level 4 data.	May 21. Five days.
Analyze and interpret level 4 data.	May 27. Five days
Write and distribute final evaluation report.	June 1. Three days.

The Level One Evaluation form assesses the reaction of participants in Time Power, a time management program.

The content of the program is very similar to TimeQuest, a program offered by Franklin Quest of Salt Lake City, Utah. A TimeQuest reaction form was one of the samples that we received in class. Because of the similarity between the programs, I used the TimeQuest form as a point of reference in creating a Level One evaluation for Time Power.

The form tests participant reaction to the knowledge and presentation skills of the seminar leader. It goes on to evaluate the overall organization and sequence of the material. It also evaluates the explanation of the two pivotal elements: the relationship between "unifying principles" (values) and goals and the use of the Day-Timer as a time management tool. Because the program is a holistic approach to time management, the form asks the participants about the value of the material for both their business and personal lives.

The form tests participant reaction to the overall pace of the program and whether they thought the pace was appropriate.

The questionnaire asks two open-ended questions to determine what the participants liked the most and the least.

The test presents eleven multiple choice questions which cover the key elements of the program. It is designed for implementation at the end of the seminar.

I had originally planned to include a set of matching questions. I later ruled that out because the items were not sufficiently homogeneous, and many could have been selected through the process of elimination.

The test includes questions about such abstract elements as the definition of time management and the relationship between "urgent" and "important" when setting priorities.

It also tests participants on their understanding of some of the specific behaviors that the program teaches. Examples include the ways for scheduling appointments and for retrieving data.

The Level Three Evaluation form assesses tests how well Time Power participants have put the program into practice. It covers the behavior goals that were set in the evaluation plan.

I have ruled out performance contracts and observation. Either of these methods would put a participant in the position of being monitored while caring out the program. This would tend to contaminate the results. For example, Time Power recommends that a person spend fifteen minutes "in solitude" to plan each day. Likewise, the involvement of another party would affect other elements of the program such as the "unifying principles," personal life goals, and goals with the company.

The survey would be sent to each person by mail. Persons who have not responded within two weeks would be given the same survey over the telephone.

The Level Two Evaluation form assesses the effect of the program on business results.

The main focus is on any increase in sales generated by salespeople who received the training. Their increased sales profits are compared to a control group. The comparison is based on available accounting records. If no control group is available, the pre-training records of the salespeople would be used as a baseline.

The evaluation also assesses the value of reduced meeting time. It is quantified by computing the annual salary cost of the people who attend the meetings. Savings can be realized by having fewer meetings, by having shorter meetings, and by eliminating participants who do not need to be present. The cost of meetings led by training group and control group employees would be compared.

The evaluation also looks for any reduction in customer complaints. If the training group compares favorably to the control group, this can be attributed to the follow-up system included in the program. The cost of a customer complaint would require a value judgment by a knowledgeable person within the company.

This program is presented by a training company, rather than an in-house trainer. Therefore, there is no design or development cost on the part of training department. The cost varies according to the number of participants and includes all course materials.

A reaction evaluation will be conducted immediately before the "Summary for Action" that concludes the program. All participants will be asked to complete a Level One questionnaire. A member of the evaluation team will distribute the form while the instructor is not present.

The form has been created by the evaluation team. It will be used after the course development organization has certified it as acceptable.

The desired response rate is 90 %, but a response closer to 100% is likely under the circumstances.

Success for the program requires that at least 80% of the students give a "good" (or higher) rating for the program as a whole.

A test of learning will follow the "Summary for Action" at the end of the last session. The test assesses the participants’ knowledge of the key learning objectives in the evaluation plan. The test instrument will be used after it has been certified as acceptable by the course development organization.

Each copy of the instrument will have its own number in small type at the bottom of the sheet. The participants will be assured that the data will be reported as a whole and not be connected to any individual.

Test scores of 80% or better for 80% of the students will be necessary to prove a transfer of learning.

A behavior questionnaire will ascertain how well the learning from the program has been put into practice. It measures behavior according to the application objectives of the evaluation plan.

The behavior evaluation instrument has been developed by the evaluation team. It will be certified as acceptable by the company which presents the program.

Six months after training, all participants will receive a Level Three questionnaire in the mail. Each person will also receive a stamped, pre-addressed envelope for returning the survey to the evaluation team.

The postage stamps on the envelopes will code the surveys for follow-up. The stamp on each envelope will be different from each of the others.

This is not as much trouble as it may seem. The U.S. Postal Service offers a wide variety of commemorative stamps. Many of these depict a number of different personalities or events on one low-cost sheet. In addition, a stamp could be coded by tearing off one of its four corners. This would allow each type of stamp to be used four times.

The evaluator will record which participant receives each stamp and will log the individual participants as they return their surveys.

Three weeks after the mailing, a member of the evaluation team will contact anyone who has not responded and will conduct the survey over the telephone. All interviewers will be uniformly trained.

An evaluation of business results will also be conducted six months after training. The business results of the training group will be compared to a control group of equal size. The control group (salespeople who did not receive training) will be selected through simple random sampling.

Using available business records, the performance of the control group and training group will be compared for the same six month period. The evaluation team will obtain commitment from upper management for access to the records.

The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.

A 0.05 level of significant will be necessary to attribute any difference to training.

The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.

The contribution to business results will be determined by subtracting the cost of training from the value of training.

The following stakeholders will receive the evaluation report: The CEO and other company leadership, the Sales Manager, the Training Manager, the participants, and training provider.

The customers are the only stakeholders who will not receive the report. Although customer complaints are studied in the Level Four evaluation, the customers are not directly involved in either the training program or the evaluation process.

The report will include a front cover, an executive summary, the background of the program, a description of the evaluation procedures, and the evaluation results. The report will offer conclusions and recommendations.

The main report will be followed by appendices. These will include the tests and data collection instruments, statistical summaries, and raw data.

The CEO, the company leadership, and the participants will receive only the executive summaries with a cover letter.

The Training Manager, the Sales Manager, and the training company will receive the full report. The training company copy will be sent to the Training Resource Manager at Day-timers, Inc.

The main report will present data in frequency histograms. The executive summary will include frequency distribution charts for key evaluation questions.

The Level One evaluation includes nine closed-response questions. Each will be analyzed on a frequency histogram.

The open-end questions will be coded according to their meaning and grouped into categories. Each response will be converted into a frequency histogram and included in the evaluation report.

Each participant’s raw test score will be converted into a percentage of correct items.
The following items will be included in the evaluation report:

No index of learning gain is included because the evaluation does not include a pre-test for learning.

The post-test will be checked for reliability using the KR21 formula. This analysis will be included in the evaluation report.

The Level Three analysis will include mean scores, standard deviations, frequency distributions, and frequency histograms for each test item.

Several of the survey items contain more than one question. In some instances, a filter question determines whether or not a follow-up question should be answered. This data will be reported as a Use of Training Skills table, a format which allows for filter questions.

Open-ended questions will be coded and converted into raw data. In the evaluation report, each question will be presented as a table of employee responses with the comment code on the left and the number of responses on the right.

The business results will be analyzed by comparing a training group to a control group of equal size for a six month period.

A 0.05 level of significance will be necessary to attribute any difference to training.

The contribution to business results will be determined by subtracting the cost of training from the value of training.

The evaluation report will include a table which compares the business results for the training and control groups. A separate table will present the costs of training.

The Vasthead is the professional web site of
Grady McAllister of Houston, Texas.

TIME POWER

REPORT STRATEGY

ANALYSIS STRATEGY