No matter your level of priorities, emails and calls, you're only MINUTES from finding a balance. A balance between busy, yet relaxed. A balance between work and home. David Allen's 5-phase automated system reveals your simpler, most successful life.

TIME POWER

A TRAINING EVALUATION PLAN

Presented April 20, 1996
The University of Houston
College of Technology

This paper was written by Grady McAllister. It was completed as a project for a class in Occupational Technology at the University of Houston. The ideas expressed do not represent the views of the University of Houston or the College of Technology.

This material is a case study. Although this report refers to certain published facts about Day-Timers, Inc., the writer is not connected to that organization in any way.

The training proposals offered in this material are entirely hypothetical.  The training evaluation proposed is for a fictionalized company.

Time Power is a real training program.  The published details of the program have been adapted to fit this case study.

BACKGROUND INFORMATION

The Time Power curriculum was developed by Dr. Charles R. Hobbs of Salt Lake City, Utah.

Hobbs received his doctorate from Teacher’s college at Columbia University. In 1974, he left his position as Associate Director of the Teacher Development Program and spent the next eighteen months developing his Time Power system.

Hobbs sees his curriculum as the logical outgrowth of his "life career question": "How can a teacher bring about change in the lives of people through group instruction?"

Hobbs’ time management system relies heavily on the use of the Day-Timer, a time management tool sold by Day-Timers, Inc., in Allentown, Pennsylvania. At the time that he developed his program, Hobbs had no direct connection with Day-Timers. Hobbs’ training company later became a Day-Timers subsidiary. Although Hobbs has retired, Day-Timers still offers his Time Power system.

For over forty years, the Day-Timer planner was available only by mail. Recently, Day-Timers began selling the Day-Timer in regular retail outlets. Time Power time management materials are part of the package sold in stores.

PROGRAM GOALS

As the result of attending the Time Power program, participants will:

Have increased awareness of the value of more effectively managing time.

Have in writing clearly defined productivity goals.

Have a system for writing their personal goals with the company and their personal life goals.

Have a system for planning and control in carrying out the goals.

Carry out the goals.

TARGET AUDIENCE

Salespeople, their managers, and all staff involved in the marketing effort.

PROGRAM IMPLEMENTATION

The program was designed for presentation in a classroom setting. Implementation also includes a five hour personal planning session on the day after the last session. There is also a follow-up consultation six weeks after the training sessions. At that time, participants assemble in smaller groups for additional study.

PROGRAM DELIVERY

Time Power is delivered by the Sales and Training Resource Division of Day-timers, Inc.

PROGRAM PREREQUISITES

The program assumes no prior knowledge of time management techniques.

REASONS FOR THE EVALUATION

The following table describes the evaluation needs for each stakeholder:
 

PERFORMANCE OBJECTIVES

After their completion of the Time Power program, the participants will…

Conduct a time log to discover time wasters.

Maintain a system for recording and retrieving information.

Maintain a set of written goals for each area of responsibility.

Maintain a set of written productivity goals. Productivity goals address specific time management problems for each individual. Each person identifies these needs during the program and revises the list regularly.

Reduce time and increase effectiveness when leading meetings.

Reduce time and increase effectiveness when participating in meetings.

Devote 15 minutes to planning at the beginning of each work day; rank each task according to its priority.

Maintain a system for following up on commitments and customer contacts.

Apply the nine steps for effective delegation.

Avoid procrastination.

Use the "spontaneous goal" as a way to "get to the point" in conversations on the job.

LEARNING OBJECTIVES

At the end of the Time Power Program, the participants will have an understanding of…

The five conditions for controlling events.

The concepts of urgency and importance.

The common barriers to personal productivity (time management problems).

The seven steps for goal planning.

The relationship between personal values, goals, and the management of time.

A framework for writing personal life goals and goals with the company.

A process for delegating effectively.

Methods for conducting effective meetings.

Methods for reducing meeting time when leading the meeting.

Methods for reducing meeting time when participating in the meeting.

Methods for controlling interruptions.

Methods for reducing time on the telephone while maintaining courtesy and effectiveness.

A system for daily planning.

The use of the Day-Timer for recording and retrieving information.

The concept of the "Spontaneous Goal."

William James’ "Four Rules of Habit."

Six procedures for overcoming preoccupation.

Methods for controlling paperwork and desk clutter.

Methods for avoiding procrastination

The use of a time log to discover productivity problems.

BUSINESS OBJECTIVES

The business results of the Time Power program will include…

An increase in profits due to sales.

A reduction in meeting cost.

A reduction in costs due to customer complaints.

AUDIENCE PROFILE

The participants are high school and college graduates. All participants have average to above average intellectual capacities.

They have no prior exposure to time management training and are not aware of the value of managing their time.

Total work experience, time on the job, and physical abilities are not factors in the selection of participants.

SUCCESS INDICATORS

LINKAGE BETWEEN PROGRAM GOALS

AND STRATEGIC GOALS

 
 
LINKAGE BETWEEN TIME POWER SKILLS

AND PERFORMANCE APPLICATIONS

The Level One Evaluation form assesses the reaction of participants in Time Power, a time management program.

The content of the program is very similar to TimeQuest, a program offered by Franklin Quest of Salt Lake City, Utah. A TimeQuest reaction form was one of the samples that we received in class. Because of the similarity between the programs, I used the TimeQuest form as a point of reference in creating a Level One evaluation for Time Power.

The form tests participant reaction to the knowledge and presentation skills of the seminar leader. It goes on to evaluate the overall organization and sequence of the material. It also evaluates the explanation of the two pivotal elements: the relationship between "unifying principles" (values) and goals and the use of the Day-Timer as a time management tool. Because the program is a holistic approach to time management, the form asks the participants about the value of the material for both their business and personal lives.

The form tests participant reaction to the overall pace of the program and whether they thought the pace was appropriate.

The questionnaire asks two open-ended questions to determine what the participants liked the most and the least.

The Level Two Evaluation form assesses the learning of Time Power participants.

The test presents eleven multiple choice questions which cover the key elements of the program. It is designed for implementation at the end of the seminar.

I had originally planned to include a set of matching questions. I later ruled that out because the items were not sufficiently homogeneous, and many could have been selected through the process of elimination.

The test includes questions about such abstract elements as the definition of time management and the relationship between "urgent" and "important" when setting priorities.

It also tests participants on their understanding of some of the specific behaviors that the program teaches. Examples include the ways for scheduling appointments and for retrieving data.

The Level Three Evaluation form assesses tests how well Time Power participants have put the program into practice. It covers the behavior goals that were set in the evaluation plan.

I have ruled out performance contracts and observation. Either of these methods would put a participant in the position of being monitored while caring out the program. This would tend to contaminate the results. For example, Time Power recommends that a person spend fifteen minutes "in solitude" to plan each day. Likewise, the involvement of another party would affect other elements of the program such as the "unifying principles," personal life goals, and goals with the company.

The survey would be sent to each person by mail. Persons who have not responded within two weeks would be given the same survey over the telephone.

The Level Two Evaluation form assesses the effect of the program on business results.

The main focus is on any increase in sales generated by salespeople who received the training. Their increased sales profits are compared to a control group. The comparison is based on available accounting records. If no control group is available, the pre-training records of the salespeople would be used as a baseline.

The evaluation also assesses the value of reduced meeting time. It is quantified by computing the annual salary cost of the people who attend the meetings. Savings can be realized by having fewer meetings, by having shorter meetings, and by eliminating participants who do not need to be present. The cost of meetings led by training group and control group employees would be compared.

The evaluation also looks for any reduction in customer complaints. If the training group compares favorably to the control group, this can be attributed to the follow-up system included in the program. The cost of a customer complaint would require a value judgment by a knowledgeable person within the company.

This program is presented by a training company, rather than an in-house trainer. Therefore, there is no design or development cost on the part of training department. The cost varies according to the number of participants and includes all course materials.
 

Level One

A reaction evaluation will be conducted immediately before the "Summary for Action" that concludes the program. All participants will be asked to complete a Level One questionnaire. A member of the evaluation team will distribute the form while the instructor is not present.

The form has been created by the evaluation team. It will be used after the course development organization has certified it as acceptable.

The desired response rate is 90 %, but a response closer to 100% is likely under the circumstances.

Success for the program requires that at least 80% of the students give a "good" (or higher) rating for the program as a whole.

Level Two

A test of learning will follow the "Summary for Action" at the end of the last session. The test assesses the participants’ knowledge of the key learning objectives in the evaluation plan. The test instrument will be used after it has been certified as acceptable by the course development organization.

The entire population of the training group will be asked to complete the test.

Each copy of the instrument will have its own number in small type at the bottom of the sheet. The participants will be assured that the data will be reported as a whole and not be connected to any individual.

Test scores of 80% or better for 80% of the students will be necessary to prove a transfer of learning.

Level Three

A behavior questionnaire will ascertain how well the learning from the program has been put into practice. It measures behavior according to the application objectives of the evaluation plan.

The behavior evaluation instrument has been developed by the evaluation team. It will be certified as acceptable by the company which presents the program.

Six months after training, all participants will receive a Level Three questionnaire in the mail. Each person will also receive a stamped, pre-addressed envelope for returning the survey to the evaluation team.

The postage stamps on the envelopes will code the surveys for follow-up. The stamp on each envelope will be different from each of the others.

This is not as much trouble as it may seem. The U.S. Postal Service offers a wide variety of commemorative stamps. Many of these depict a number of different personalities or events on one low-cost sheet. In addition, a stamp could be coded by tearing off one of its four corners. This would allow each type of stamp to be used four times.

The evaluator will record which participant receives each stamp and will log the individual participants as they return their surveys.

Three weeks after the mailing, a member of the evaluation team will contact anyone who has not responded and will conduct the survey over the telephone. All interviewers will be uniformly trained.

A response rate of 90% will be considered adequate for the evaluation.

Level Four

An evaluation of business results will also be conducted six months after training. The business results of the training group will be compared to a control group of equal size. The control group (salespeople who did not receive training) will be selected through simple random sampling.

Using available business records, the performance of the control group and training group will be compared for the same six month period. The evaluation team will obtain commitment from upper management for access to the records.

The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.

A 0.05 level of significant will be necessary to attribute any difference to training.

The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.

The contribution to business results will be determined by subtracting the cost of training from the value of training.
 

The following stakeholders will receive the evaluation report: The CEO and other company leadership, the Sales Manager, the Training Manager, the participants, and training provider.

The customers are the only stakeholders who will not receive the report. Although customer complaints are studied in the Level Four evaluation, the customers are not directly involved in either the training program or the evaluation process.

The report will include a front cover, an executive summary, the background of the program, a description of the evaluation procedures, and the evaluation results. The report will offer conclusions and recommendations.

The main report will be followed by appendices. These will include the tests and data collection instruments, statistical summaries, and raw data.

The CEO, the company leadership, and the participants will receive only the executive summaries with a cover letter.

The Training Manager, the Sales Manager, and the training company will receive the full report. The training company copy will be sent to the Training Resource Manager at Day-timers, Inc.

The main report will present data in frequency histograms. The executive summary will include frequency distribution charts for key evaluation questions.
 

Level One

The Level One evaluation includes nine closed-response questions. Each will be analyzed on a frequency histogram.

The open-end questions will be coded according to their meaning and grouped into categories. Each response will be converted into a frequency histogram and included in the evaluation report.

Level Two

Each participant’s raw test score will be converted into a percentage of correct items.
The following items will be included in the evaluation report:

• Mean score.
• Median score.
• High score.
• Low score.
• Range (high score-low score).
• The standard deviation.
• IDIF and IDIS item analyses.

No index of learning gain is included because the evaluation does not include a pre-test for learning.

The post-test will be checked for reliability using the KR21 formula. This analysis will be included in the evaluation report.

Level Three

The Level Three analysis will include mean scores, standard deviations, frequency distributions, and frequency histograms for each test item.

Several of the survey items contain more than one question. In some instances, a filter question determines whether or not a follow-up question should be answered. This data will be reported as a Use of Training Skills table, a format which allows for filter questions.

Open-ended questions will be coded and converted into raw data. In the evaluation report, each question will be presented as a table of employee responses with the comment code on the left and the number of responses on the right.

Level Four

The business results will be analyzed by comparing a training group to a control group of equal size for a six month period.

The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.

A 0.05 level of significance will be necessary to attribute any difference to training.

The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.

The contribution to business results will be determined by subtracting the cost of training from the value of training.

The evaluation report will include a table which compares the business results for the training and control groups. A separate table will present the costs of training.

The Vasthead is the professional web site of
Grady McAllister of Houston, Texas.

http://vasthead.com