The Vasthead Masthead

News Fronts / The Galveston Light / The Vasthead

James Thomson / Houston Retro Radio / Articles / Work Samples

TIME POWER

A TRAINING EVALUATION PLAN

This material was written by Grady McAllister of Houston, Texas.


BACKGROUND INFORMATION

The Time Power curriculum was developed by Dr. Charles R. Hobbs of Salt Lake City, Utah.

Hobbs received his doctorate from Teacher’s college at Columbia University. In 1974, he left his position as Associate Director of the Teacher Development Program and spent the next eighteen months developing his Time Power system.

Hobbs sees his curriculum as the logical outgrowth of his "life career question": "How can a teacher bring about change in the lives of people through group instruction?"

Hobbs’ time management system relies heavily on the use of the Day-Timer, a time management tool sold by Day-Timers, Inc., in Allentown, Pennsylvania. At the time that he developed his program, Hobbs had no direct connection with Day-Timers. Hobbs’ training company later became a Day-Timers subsidiary. Although Hobbs has retired, Day-Timers still offers his Time Power system.

For over forty years, the Day-Timer planner was available only by mail. Recently, Day-Timers began selling the Day-Timer in regular retail outlets. Time Power time management materials are part of the package sold in stores.

PROGRAM GOALS

As the result of attending the Time Power program, participants will:

  • Have increased awareness of the value of more effectively managing time.
  • Have in writing clearly defined productivity goals.
  • Have a system for writing their personal goals with the company and their personal life goals.
  • Have a system for planning and control in carrying out the goals.
  • Carry out the goals.


TARGET AUDIENCE

Salespeople, their managers, and all staff involved in the marketing effort.

PROGRAM IMPLEMENTATION

The program was designed for presentation in a classroom setting. Implementation also includes a five hour personal planning session on the day after the last session. There is also a follow-up consultation six weeks after the training sessions. At that time, participants assemble in smaller groups for additional study.

PROGRAM DELIVERY

Time Power is delivered by the Sales and Training Resource Division of Day-timers, Inc.

PROGRAM PREREQUISITES

The program assumes no prior knowledge of time management techniques.

REASONS FOR THE EVALUATION

The following table describes the evaluation needs for each stakeholder:
 

Program Stakeholder: Needs: Evaluation Link:
Company leadership Increased profits Assure that the training adds more to company revenue than it adds to company expenses.
Sales manager  Increased sales; fewer customer complaints Assure that the training contributes to sales and customer service goals.
Customers Follow-up on commitments made by salespeople Customer satisfaction
Company Training Manager Assure current effectiveness of the training Assure that the training contributes to the value of training in the organization.
Day-Timers, Inc. Day-Timers has developed a newer program which places more focus on work groups and work teams. They would like to compare the effectiveness of Time Power to the new program. Determine the current effectiveness of Time Power. Evaluate participant reaction, learning, and performance. Compare it to the new program using the same criteria. 

PERFORMANCE OBJECTIVES

After their completion of the Time Power program, the participants will…

Conduct a time log to discover time wasters.

Maintain a system for recording and retrieving information.

Maintain a set of written goals for each area of responsibility.

Maintain a set of written productivity goals. Productivity goals address specific time management problems for each individual. Each person identifies these needs during the program and revises the list regularly.

Reduce time and increase effectiveness when leading meetings.

Reduce time and increase effectiveness when participating in meetings.

Devote 15 minutes to planning at the beginning of each work day; rank each task according to its priority.

Maintain a system for following up on commitments and customer contacts.

Apply the nine steps for effective delegation.

Avoid procrastination.

Use the "spontaneous goal" as a way to "get to the point" in conversations on the job.


LEARNING OBJECTIVES

At the end of the Time Power Program, the participants will have an understanding of…

The five conditions for controlling events.

The concepts of urgency and importance.

The common barriers to personal productivity (time management problems).

The seven steps for goal planning.

The relationship between personal values, goals, and the management of time.

A framework for writing personal life goals and goals with the company.

A process for delegating effectively.

Methods for conducting effective meetings.

Methods for reducing meeting time when leading the meeting.

Methods for reducing meeting time when participating in the meeting.

Methods for controlling interruptions.

Methods for reducing time on the telephone while maintaining courtesy and effectiveness.

A system for daily planning.

The use of the Day-Timer for recording and retrieving information.

The concept of the "Spontaneous Goal."

William James’ "Four Rules of Habit."

Six procedures for overcoming preoccupation.

Methods for controlling paperwork and desk clutter.

Methods for avoiding procrastination

The use of a time log to discover productivity problems.


BUSINESS OBJECTIVES

The business results of the Time Power program will include…

An increase in profits due to sales.

A reduction in meeting cost.

A reduction in costs due to customer complaints.


AUDIENCE PROFILE

The participants are high school and college graduates. All participants have average to above average intellectual capacities.

They have no prior exposure to time management training and are not aware of the value of managing their time.

Total work experience, time on the job, and physical abilities are not factors in the selection of participants.

SUCCESS INDICATORS

Indicator: Criteria: Measurement Standards:
When participants like the training. How did the participants feel about the…
  • course materials
  • time management tools
  • quality of presentation
  • asking and answering of questions
  • pace of instruction
  • knowledge of subject
How did the participants feel about the course objectives as they relate to…
  • job requirements
  • their willingness to perform the course objectives
  • applicability on the job
Measurement standard:
 
 

An average approval rating of 80% among at least 80% of participants.
 
 
 
 
 
 
 
 
 
 
 
 

Success for the program requires that at least 80% of the students give a "good" (or higher) rating for the program as a whole.

When participants have learned enough of the material to know how to use it on the job. Learning objectives:
  • Participants are able to write a goal with the company, using the criteria for goal setting.
  • Participants are able to create a prioritized Daily Action List in a Day-Timer, using the proper symbols.
  • Participants are able to record and retrieve information in a Day-Timer.
Test scores of 80% or better for 80% of the students will be necessary to prove a transfer of learning.

.

When participants apply the key elements of the program to their jobs. How many times have the participants applied each of the above learning objectives on the job?

Have any factors on the job impeded the application of the Time Power program?

At least 95% of participants have applied learning objectives no. 1. at least one time. At least 95% of the participants have applied learning objectives nos. 2 and 3 on at least 90% of the work days since completion of training.
When participants contribute more to profits as a result of the program. What expenses were incurred purchasing and delivering the training?

What dollar value has the company received as a result of the training?

Return on investment.

LINKAGE BETWEEN PROGRAM GOALS AND STRATEGIC GOALS

Program Goals: Strategic Goals:
Have increased awareness of the value of more effectively managing time. Salespeople initiate contacts with more customers, resulting in more sales.
Have in writing clearly defined productivity goals. Employees devote time on the job to activities which contribute to company profits.
Have a system for writing goals with the company and personal life goals. Employees achieve goals which support the strategic goals of the company.
Have a system for planning and control in carrying out the goals. Employees avoid revenue losses that occur when goals are not achieved.
Carry out the goals.
Employees achieve goals which increase company profits.

 

LINKAGE BETWEEN TIME POWER SKILLS

AND PERFORMANCE APPLICATIONS

Skill: Performance Application:
The ability to establish individual goals with the company that harmonize with company goals and mission. Employees devote time on the job to actions that contribute to individual and company objectives.
The ability to write a prioritized daily action list during a morning planning period. Employees devote most time on the job to high priority actions.
The ability to identify time management problems and to write productivity goals. Employees include three productivity goals on their daily action list.
The ability to record and retrieve information.  Employees avoid delays caused when information is not recorded or can not be located. Customers are favorably impressed by command of detail.
The ability to schedule follow-up with current and potential customers.  Employees follow-up customer contacts at an appropriate time. Customers react favorably to follow-up on commitments.
The ability to write an agenda for a forthcoming meeting. Employees devote meetings to items of highest priority.
To ability to delegate effectively. Employees assume increasing levels of responsibility. Employees acquire increased knowledge and range of experience.
The ability to avoid procrastination. High priority tasks are performed as soon as possible. Salespeople eliminate "call reluctance" when contacting potential customers.

 

EVALUATION ROLES AND TIMELINE

Stakeholder Roles: Actions:
Sales manager Sales manager will provide data for comparing sales before and after training
Customers The evaluation team will study customer feedback before and after training
Development team The development team will verify validity of knowledge tested during evaluation.
Participants Participants will provide data on their knowledge and use of the system.

 

Evaluation Team Actions: Deadline & Days Allowed for Completion:
Upon conclusion of first pilot class, collect level 1 and 2 evaluation data. March 25. Three days.
Analyze data. March 28. Three days.
Upon conclusion of second pilot class, collect level 1 and 2 evaluation data. April 1. Three days.
Analyze data. April 5. Three days.
Distribute evaluation report to stakeholders. April 12. One day.
Collect level 3 data. May 1. Three days.
Analyze and interpret level 3 data. May 15. Five days.
Collect level 4 data. May 21. Five days.
Analyze and interpret level 4 data. May 27. Five days
Write and distribute final evaluation report. June 1. Three days.

Level One Evaluation

The Level One Evaluation form assesses the reaction of participants in Time Power, a time management program.

The content of the program is very similar to TimeQuest, a program offered by Franklin Quest of Salt Lake City, Utah. A TimeQuest reaction form was one of the samples that we received in class. Because of the similarity between the programs, I used the TimeQuest form as a point of reference in creating a Level One evaluation for Time Power.

The form tests participant reaction to the knowledge and presentation skills of the seminar leader. It goes on to evaluate the overall organization and sequence of the material. It also evaluates the explanation of the two pivotal elements: the relationship between "unifying principles" (values) and goals and the use of the Day-Timer as a time management tool. Because the program is a holistic approach to time management, the form asks the participants about the value of the material for both their business and personal lives.

The form tests participant reaction to the overall pace of the program and whether they thought the pace was appropriate.

The questionnaire asks two open-ended questions to determine what the participants liked the most and the least.

Level Two Evaluation

The Level Two Evaluation form assesses the learning of Time Power participants.

The test presents eleven multiple choice questions which cover the key elements of the program. It is designed for implementation at the end of the seminar.

I had originally planned to include a set of matching questions. I later ruled that out because the items were not sufficiently homogeneous, and many could have been selected through the process of elimination.

The test includes questions about such abstract elements as the definition of time management and the relationship between "urgent" and "important" when setting priorities.

It also tests participants on their understanding of some of the specific behaviors that the program teaches. Examples include the ways for scheduling appointments and for retrieving data.

Level Three Evaluation

The Level Three Evaluation form assesses tests how well Time Power participants have put the program into practice. It covers the behavior goals that were set in the evaluation plan.

I have ruled out performance contracts and observation. Either of these methods would put a participant in the position of being monitored while caring out the program. This would tend to contaminate the results. For example, Time Power recommends that a person spend fifteen minutes "in solitude" to plan each day. Likewise, the involvement of another party would affect other elements of the program such as the "unifying principles," personal life goals, and goals with the company.

The survey would be sent to each person by mail. Persons who have not responded within two weeks would be given the same survey over the telephone.

Level Four Evaluation

The Level Two Evaluation form assesses the effect of the program on business results.

The main focus is on any increase in sales generated by salespeople who received the training. Their increased sales profits are compared to a control group. The comparison is based on available accounting records. If no control group is available, the pre-training records of the salespeople would be used as a baseline.

The evaluation also assesses the value of reduced meeting time. It is quantified by computing the annual salary cost of the people who attend the meetings. Savings can be realized by having fewer meetings, by having shorter meetings, and by eliminating participants who do not need to be present. The cost of meetings led by training group and control group employees would be compared.

The evaluation also looks for any reduction in customer complaints. If the training group compares favorably to the control group, this can be attributed to the follow-up system included in the program. The cost of a customer complaint would require a value judgment by a knowledgeable person within the company.

This program is presented by a training company, rather than an in-house trainer. Therefore, there is no design or development cost on the part of training department. The cost varies according to the number of participants and includes all course materials.

 

DATA COLLECTION STRATEGY

Level One

A reaction evaluation will be conducted immediately before the "Summary for Action" that concludes the program. All participants will be asked to complete a Level One questionnaire. A member of the evaluation team will distribute the form while the instructor is not present.

The form has been created by the evaluation team. It will be used after the course development organization has certified it as acceptable.

The desired response rate is 90 %, but a response closer to 100% is likely under the circumstances.

Success for the program requires that at least 80% of the students give a "good" (or higher) rating for the program as a whole.

Level Two

A test of learning will follow the "Summary for Action" at the end of the last session. The test assesses the participants’ knowledge of the key learning objectives in the evaluation plan. The test instrument will be used after it has been certified as acceptable by the course development organization.

The entire population of the training group will be asked to complete the test.

Each copy of the instrument will have its own number in small type at the bottom of the sheet. The participants will be assured that the data will be reported as a whole and not be connected to any individual.

Test scores of 80% or better for 80% of the students will be necessary to prove a transfer of learning.

Level Three

A behavior questionnaire will ascertain how well the learning from the program has been put into practice. It measures behavior according to the application objectives of the evaluation plan.

The behavior evaluation instrument has been developed by the evaluation team. It will be certified as acceptable by the company which presents the program.

Six months after training, all participants will receive a Level Three questionnaire in the mail. Each person will also receive a stamped, pre-addressed envelope for returning the survey to the evaluation team.

The postage stamps on the envelopes will code the surveys for follow-up. The stamp on each envelope will be different from each of the others.

This is not as much trouble as it may seem. The U.S. Postal Service offers a wide variety of commemorative stamps. Many of these depict a number of different personalities or events on one low-cost sheet. In addition, a stamp could be coded by tearing off one of its four corners. This would allow each type of stamp to be used four times.

The evaluator will record which participant receives each stamp and will log the individual participants as they return their surveys.

Three weeks after the mailing, a member of the evaluation team will contact anyone who has not responded and will conduct the survey over the telephone. All interviewers will be uniformly trained.

A response rate of 90% will be considered adequate for the evaluation.

Level Four

An evaluation of business results will also be conducted six months after training. The business results of the training group will be compared to a control group of equal size. The control group (salespeople who did not receive training) will be selected through simple random sampling.

Using available business records, the performance of the control group and training group will be compared for the same six month period. The evaluation team will obtain commitment from upper management for access to the records.

The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.

A 0.05 level of significant will be necessary to attribute any difference to training.

The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.

The contribution to business results will be determined by subtracting the cost of training from the value of training.

REPORT STRATEGY

The following stakeholders will receive the evaluation report: The CEO and other company leadership, the Sales Manager, the Training Manager, the participants, and training provider.

The customers are the only stakeholders who will not receive the report. Although customer complaints are studied in the Level Four evaluation, the customers are not directly involved in either the training program or the evaluation process.

The report will include a front cover, an executive summary, the background of the program, a description of the evaluation procedures, and the evaluation results. The report will offer conclusions and recommendations.

The main report will be followed by appendices. These will include the tests and data collection instruments, statistical summaries, and raw data.

The CEO, the company leadership, and the participants will receive only the executive summaries with a cover letter.

The Training Manager, the Sales Manager, and the training company will receive the full report. The training company copy will be sent to the Training Resource Manager at Day-timers, Inc.

The main report will present data in frequency histograms. The executive summary will include frequency distribution charts for key evaluation questions.

ANALYSIS STRATEGY

Level One

The Level One evaluation includes nine closed-response questions. Each will be analyzed on a frequency histogram.

The open-end questions will be coded according to their meaning and grouped into categories. Each response will be converted into a frequency histogram and included in the evaluation report.

Level Two

Each participant’s raw test score will be converted into a percentage of correct items.
The following items will be included in the evaluation report:

      • Mean score.
      • Median score.
      • High score.
      • Low score.
      • Range (high score-low score).
      • The standard deviation.
      • IDIF and IDIS item analyses.

No index of learning gain is included because the evaluation does not include a pre-test for learning.

The post-test will be checked for reliability using the KR21 formula. This analysis will be included in the evaluation report.

Level Three

The Level Three analysis will include mean scores, standard deviations, frequency distributions, and frequency histograms for each test item.

Several of the survey items contain more than one question. In some instances, a filter question determines whether or not a follow-up question should be answered. This data will be reported as a Use of Training Skills table, a format which allows for filter questions.

Open-ended questions will be coded and converted into raw data. In the evaluation report, each question will be presented as a table of employee responses with the comment code on the left and the number of responses on the right.

Level Four

The business results will be analyzed by comparing a training group to a control group of equal size for a six month period.

The evaluation will compare increased profits due to sales, reduced meeting time, and reduction in customer complaints. The value of training will be determined by subtracting the contribution of the control group from the contribution of the training group.

A 0.05 level of significance will be necessary to attribute any difference to training.

The cost of the training will be extrapolated from accounting records. It will include the cost of administering the program, the cost of participant time and expenses, the amount paid to the training provider, classroom and equipment costs, and the cost of evaluation.

The contribution to business results will be determined by subtracting the cost of training from the value of training.

The evaluation report will include a table which compares the business results for the training and control groups. A separate table will present the costs of training.

 

Total Training Online (Software Training for Adobe & Microsoft)

The Vasthead is the professional web site of
Grady McAllister of Houston, Texas.

http://vasthead.com

silhouette of arrow sign with water in background

Day-Timer, Inc

 

 

Presented April 20, 1996 The University of Houston College of Technology