Program Learning Outcome Six:
Evaluate technology resources to facilitate effective assessment and evaluationProgram Learning Outcome Seven:
Utilize technology to collect and analyze data, interpret results, and communicated findings
Introduction
Assessment
and evaluation is important in any educational or work setting.
Assessment/evaluation of programs, employees, lessons, teachers,
administrators, or even technology provides valuable information on whether or
not success is occurring within either setting. This evaluative
information helps those in charge make decisions on whether or not the program,
employee, lesson, teacher, administrator, or technology is being effective, how
it is being effective, and if change needs to occur to promote more success in
the educational or work setting. This
paper will discuss the meanings of assessment and evaluation as well as the
types of technology that could be used to help designers and instructors
administer assessments and evaluations to learners. The paper will also discuss the MATLT Program
Learning Outcomes (PLO) Six; “technology resources to facilitate effective
assessment and evaluation” and Seven; “utilize technology to collect and
analyze data, interpret results, and communicate findings”. A prior MATLT activity will be redesigned in
order to meet both of these Program Learning Outcomes. An explanation of the principles and theory
used in the redesign will be included in the discussion. Challenges generally arise with any design of
an activity as well as a redesign. There
were challenges with the redesign and those will be discussed as well in this
paper.
Assessment and
Evaluation Using Technology
According to
Newby, Stepich, Lehman, Russell, and Ottenbreit-Leftwich (2011), assessment is
the process of gathering evidence of what learners know and can do. Evaluation is the process of gathering
information from multiple sources in order to judge the worth of a project,
program, or entity (Newby et al., 2011).
There are two types of assessment; formative and summative. According to Bedard-Voorhees, Johnson, and
Dobson (2011), formative assessment takes place during an activity to help
improve skills and knowledge to meet standards or objectives. Summative assessment takes place at the end
of a project or activity and gives the designer or instructor time to reflect
on the successes or problems of the project or activity as well as create avenues
for improvement (Newby et al., 2011).
Formative and summative reflection activities serve to deepen thinking
as represented by Bloom's Taxonomy (Bedard-Voorhees, Johnson, & Dobson,
2011). There are many Web 2.0 technology
tools available to help designers and instructors help support learners
success. Such tools might include tests,
surveys, blogs, journals, virtual worlds, e-portfolios, digital imaging, as
well as videos (Bedard-Voorhees, Johnson, & Dobson, 2011). Both formative and summative assessment will
take place during within the activities discussed in this paper. The surveys as well as the reflection/journal
activity will serve as program measures and evaluations for the activities
using technology to help gather information to help the designers determine
whether or not the activities were effective and the learning outcomes were
met. According to the United States Government
Accountability Office (2005), there are two types of systematic program
assessments; performance measures and program evaluations. Performance measures are the continuous
monitoring and reporting of successes of a project, activity, or policy that
has a specific objective or purpose (United States Government Accountability
Office, 2005). Program evaluations are performed
occasionally to examine how well a program is working (United States Government
Accountability Office, 2005). There are
four types of program evaluations; process evaluation, outcome evaluation,
impact evaluation, and cost-benefit/cost-effectiveness analyses. According to the United States Government
Accountability Office (2005), outcome evaluation assesses the extent to which a
program or activity achieves its outcome-oriented objectives. This type of evaluation will be represented
by using the data gathered from the surveys and reflection/journal activity to
help the designer or instructor decide whether or not the activities achieved
the outcome-oriented objectives.
Original Activities: EDU656 Technology Solutions for Just-in-Time
Training and Learning Online
First Activity: Just-in-Time
Approach
The first activity was taken from a discussion post
explaining the Just-in-Time approach to learning. This post is included in order to give the
reader background information for the survey activity created to collect and
analyze data, interpret results, and communicate findings. The survey activity follows the discussion on
the Just-in-Time approach.
Explain the Just-in-Time (JIT) approach in your own words
In my opinion, the JIT approach is
a gathering of information when that information is needed for a project or
task. This seems to be a "need for speed"-"instructional
rigor" approach (Weintraub &Martineau, 2002). There is a need
for balance there.
This type of approach is happening
everyday in just about everyone's personal or workplaces lives. When I
need to make something, say bake something, I seek out the resource that will
give me what I need in the quickest way possible. That resource for me is
the Internet. I wanted to make some gifts for Christmas this year.
Where did I go to find ideas? Pintrest! I also simply searched the
Internet for images of what I wanted to make and gained lots of ideas just from
the images I viewed. I am teaching my students about the Bald
Eagle. I have used the Web to seek out videos or websites to find
information or pictures for the kids to see. It also seems like this
approach is used in the online courses that we are currently participating
in. We gather information about a topic only when that topic is presented
to us in an assignment. We use that information to help us complete the
assignment.
Identify
a problem that can be addressed by JIT training/learning, either in your
workplace or academic setting
A current problem in my workplace
setting is a need for training on the new technology devices that our various
community organizations are providing grants for teachers to purchase. In
the past year I have acquired a smart board and two iPads. I am thrilled
to have these technology devices in my hands but am not comfortable yet with
the best ways to utilize these devices in my classroom.
Describe
at least three possible JIT training/learning technology solutions that might
address this problem
Weintraub and Martineau (2002)
explain in their article that most learning occurs informally through processes
that are not structured or sponsored by a person's workplace. Learning
generally occurs as employees go about their daily tasks or work-related
activities. They explain that learning is the result of four things;
experience, collaboration, observation, and reading. They also describe four
technologies which support JIT training/learning; databases,
Internet-intranets, portals, and content management systems.
Using databases and the Internet
would be a solution to my current problem. I also like the idea of
informal learning through experience, collaboration, observation, and
reading.
Explain
how you think each possible solution could address this problem
Using the Internet or database to
search for articles or tutorials on smart board or iPad information would be a
quick and easy way for me to learn more about these types of technology
devices. I could type in a particular question into the search bar such
as, "how to use a smart board effectively in the kindergarten
classroom" and I would probably find a wealth of links that could help me
with my problem.
Learning informally on my own time
is probably one of the best ways to handle this problem but finding the time to
do just that is another issue. I would love it if my district would give
us in-service time for informal learning and time to just explore the
technology devices, the Internet for video tutorials, or collaborate with other
teachers who have more knowledge about these devices. This would be a
much more valuable use of time compared to brining in some "expert"
to perform professional development where we watch that person use the
device. Having the device in my hands or allowing my hands to
use the device while someone is talking me through it is more valuable to me
and matches my learning style. Learning about how to do something by
reading about it is not best for me when I am under a time constraint. I
do well with that type of learning when I have loads of time to read and then
use what I have read in a direct learning experience.
I set up a wiki space as a
resource for Smart Board use for a previous course which has a few tutorials
and links saved. I invited a few teachers in my building to participate
but have yet to see much added by them. Time is a factor as teachers have
so much on their plate with the daily grind that taking time during the school
day to add to such a site is almost impossible. I have not gone back to
the site myself.
Finding time to learn informally
while away from work is very difficult. Life at home takes priority and
the next thing you know the day is done. I truly believe that informal
learning through experience, collaboration, observation, and reading that is
programmed into the workplace would be the best possible solution to my current
problem of gaining knowledge of how to best use new technology devices in my
classroom.
Second Activity: Needs
Assessment Survey Summary
The
author created a survey using Survey Monkey to gather data from colleagues
concerning a need for teacher training on new technology devices. The link to the survey is included in the
reference section of this paper. The
following paragraphs communicate the findings of the data collected and
analyzed along with an interpretation of the results of the survey.
My needs assessment survey was
based on the week one discussion post assignment discussing the Just-in-Time
Training/Learning approach. I identified
a need in my school for teacher training on the new technology devices (iPads
and Smart Boards) that our various community organizations are providing grants
for teachers to purchase. I based my
survey questions on the JIT solutions I thought would best work for the problem
I identified. Those solutions included
using the Internet or a database to search for articles or tutorials on Smart
Board or iPad information, setting up a wiki space as a resource for these
devices, and learning informally on my own or on school time with hands on
experience, teacher observation, teacher collaboration, or through reading.
I included ten questions on my survey ranging from
questions about devices that the survey participants use daily in their work
place to questions about how much time the participants spend outside of work
learning about the devices identified in the survey. I sent out survey requests to twenty-six
people. Of the twenty-six people,
eighteen of those were sent to students taking EDU656 and eight were sent to my
colleagues at the school in which I work.
Of the eight I sent to my work colleagues, five of those were teachers
at the primary level. The other three
were my principal, the school psychologist, and the computer lab
instructor. Of the twenty-six requests I
sent out, fourteen actually participated in my survey. I am unaware from the survey results, how
many from my work place participated and how many from EDU656.
The majority of results of my survey did not surprise
me. I invited people from many different
professions so I expected the results to vary a little but truly there was not
much of a difference in the results due to people from different professions
participating. Most professions utilize
the various technology devices I asked about in the survey. People from various professions do some form
of learning outside the work place and utilize many of the devices I asked
about away from work. Eleven of the fourteen participants use a desktop
computer at their work place. Just over
half use an iPad and just over a quarter use a Smart Board. I was not surprised of how the participants
answered the question I had about whether or not they had heard of JIT
Training/Learning. Half had not heard of
JIT Training/Learning and my guess is that most of those who answered that way
were those from my school. I would
suspect that my course mates would have answered in favor of knowing about JIT
being that we have been learning about it.
Not many have used a wiki site before.
I had not even heard of one until I started taking online courses for my
master’s. The results of that question
did not surprise me. Using a wiki site
as a resource would not be the best solution due to the fact that many
participants have never participated in a wiki site before.
One
hundred percent of those who participated in the question about how they learn
best, formally or informally, answered informally. Weintraub and Martineau (2002) pointed out in
their article that most learning occurs informally through processes that are
not structured. I was happy to see that
one hundred percent of those who participated in the survey were in favor of
their work place providing some work release time during their work day to
explore and further learn informally about the technology devices they use
daily. Just over eighty five percent of
the participants were in favor of sharing what they know with their peers in a
small group setting and would allow their peers to observe them using the
various technology devices.
Based on the results of my survey, I feel that informal
learning through on site, hands on, peer collaboration and observation would be
the best JIT solution for training teachers on the technology devices used
daily in the school setting. One hundred
percent of the participants felt that they would learn best if offered time
during the work day to explore and work collaboratively in small groups on the
technology devices they are using daily.
One hundred percent of the participants said they learn best informally. Using the Internet as a reference for
tutorials or articles on how to use the technology devices would also be a good
JIT solution. One hundred percent of the
participants said that they use the Internet as a source when there is a need
to gather information. I believe that
using the work day to give teachers time to learn informally coupled with using
the Internet for gathering information during this time would be the best
solution for finding more uses in the classroom for the technology devices
provided to these teachers.
Redesigned Activity
The activities meet Program Learning Outcome Seven;
“Utilize technology to collect, analyze data, interpret results, and
communicate findings”. The activities do
not fully cover Program Learning Outcome Six; “Technology resources to
facilitate effective assessment and evaluation”. In order to meet Program Learning Outcome
Six, the activity was redesigned to include an evaluation to be administered to
the teachers after they had completed independent and collaborative work using
the Internet and working with colleagues to find more uses in the classroom for
technology devices. Survey Monkey would
be used once again to gain an understanding of whether or not learning had
occurred while the teachers worked independently and with their colleagues to
find more uses in the classroom for the technology devices they were
provided. The survey would ask questions
about how much time the teachers actually spent during the work day
collaborating and working independently on given tasks. Questions would also be asked if the teachers
felt this type of independent learning was more beneficial than providing them
with an expert in this type of professional development. The survey would ask the teachers which types
of Internet sites they found most helpful or useful to them as they searched
for additional uses in the classroom for the technology devices they had
acquired.
Another type of formative assessment would be added to
the redesign of this activity. This formative
assessment would be a reflection/journal activity which would take place during
the independent/collaborative professional development days. The teachers would be asked to keep an online
journal using such technology as a wiki site, website or blog to reflect on
their learning for each day of their independent/collaborative research. They would be given the option to submit
their reflection/journal individually or collaboratively. They would be expected to give their
professional development leader or administrator a link to their site in order
for the leader/administrator to use the information gathered to help decide if
this type of professional development met with success as well as creatively
work out any issues that may have come up during the activity.
Instructional Design Principles
and Theories
During the redesign process, several design principles
and theories were considered. The theory
the author feels the redesign process was patterned after was the Component
Display Theory and would fall under the Cognitivism/Pragmatism
heading. Dabbagh (2006) explains
this theory suggests for a given objective or learner, there is a distinctive mixture
of presentation forms that results in the most valuable learning experience. The different assessment and evaluation
options presented in the activities of this redesign provide unique
presentation forms which should result in effective learning experiences for
the participants as well as the designers and instructors. There were two basic principles the redesign
was based on. The Objectivism/Behaviorism principle as stated by Dabbagh
(2006) lists the following; “emphasis is on observable and measureable
behavior”. The observable and measurable
behavior was found in the results of the assessment and evaluative information
collected and gathered through the redesigned activities. The Constructivism/Interpretivism principle also
stated by Dabbagh (2006) listed the following; “create novel and situation-specific
understandings by “assembling” knowledge by diverse sources appropriate by the
problem at hand (flexible use of knowledge).
This principle was evident in the survey and reflection/journal
activities described in the redesigned activities. Offering diverse sources appropriate to the
task or problem at hand helps the learners and designers/instructors gain
understanding of the successes and issues that result from the
assessments/evaluations created in the redesign of the prior MATLT activities.
Challenges
The challenges with the redesign of this activity would
include the time it would take to generate the additional survey, issues that
may arise with teachers who may not know how to create a website, wiki, or blog
to house their reflection/journal, as well as accountability of those teachers
to actually use their time wisely during the independent/collaborative
professional development days. Time is
always an issue when creating something new.
The designer would need to take the time to fully think through the
activity/activities to make sure to include the appropriate questions that will
lead to accurate data to reflect on during and at the end of the
activity/activities. Allowing the
teachers to work collaboratively may alleviate some of the issues when it came
to submitting the reflection/journal activity.
The designer or instructor may want to include some tutorials on how to
create a website, wiki, or blog so those who chose to work independently would
be able to use the tutorials to help them when creating their
reflection/journal activity. Monitoring
the teachers as they work during these professional development days would help
to ensure accountability in completing the assigned tasks as well as making
good use of their time. The assessment
and evaluation surveys and reflection/journal activities will provide the
designer/instructor valuable information on whether or not the teachers will
have met the outcome-oriented objective as well as the effectiveness of the
activities.
Conclusion
Assessment and evaluation provide valuable information in
the educational and work setting as to whether or not programs, activities, or
employees are being successful. This
information also helps those in charge of these programs, activities, or
employees the ability to reflect and create new ways for success if the
information shows issues or problems.
Technology offers designers and instructors creative ways to assess and
evaluate these programs, activities or employees as well as places to store the
information for use during summative conferences where the effectiveness of all
would be shared. The MATLT Program
Learning Outcomes of; technology resources to facilitate effective assessment
and evaluation as well as the various ways to utilize technology to collect and
analyze data, interpret results, and communicate findings were met in the
redesign of a prior MATLT activity and discussed in this paper. Effective use of technology tools for
assessment and evaluation offer learners/employees as well as designers/instructors
valuable feedback which increases achievement as well provides information on
the success the learners/employees experience while participating in various
programs and activities in educational and work settings.
References
Bedard-Voorhees, A., Johnson, L.M.,
& Dobson, P. (2011). Letting them show what they know: Digital
assessment strategies [Book Submission Chapter Final Version]. In S. Hirtz
and K. Kelly (Eds.) Education for a Digital World 2.0, Section F: eAssessment:
Measuring in Ways that Matter. British Columbia: Province of British Columbia.
Retrieved from your online course.
Dabbagh, N. (2006). Instructional design knowledge base. Retrieved from
http://classweb.gmu.edu/ndabbagh/Resources/IDKB/models_theories.htm Dumitru, P. (2005, October 07). Learning objects make learning
easier. Retrieved fromhttp://myeurope.eun.org/ww/en/pub/myeurope/home/news/headlines/los.htm
Newby, T. J., Stepich, D. A., Lehman,
J. D., Russell, J. D., & Ottenbreit-Leftwich, A.
(2011).Educational technology for
teaching and learning (4th
ed.). Boston, MA: Pearson
Education, Inc.
Education, Inc.
Survey Monkey Link: http://www.surveymonkey.com/s/V6FWTNL
United
States Government Accountability Office. (2005). Performance measurement
and evaluation: Definitions and relationships. Washington, DC: Government
Printing Office. Retrieved from http://www.gao.gov/new.items/d05739sp.pdf
Weintraub, R. S. &
Martineau, J. W. (2002). The
Just-in-Time imperative. T+D 56(6), 50-58. Retrieved from EBSCOhost.
No comments:
Post a Comment