Cloudworks is no longer accepting new user registrations, and will be closing down on 24th June 2019. We hope to make a read-only archive of the site available soon after.
Grant Penny's Design narrative of creating a computer-based student evaluation system.
Cloud created by:
29 March 2015
Design narrative of creating a computer-based student evaluation system.
'Programmer' and content creator.
Where and when did this occur: At my workplace. Began last year, and still continuing.
What kind of space: Office based.
Who were the key actors: Myself, two coworkers for Japanese translation, supervisor and manager for oversight.
What were the relations between them: Myself and one coworker work in the same office, on much the same work. Our relation with the third coworker is normally only assistance and recommendation based - as she heads customer service. Within this task, she took a key role in being a third eye in reading the outcomes of the computer system. The supervisor holds the same authority over all of us, who reports directly to the manager, who has final say over a project.
What where the beliefs and desires which shaped their interaction: That a computer-based system could replace human typed evaluations, both in quality and authenticity.
What were you trying to achieve: I was creating a database of possible comments at various achievement levels for our 'level 3' students. These comments had to account for various issues the students might have, and maintain a natural voice. They needed to maintain this voice once translated into Japanese and collaborated.
What was your measure of success? 1) Do the comments sound natural. 2) Did they account for all difficulties faced by students (i.e. difficulty in various aspects of the work, or success, or areas for improvement). 3) Did it save myself and my coworkers time (translation time is the biggest consumer).
1. Identified the quantity of comments needed. Done by noting each report contains five fields, and within each field, a mark of 1-5 is generated. This resulted in a minimum of 5*5 comments. Considering varying causes or results from their performance, two comments for each section were made.
Effect: About a weeks worth of work, I created soundbite comments and had them translated.
Challenge: Agreeing on the amount of work for myself and coworkers.
2. Designing the computer system.
Effect: Began to understand a way to effectively combine the comments, and detailed expected challenges.
Challenge: Ensuring the system is easy to use. As well as ensuring that human error in populating the system is identified.
3. Inserting comments into the system, and linking these with the Japanese comment.
Effect: Could begin testing the combining of comments.
Challenge: Handling gendered comments (he/she etc.). These needed to be coded according to student name. Handling the combining of text based comments, where an array could be selected, from various points.
4. Testing the system.
Effect: I got feedback from coworkers and supervisor.
Challenge: Was tasked to improve the automation of the system (from software>word processor for translation). This was very difficult to do, and would need a different software solution potentially.
5. Trailing the system.
Effect: Could mass test the system across multiple actors, receiving real experience feedback.
Challenge: Discovered there were not enough comments to cover all student scenarios. This will be added in the next round of development. Furthermore, work was required from the my coworker to improve some translations to sound more natural in the context.
Expected and unexpected outcomes of your actions:
Expected: It saved me time in creating student evaluations.
Unexpected: It did not save enough time for the translator, as it still required a further look before sending to students.
To what degree did you meet your objectives? What additional outcomes did you engender: Given the initial objective of creating a system to streamline the evaluation process, this was achieved. However, the final outcome did not have as large a degree of success and more work will be needed, especially on the variety of comments and their translations. Regarding additional outcomes, from a personal note I discovered with much better accuracy how to create valuable and easily translatable comments.
Provide evidence to back your claims: My time spent creating comments was reduced by well over 50%. The quality of comments was regarded as high from coworkers. During post-brief, and towards the next set of project development, a key objective has been identified as adding more comments, covering a greater selection of student outcomes.
What transferable insights did you gain? I gained insights into the complexities of translation, including the cultural considerations of comments (i.e. how negative comments are delivered and perceived in Japanese vs. English). I improved my excel skills.