Source: http://www.amazon.com

Figure 1: 3D Home Architect Version 3 Cover

 

Usability Assessment Test Report

3D Home Architect Deluxe Version 3

 

Matthew Starke

mrstarke@bsu.edu

February 18, 2011

 

Table of Contents

  1. Introduction

  2. Methodology

  3. Test Results

  4. Data Analysis

  5. Conclusions

  6. References

  7. Appendix

 

Introduction

Usability Testing Introduction

The term "usability" can be interpreted many different ways when attempting to come up with a definition in the engineering world. In terms of this assessment test, usability is best defined as, "The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user" (Usability, 2011). In other words, usability testing is a product test approach that assesses the usability of an interface, system or product (Chugh, 1997). The purpose of usability testing is to determine information leading to the production and support of products that are easy to learn, easy to use, satisfying and provide much utility valued by users. According to Rubin (1994) there are four main types of usability testing including exploratory, assessment, validation, and comparison tests.

For this particular test, a usability assessment is being taken for 3D Home Architect Version 3 (See Figure 1). Usability in terms of this program is being able to easily and consistently construct the desired portions in the interior and exterior of a home. In other words, a user needs to be able to locate and employ many functions in 3D Home Architect efficiently to create a desired outcome. This particular product was chosen to inform the product redesign team about usability issues that arose so that product can be improved for future versions.

The program is important in Technology Education classrooms, which is why it was chosen for a usability assessment test. 3D Home Architect is used to teach students architecture and home design without the need to be looking around a physical building. This includes how studs, walls, doors, floors, landscape, and other important parts of the home are created. Besides constructing classrooms, some classrooms use the program to create floor plan layouts which are used to build scale models of houses.

Technology Identification

3D Home Architect Deluxe Version 3 allows the virtual creation of home and landscape designs. This gives the ability to show customers internal and external design ideas for their property and home. Figures 2 and 3 show the user interface and menus while Figures 4 and 5 show a professional and student sample of a floor plan that was developed.

"With 3D Home Architect Deluxe, you can quickly and easily produce accurate and complete floor plans for a remodel, and addition, or even an entire home. The program will handle multiple floors, check your design for obvious errors, determine building materials you will need, and even let you see and work with your design in three dimensions."

- 3D Home Architect Version 3 Software Description (Broderbund, 1999)

Figure 2: 3D Home Architect Deluxe Version 3 Header

Figure 3: Build Menu Example (Click picture below to enlarge)

Figure 4: Professional Three Dimensional Sample (Click picture below to enlarge)

Source: 3D Home Architect Version 3

Figure 5: Student Floor Plan Sample (Click picture below to enlarge)

Competitor Products

Besides 3D Home Architect, there are a couple of other architecture design products. Version 3 was chosen for the assessment test because it is the current one that is available to the teacher to utilize in the classroom. The most current version is 3D Home Architect Version Ten. There are three other home design software products besides the one used in the assessment test. Chief Architect is a more sophisticated version of 3D Home Architect, and is created from the same company (http://www.chiefarchitect.com/). As seen in Appendix F, Chief Architect is geared more for professional use with full construction documents. Home and Landscape Design offers a wide variety of different types of software that allows a user to design different elements of a home. This includes interior, room by room, and exterior software packages to tailor the product to specific user desires (http://updates.3dhaonline.com/updates.html). Autodesk, a company that creates many computer aided design software products for engineers, offers a product called AutoCAD Architecture. This program uses a similar interface to other Autodesk products like Inventor, but is tailored to an architectís needs (http://usa.autodesk.com/adsk/servlet/pc/index?id=13799652&siteID=123112).

Key Elements

There are a few elements of 3D Home Architect that are important in using the software. Key parameters need to be mastered by users of 3D Home Architect. Below are a few elements of the program that should be mastered by anyone using the program. These elements will be included in the usability test of the program.

  • Opening 3D Home Architect: Necessary for a home design to be started.
  • Creating Walls: In order for a house to be created, walls need to be made for the house to be enclosed.
  • Creating Doors: To travel throughout the house, openings need to be made to move throughout the house.
  • Placing Cabinets: Whether to use as countertop space, storage, or fixtures, cabinets are needed in a functional house
  • Constructing a Roof: Needs to be placed in order for protection of the house from outside elements
  • Looking at the House Three Dimensionally: A key part of the program to see what the house looks like from the outside.

There are a few items of 3D Home Architect that are not being addressed. For one, installation and setup are not in the assessment test. These items are not relevant when testing 3D Home Architect in terms of software usability. In other words, this assessment test is looking at using the software once it is installed on a computer. For similar reasons, saving, exporting and printing what is created are not addressed in this test as well. This portion of the software is similar to many other programs, making it not necessary when an assessment is being generated on how to create a structure. Also, installation instruction is provided by the compact disk (See Appendix E).

 

Methodology
Instrument Development

The written tools used to gather data are called instruments. Based on the recommendations by Rubin (1994), it is important to gather qualitative and quantitative data based on the observations seen during the usability test. There are a few instruments that should be used in this test. This usability test needs specific individuals with varying abilities. This is later discussed in the participant selection section. In other words, these participants have been selected by the test administrator for their range of skills, making a pre test questionnaire redundant.

For this usability assessment test, an introductory script is needed before the test begins. This introductory script (Appendix A) gives the background needed for the participant to begin the test. The script includes important information like thanking the participant for their help, giving background information to the test, confidential information, thinking aloud, what the participant will be doing, and asking any questions. A copy of the introductory script will be placed next to the testing computer. This gives a foundation for the user to go by as they complete each task in the test.

An observation recording sheet (Appendix B) is necessary for this test. The recording sheet gives the test administrator a place to record two main types of information. For one, this document allows the test administrator to record when the participant completes a task with a completion check and a time that it took to complete the task.

Besides these two documents, a post test questionnaire (Appendix C) is also important for this usability assessment test. This questionnaire is composed of a few open ended questions to give the test administrator feedback about the test. In other words, these questions give a starting point for the participants to relay thoughts and opinions about the software that is being tested.

Instrument Pilot Testing

To ensure that the test would be valid and reliable, a pilot test was conducted before the actual test was made. This pilot test was done with two professionals, one that is in the technology education field, and one that is an elementary educator. From these two pilot tests, there were a couple of things that were changed. Both participants felt some of the wording in the introductory became confusing to the users. For example, the first version of the introductory script used the word, "usability," too frequently. The introductory script was then changed to define what a usability test is, and then kept the wording to a minimum.

Also, the pilot test showed that the post questions were too specific for the participants to discuss with the test administrator. In other words, the questions did not give room for the participant to answer some questions more specifically than others. For the second version, the questions were modified to allow a more open ended discussion about specific topics.

Participant Selection

There are a few important factors to remember when selecting participants. First of all, participants need to be selected from all different abilities. For 3D Home Architect, this means that participants should have different abilities with computers and computer modeling. From these resources, there are a few factors that will be noted when choosing participants.

  • Basic computer program abilities (Microsoft Word, Excel, and PowerPoint)
  • Participant age (Professional, College, High School)
  • Computer Modeling abilities
  • Where program would be used in participantís life

Participant Introduction

For the usability assessment test on 3D Home Architect Version 3, a total of five participants will be used. A total of five participants will have three main differences for these experiments. As discussed in participant selection, each of the participants will have varying differences in age, computer software ability, computer modeling skill, and where the program would be used in participantís life. These five participants will be recruited based on their abilities and qualities listed below:

  • The first participant (Participant A) will be one that has had computer modeling experience prior to the usability test. This person may have already used 3D Home Architect in the past. The participant will be a skilled Technology Educator in a local educational school district. This person will give feedback on how well a professional in the architecture field is able to use the software.
  • The second participant (Participant B) in the usability study will be a professional adult with limited computer modeling experience. In other words, this person will hold a professional job in a local educational facility. The person will have experience with basic computer programs like Microsoft Word, Excel, and PowerPoint. However, this person will have limited experience with computer modeling software. This participant will give feedback by seeing how someone with abilities in other software programs compares to the one being tested. Outside of the usability assessment test, this person may or may not have used 3D Home Architect.
  • The third participant (Participant C) will be an adult with limited computer software ability. This adult works in the educational field, but has not been exposed heavily in basic computer programs. Outside of the usability assessment test, this person may or may not have used 3D Home Architect.
  • The fourth participant (Participant D) will be a college student who has abilities in basic computer software programs like Microsoft Excel, Word, and PowerPoint. The participant would be enrolled in a mechanical engineering degree at a university, and has a strong possibility of using this or similar software outside of the test.
  • The fifth participant (Participant E) will be a teenage high school student who is a relative of the test administrator. This person has some exposure to basic computer programs. Depending on the future chosen by the participant, the individual may encounter similar programs outside of the test.

Test Environment

The test environment is set up in a middle school Technology Education classroom. The computers in the room have 3D Home Architect installed on them for architecture units. These pictures in Figures 6 and 7 show the classroom environment that the participants will conduct the assessment test in. For the test, the room will be quiet as the individual completes tasks described in the introductory script.

Figures 6 & 7: Test Environment (Click picture below to enlarge)

 

 

Test Results
 

3D Home Architect Usability Test Results

  Opening Program

Creating 4 Walls

Creating a Door

 Placing a Cabinet

Constructing a Roof

3D View
    Time   Time   Time

 

Time

 

Time

 

Time

A 0:06 0:12 0:08 0:11 0:20 0:12
B 0:08 0:20 1:20 0:09 0:52 0:51
C 0:09 1:30 0:20 0:18 0:33 0:24
D 0:08 1:34 1:56 0:18 0:58 0:36
E 0:11 2:12 1:08 0:48   NA 1:58

 

 

Data Analysis

Observations With Diagrams

There were observations from the test that are shown by the data in the test results. For example, everyone except the professional (Participant A) had an initial problem of figuring out what each icon in 3D Home Architect represents.

As seen in Figure 2, 3D Home Architect has a header with icons that can be selected by the user. The central problem that users came across was being able to identify what each icon was for. When the cursor was hovering over or clicked on an icon, nothing visual would appear to tell the user what the icon that was selected was. Looking at the data, creating walls and doors was fairly simple for some participants but not others. Looking at this header, these icons are fairly easy to recognize. However, placing a cabinet, roof, and looking three dimensionally did not come as easily as the participants tried to complete the tasks. Frustration was visually evident on their faces as they attempted to see if an icon would see what they were doing. A trend was discovered when the description of each icon was not easily recognized by the individuals. Figure 8 shows this viewing description issue visually.

Figure 8: Viewing Description Issue (Click the picture below to enlarge)

Nielson (1994) discusses the idea that a product, "Needs to be easy to learn so that the user can rapidly start getting some work done with the system." This viewing description issue shows there is learning interference while completing tasks. Looking at the data, a trend appears directly after a user figured out the descriptions were at the bottom of the window. For example, Participant B figured out that where the descriptions were while creating a door. The next task of placing a cabinet was completed much quicker because the person knew where to look to find out the information for each icon. Similar outcomes happened for Participants C, D and E as well. Most of these users made a comment at this point in their respective tests similar too, "Oh, now I know what all of these buttons do."

Novices also had similar issues viewing walls three dimensionally. Most of the users, excluding Participant D, figured out rather quickly that clicking the "eye" in the header was used to view the constructed house. Participant D had considerable frustration with this until the thought of "viewing" could mean "seeing" was verbally expressed. However, once the "eye" was selected, almost everyone had trouble viewing the house three dimensionally.

Figure 9: How to View a Wall (Click the picture below to enlarge)

As seen in Figure 9, when the viewing icon is selected, the camera icon is automatically selected. For the camera icon, the directions say, "Drag a line from your viewpoint in the view direction." The diagram in Figure 9 shows what you are supposed to do with the cursor and mouse. There was significant difficulty with this stage of the test. Most users had issues with this feature. Participant A figured out that the "house" icon was a more efficient way to view the entire house three dimensionally. Over time, all of the participants figured out how to use the camera or house icons to view the house three dimensionally.

Another difficulty that was seen by users was constructing the roof for a structure. During the tests, all users hesitated while completing this task. As the data shows, Participant E actually never completed this task. After quite some time, they decided to move on to the next task. Frustration and struggle could definitely be seen from all the participants with this task. While thinking aloud, participants made comments like, "I am having trouble figuring this out," or, "I do not understand where to go or what to click on." For the participants with little computer software experience, this task was especially difficult. When discussing with participants after the test, I found that the terminology used in this part of the program was the most difficult to understand. The roof command includes terms like truss, gable, fascia and rafter, which only professionals understand (See Figure 10). For most participants, from this command window they would click, "Ok," while saying, "I hope that this works."

Figure 10: Roof Menu (Click the picture below to enlarge)

 

 

Conclusions

With these improvements, a higher degree of efficiency will be achieved. According to Nielson (1994), efficiency is seen when, "Once the user has learned the system, a high level of productivity is possible." From the data and observations, there are a few conclusive recommendations to make 3D Home Architect Version 3 more efficient.

From the test analysis, there are clearly a few usability issues with 3D Home Architect Version 3. For one, the visibility issue is one that was only overcome by the professional. In other words, most users had trouble discovering that the labels for each icon were located in the bottom of the viewing screen. From these observations, I would suggest that the next version have hover text over each icon when a user's cursor is over an icon. This can already be seen in programs like Microsoft Word, where hover text allows the user to see a description before selecting something.

Figure 11: Microsoft Word Hover Text Example

Besides the hover text seen in Figure 11, one participant suggested moving the description bar to directly under the icons. This change would be an effective solution as well.

As discussed with Figure 10, some menus are hard to understand with using the professional terminology for users that are not in the trade. There are a couple of recommendations that could be implemented with this problem. For one, it is possible to incorporate the definitions to these words inside the menu itself. This could be done with hover text, or a link that task a user to the help menu definition. Also, the software could use simpler terms or small graphics to represent complicated words like gable and fascia. Participant D actually suggested something similar stating, "The menu could have little pictures like in the header to represent the words." Support SelectSoft (2011) discusses a similar idea behind making the menus in the program readable for basic users.

Another visibility issue was a lack of tutorials in 3D Home Architect Version 3. Multiple users in the assessment test (Participants D and E) expressed an interest in the test debrief to have better tutorials than the current ones. These users went into the help menu to find out how to complete tasks after frustration. They expressed disapproval in the help menu that was available to them. This help menu is setup like an instruction manual with step by step instruction.

Figure 12: Help Menu (Click the picture below to enlarge)

As seen in Figure 12, the help menu has small graphics to help the user figure out how to do something. As expressed by the participants, this instructional layout needs to be improved on to better help users. To fix this usability issue, diagrams, pictures, and supplemental videos could be incorporated into the next version of 3D Home Architect.

 

References

Broderbund Software, Inc (1999). 3D Home Architect Deluxe Version 3.0 User Manual. Washington, D.C.

Chief Architect. Professional 3D Home Design Software. Retrieved on February 10, 2011, from http://www.chiefarchitect.com/

Chugh, Jase (1997). Usability Testing. Retrieved February 10, 2011, from http://grouplab.cpsc.ucalgary.ca/saul/681/1997/jas/

Nielsen, J. (1994). Usability laboratories: A 1994 survey. Freemont, CA: Author. Retrieved on February 8, 2011,  from www.useit.com/papers/uselabs.html

Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. NY: Wiley Technical Communication Library.

Support SelectSoft. 3D Home Architect Deluxe Version 3.0. Retrieved on February 14, 2011, from http://support.selectsoft.com/products/0123456789/LL3DHARD3J.htm

Usability Professionals' Association. What is Usability. Retrieved on February 9, 2011, from http://www.upassoc.org/usability_resources/about_usability/definitions_of_usability.html

 

Appendix
  1. Introductory Test Script

  2. Observation Recording Sheet

  3. Post Test Questionnaire

  4. 3D Home Architect Version 3 Quick Start Instruction Manual

  5. 3D Home Architect Version 3 Installation Instructions

  6. Chief Architect Promotional Flyer