Saturday, January 6, 2007
Windows Vista Lab 2007: Day One
Posted by Jason Dunn in "THOUGHT" @ 11:30 PM
Shortly after arriving, I was given a Windows Vista "Start Box" that had a whole bunch of goodies in it: a Windows Vista "storybook", a Vista mousepad (do people still use those?), a Vista Media DVD, a Belkin (Boo! Hiss!) Easy Transfer Cable, a Windows Live USB flash drive (125 MB in capacity it seems, which is a strange size), a Sandisk 2 GB USB flash drive that works as a ReadyBoost drive, a T-Mobile Day Pass (one day of free T-Mobile Hotspot use), Windows Vista Marketplace card, Windows Vista Magazine, and a copy of the Lego II Star Wars game. Cool it was a fun way to start things.
Figure 1: The swag box we received after entering.
Figure 2: There were dozens of Alienware systems set up in the room - about 25 in total. Each was running Windows Vista, with a performance score of 5.3 (if memory serves), meaning they were powerful enough to guarantee a smooth ride with Vista.
Figure 3: A whole lot of geeks in one room!
Session 1: Windows Vista Logo Program (Dave Block)
The Vista Lab 2007 kicked off with a presentation about hardware under Vista and the logo certification program. I was initially thinking this was going to be a dry marketing presentation about how great the logo program was, but it ended up being quite interesting hearing about the program, and how hardware is supposed to work under Vista. Microsoft seems to keenly recognize that the experience of connecting new hardware, and installing new software, has to be easier for end users.
Unlike the Windows XP program, the Vista logo program is about more than just compatibility: it's about the overall user experience. Simple implementations, increased satisfaction, and make adding hardware and software less frustrating are the goals. He gave the example of some retailers in Europe seeing return rates in the 30% to 40% range with networking hardware this is bad news for the retailers, who are already operating on thin profit margins. They talked about the concept of "no specs necessary", meaning consumers wouldn't need to know about B or G or A. That sounds good in theory, but I think it will take several years for users to buy the newer hardware and dump the older stuff.
Figure 4: The Vista logo certification guys.
There are 17 product categories including new categories such as picture frames. He talked about products that need to pass the Windows Vista colour process test say, a printer and a monitor that are both certified, both giving you accurate colour representation. Technically, you can achieve the same thing today if you have a monitor with the proper colour profile loaded, an application using the right colour profile, and a printer with the right profile, using the right profile for the paper. This is a complicated mess of course and certainly anything that Microsoft can do to streamline this process is appreciated. Will consumers pay attention to it? Doubtful it's not like Joe User goes into a store and looks for an MP3 player that has the PlaysForSure logo on it. My hope is that all of the companies creating the hardware will feel the competitive marketing need to be in lock step with Microsoft on this issue, so everything will be better by default rather than requiring a
The first demo was of a D-Link pre-N wireless router, and it failed. It was a freshly reset unit, but it wouldn't work, so he switched over to talking about digital cameras with Windows Vista. The Nikon D40, a new camera from Nikon that will retail for $699 USD, has embedded MTP. That's very interesting because it means you could connect the camera to any Windows XP or Vista machine and be able to view, and copy to/from the camera without needing any drivers. It also means that the camera will show up to the Vista Gallery program, allowing the user to import photos directly. He said that the major camera manufacturers are on board to release firmware updates for their cameras to enable MTP. There are 15 cameras that have been submitted for certification so far, and many more are expected by the end of the year.
They tried the D-Link demo again, and it worked much better after a reboot of the PC (that doesn't say much for Vista, does it?). Basically, the router showed up in the Vista networking area, and he was able to initiate a connection between the laptop and the router, making it a secure connection, all from within Vista. He was also able to copy the router security settings to a USB flash drive, which was then passed around the room and we were all able to connect to the router even the Windows XP users. This is all done without him having to log into the router directly, and without installing any third-party client software on the PC, which is am amazing improvement over what we have today.
As a side note, I have a theory about networking hardware and Vista. I believe that when Microsoft started talking to networking vendors about making their hardware work with Vista in a seamless manner two or three years ago, the answer they got back was something along the lines of "Why should we work with you you're making your own networking hardware and competing with us." I believe Microsoft looked at their networking business which probably wasn't a big money maker for them anyway and decided it was worth killing off their own product line in order to bring all the networking vendors under the Vista tent and improve the end user experience for all users. The results seem to have paid off in a big way.
The presenter then held up a Canon HV10 camera (an HD product) and talked about it a bit the Vista compliant logo means that it must pass the streaming test and be able to stream HD video directly into Windows Movie Maker. Video cameras don't need to be MTP-compliant in order to be Vista certified, but some of them are because they incorporate digital cameras.
A great question was asked: does the Windows Vista compliant logo ensure that it has 64-bit drivers? The answer is sort of: software with the logo must work on 64-bit versions of Vista, but certified hardware does not need to have 64-bit drivers in order to be logo compliant. This cause a bit of an uproar in the room, and rightly so: if something has a Windows Vista logo, the end user doesn't know or care about 32-bit and 64-bit versions of Windows. He shouldn't have to, he just wants it to work.
I asked about Windows Vista certification for Windows Mobile devices, and was told that they have no solution in place. They're looking at it for the future, but they're unsure about exactly what would make a device compliant.
How are products tested? When it was first started five years ago, they had about 1000 products, and they created a system where each team submitted requirements for their own area. So the team that handles printing in Windows would come up with requirements that printers would have to meet in order to qualify for the logo. All hardware is machine tested (as in, tested by a computer) I'm not entirely sure how that works, but it sounds impressive. He talked about how they analyze device driver crashes, and how if they see a consistent problem, they notify the vendor. The vendor then has 60 days to update their driver to solve the problem if they don't make a driver update, they lose their certification for that product. Microsoft has no authority to force a recall of the product, however, so this is a paper tiger. Most companies wouldnt want to lose their certification (they'd have to print new boxes) so I imagine they'd try to comply.
This part was great to hear: when a customer plugs in a device, and Windows Vista doesn't have a driver, it reports that back to Microsoft. Microsoft then collects this data, and they're able to go to the vendors and encourage them to write drivers for their customers that obviously need them. I think this is a great system, but the results are what matter we'll see how many Vista drivers there are six months after launch. There are over 1.5 million device drivers out of the box with Windows Vista, which is great starting coverage, though the presenter admitted that many of those are essentially duplicated drivers for things like USB flash drives. Driver issues are one of the most painful parts of moving to a new operating system, so I sincerely hope that Microsoft and their partners get this right come launch time.
Session 2: A Day in the Life of a Bug (Paul Donnelly)
Paul started off by talking about how they started the Vista beta program. By Beta 2 they had 10,000 people who were testing Windows Vista. When they opened it up further with the preview program, they set the Customer Experience Program switch to on by default, which reported back which version of Windows Vista was installed. By coordinating this information with the release of product keys, they were able to figure out which groups of people were installing and using the Vista beta product that enabled them to allocate their resources accordingly in terms of ramping up efforts to get certain groups of people to beta test the product.
One of the first things they did with after Beta 1 was realize how out of luck people were with device drivers: they had software that submitted information about which device drivers were available, which got them the data they needed to push harder for drivers. As I talked about above, we'll see how things are after launch I really hope that hardware partners (HP, Canon, Epson, etc.) are on the ball and have drivers waiting to go. I'll be asking about Vista drivers for my R1800 when I see an Epson rep at CES, that's for sure!
Paul talked about how hard it was to track down some bugs. He gave the example of how, during testing, they were getting reports from several people who weren't able to open up Task Manager it would hang. After two weeks of investigating, they discovered that it was a specific version of an anti-virus program that was causing the problem. In some cases they've come close to asking the customer to ship them their system just so they can figure it out they haven't had to do that yet, but the team has gone out and bought things like video cards to track down problems themselves. Now that's dedication!
Some bugs are really bugs, and some are product design. If they had a dozen beta testers reporting the same "bug" that was actually the way the software was designed, they'd go to the developers and make the case that the software design should be changed. They try to come up with a solution that addresses the problem without overly complicating the software design. He said that the one thing that got the developers to pay attention was that they actually had real customer data that showed end users were having difficulty. List View in Windows Vista was one such example: the development team didn't have any clue how many people used List View, and the fact that it was included in the shipping product was a direct result of beta testing feedback.
UAC (User Access Control) received a lot of feedback, because it's such an up-front issue. I didn't get a lot of experience with early betas, so I'm not sure how it worked early on. They were able to track UAC prompts, which I find fascinating they were able to measure how many UAC prompts people received in Beta 1, then compare that data to Beta 2 and see if things were getting better.
Paul said his team is full of people that are customer advocates: their goal is to show the development team that customers are having real problems with the issues they are reporting. He also said that his team isn't afraid to try proving the developers and testers wrong if they feel a customer is having an issue with the software.
By the time Microsoft reached the RTM stage of Windows Vista, they had 30,000 people testing the product. There was no shortage of people giving feedback the problem was getting high-quality bug reports. For instance, if a bug report is filed that says "Vista won't install", the bug team can't do much with that because there's so little information. If another tester reports a less severe bug, but does so with greater detail, they have a much greater chance of nailing down the problem and fixing it. I can't imagine how complicated it would be to deal with the sheer number of variables they have to deal with things like bad RAM, overclocked hardware, older revisions of video cards that behave differently than newer revisions the number of permutations is staggering. Given all that, I'm really impressed with what this team has been able to accomplish. I know my own Vista testing (limited though it may be) has found the operating system to be remarkably bug-free (so far).
Figure 1: The swag box we received after entering.
Figure 2: There were dozens of Alienware systems set up in the room - about 25 in total. Each was running Windows Vista, with a performance score of 5.3 (if memory serves), meaning they were powerful enough to guarantee a smooth ride with Vista.
Figure 3: A whole lot of geeks in one room!
Session 1: Windows Vista Logo Program (Dave Block)
The Vista Lab 2007 kicked off with a presentation about hardware under Vista and the logo certification program. I was initially thinking this was going to be a dry marketing presentation about how great the logo program was, but it ended up being quite interesting hearing about the program, and how hardware is supposed to work under Vista. Microsoft seems to keenly recognize that the experience of connecting new hardware, and installing new software, has to be easier for end users.
Unlike the Windows XP program, the Vista logo program is about more than just compatibility: it's about the overall user experience. Simple implementations, increased satisfaction, and make adding hardware and software less frustrating are the goals. He gave the example of some retailers in Europe seeing return rates in the 30% to 40% range with networking hardware this is bad news for the retailers, who are already operating on thin profit margins. They talked about the concept of "no specs necessary", meaning consumers wouldn't need to know about B or G or A. That sounds good in theory, but I think it will take several years for users to buy the newer hardware and dump the older stuff.
Figure 4: The Vista logo certification guys.
There are 17 product categories including new categories such as picture frames. He talked about products that need to pass the Windows Vista colour process test say, a printer and a monitor that are both certified, both giving you accurate colour representation. Technically, you can achieve the same thing today if you have a monitor with the proper colour profile loaded, an application using the right colour profile, and a printer with the right profile, using the right profile for the paper. This is a complicated mess of course and certainly anything that Microsoft can do to streamline this process is appreciated. Will consumers pay attention to it? Doubtful it's not like Joe User goes into a store and looks for an MP3 player that has the PlaysForSure logo on it. My hope is that all of the companies creating the hardware will feel the competitive marketing need to be in lock step with Microsoft on this issue, so everything will be better by default rather than requiring a
The first demo was of a D-Link pre-N wireless router, and it failed. It was a freshly reset unit, but it wouldn't work, so he switched over to talking about digital cameras with Windows Vista. The Nikon D40, a new camera from Nikon that will retail for $699 USD, has embedded MTP. That's very interesting because it means you could connect the camera to any Windows XP or Vista machine and be able to view, and copy to/from the camera without needing any drivers. It also means that the camera will show up to the Vista Gallery program, allowing the user to import photos directly. He said that the major camera manufacturers are on board to release firmware updates for their cameras to enable MTP. There are 15 cameras that have been submitted for certification so far, and many more are expected by the end of the year.
They tried the D-Link demo again, and it worked much better after a reboot of the PC (that doesn't say much for Vista, does it?). Basically, the router showed up in the Vista networking area, and he was able to initiate a connection between the laptop and the router, making it a secure connection, all from within Vista. He was also able to copy the router security settings to a USB flash drive, which was then passed around the room and we were all able to connect to the router even the Windows XP users. This is all done without him having to log into the router directly, and without installing any third-party client software on the PC, which is am amazing improvement over what we have today.
As a side note, I have a theory about networking hardware and Vista. I believe that when Microsoft started talking to networking vendors about making their hardware work with Vista in a seamless manner two or three years ago, the answer they got back was something along the lines of "Why should we work with you you're making your own networking hardware and competing with us." I believe Microsoft looked at their networking business which probably wasn't a big money maker for them anyway and decided it was worth killing off their own product line in order to bring all the networking vendors under the Vista tent and improve the end user experience for all users. The results seem to have paid off in a big way.
The presenter then held up a Canon HV10 camera (an HD product) and talked about it a bit the Vista compliant logo means that it must pass the streaming test and be able to stream HD video directly into Windows Movie Maker. Video cameras don't need to be MTP-compliant in order to be Vista certified, but some of them are because they incorporate digital cameras.
A great question was asked: does the Windows Vista compliant logo ensure that it has 64-bit drivers? The answer is sort of: software with the logo must work on 64-bit versions of Vista, but certified hardware does not need to have 64-bit drivers in order to be logo compliant. This cause a bit of an uproar in the room, and rightly so: if something has a Windows Vista logo, the end user doesn't know or care about 32-bit and 64-bit versions of Windows. He shouldn't have to, he just wants it to work.
I asked about Windows Vista certification for Windows Mobile devices, and was told that they have no solution in place. They're looking at it for the future, but they're unsure about exactly what would make a device compliant.
How are products tested? When it was first started five years ago, they had about 1000 products, and they created a system where each team submitted requirements for their own area. So the team that handles printing in Windows would come up with requirements that printers would have to meet in order to qualify for the logo. All hardware is machine tested (as in, tested by a computer) I'm not entirely sure how that works, but it sounds impressive. He talked about how they analyze device driver crashes, and how if they see a consistent problem, they notify the vendor. The vendor then has 60 days to update their driver to solve the problem if they don't make a driver update, they lose their certification for that product. Microsoft has no authority to force a recall of the product, however, so this is a paper tiger. Most companies wouldnt want to lose their certification (they'd have to print new boxes) so I imagine they'd try to comply.
This part was great to hear: when a customer plugs in a device, and Windows Vista doesn't have a driver, it reports that back to Microsoft. Microsoft then collects this data, and they're able to go to the vendors and encourage them to write drivers for their customers that obviously need them. I think this is a great system, but the results are what matter we'll see how many Vista drivers there are six months after launch. There are over 1.5 million device drivers out of the box with Windows Vista, which is great starting coverage, though the presenter admitted that many of those are essentially duplicated drivers for things like USB flash drives. Driver issues are one of the most painful parts of moving to a new operating system, so I sincerely hope that Microsoft and their partners get this right come launch time.
Session 2: A Day in the Life of a Bug (Paul Donnelly)
Paul started off by talking about how they started the Vista beta program. By Beta 2 they had 10,000 people who were testing Windows Vista. When they opened it up further with the preview program, they set the Customer Experience Program switch to on by default, which reported back which version of Windows Vista was installed. By coordinating this information with the release of product keys, they were able to figure out which groups of people were installing and using the Vista beta product that enabled them to allocate their resources accordingly in terms of ramping up efforts to get certain groups of people to beta test the product.
One of the first things they did with after Beta 1 was realize how out of luck people were with device drivers: they had software that submitted information about which device drivers were available, which got them the data they needed to push harder for drivers. As I talked about above, we'll see how things are after launch I really hope that hardware partners (HP, Canon, Epson, etc.) are on the ball and have drivers waiting to go. I'll be asking about Vista drivers for my R1800 when I see an Epson rep at CES, that's for sure!
Paul talked about how hard it was to track down some bugs. He gave the example of how, during testing, they were getting reports from several people who weren't able to open up Task Manager it would hang. After two weeks of investigating, they discovered that it was a specific version of an anti-virus program that was causing the problem. In some cases they've come close to asking the customer to ship them their system just so they can figure it out they haven't had to do that yet, but the team has gone out and bought things like video cards to track down problems themselves. Now that's dedication!
Some bugs are really bugs, and some are product design. If they had a dozen beta testers reporting the same "bug" that was actually the way the software was designed, they'd go to the developers and make the case that the software design should be changed. They try to come up with a solution that addresses the problem without overly complicating the software design. He said that the one thing that got the developers to pay attention was that they actually had real customer data that showed end users were having difficulty. List View in Windows Vista was one such example: the development team didn't have any clue how many people used List View, and the fact that it was included in the shipping product was a direct result of beta testing feedback.
UAC (User Access Control) received a lot of feedback, because it's such an up-front issue. I didn't get a lot of experience with early betas, so I'm not sure how it worked early on. They were able to track UAC prompts, which I find fascinating they were able to measure how many UAC prompts people received in Beta 1, then compare that data to Beta 2 and see if things were getting better.
Paul said his team is full of people that are customer advocates: their goal is to show the development team that customers are having real problems with the issues they are reporting. He also said that his team isn't afraid to try proving the developers and testers wrong if they feel a customer is having an issue with the software.
By the time Microsoft reached the RTM stage of Windows Vista, they had 30,000 people testing the product. There was no shortage of people giving feedback the problem was getting high-quality bug reports. For instance, if a bug report is filed that says "Vista won't install", the bug team can't do much with that because there's so little information. If another tester reports a less severe bug, but does so with greater detail, they have a much greater chance of nailing down the problem and fixing it. I can't imagine how complicated it would be to deal with the sheer number of variables they have to deal with things like bad RAM, overclocked hardware, older revisions of video cards that behave differently than newer revisions the number of permutations is staggering. Given all that, I'm really impressed with what this team has been able to accomplish. I know my own Vista testing (limited though it may be) has found the operating system to be remarkably bug-free (so far).
