An Update on Bug Reporting, Part 2
Posted: Fri Aug 16, 2013 2:47 pm
http://community.eveonline.com/news/dev ... ng-part-2/
Hello again, CCP Goliath here to show you our new bug reporting tools! I discussed the background of bug reporting and EVE in my previous blog. Today I want to talk about the future, and how you can help us to make EVE an even better experience for everyone.
Let‘s begin by discussing briefly how we go about ensuring EVE is as bug-free as possible (but if you just want to see the new stuff, jump ahead a few paragraphs!). In our various stages of development, the QA Analyst of each feature team will work on creating a test plan, to ensure that everything their team works on goes through an appropriate amount of testing. They look at functional testing, to ensure that the core behaviour of their feature works as intended, then get into some destructive testing, to test the limits of the system when it is used for different purposes than it was built for, and finally round it out with some exploratory testing, checking out the systems that are touched by the new feature to ensure they have not been changed or broken. This sounds like a Herculean task of course, but they are aided by their team in an initiative we call Whole Team Quality – essentially everyone pitching in when there‘s work to be done! The testing doesn‘t end there though – in parallel to this development we have a team of 10 running our regression suite. This is a series of test cases covering all of the core features of EVE (and then some!). It takes 2 weeks for the 10 to run through all of the tests, so it‘s extensive and gives us a lot of confidence in our product.
In further addition to this, our volunteer Bug Hunters will be checking things out even before we go public. They tend to work in a more exploratory fashion and are guided by the ever capable CCP Vertex. Once things are in a state we are happy to move forward with, public testing commences. We deploy our build to Singularity and open the gates to those interested in getting an early look at our new features. These players submit bug reports to us, which we investigate and attempt to reproduce, prioritize and fix before our deployment to Tranquility. We also run mass tests on Singularity to replicate as best we can the conditions of a large fleet fight. This is a great workout for Time Dilation and rewarding for those who participate, as they get 2 million Skill Points on Singularity to let them try out some stuff before skilling for it on Tranquility.
Despite all of this rigorous work that goes into assuring the quality of EVE, some bugs inevitably slip through our net. Typically these are issues that you, our devilishly intelligent players, have discovered through inventive uses of our systems (putting it mildly). When such issues arise, it‘s obviously important to get that information to us so that we can reproduce it, diagnose it and get it fixed as quickly as possible. This is where bug reporting comes in.
Firstly, we want to show off the new bug reporting site itself. Developed from scratch by Team Roundhouse Kick to fit with the new look and feel of the EVE websites, it‘s a sleek, easy to use method of getting information to us. Let‘s take a look at some screenshots!
As you can see, we‘ve improved and explained the category selection system so that you can be sure that the correct people will end up looking at your report.
We‘ve included example text in all of the fields so that you don‘t need to worry if you‘re writing the correct thing in the correct place. We‘ve also automated the collection of build number when you select your server from the dropdown, which lets us pinpoint the defect in our code. Additionally we have added the ability to upload multiple files at once, since invariably each report would ideally include a screenshot or two, a logfile and a dxdiag file. You can also view the progress of the uploads in real time. We‘ve even added a language field when you report a bug in the localization category, so that our Russian, German and Japanese players can get their reports to people that understand them – though do note that there is no plan to localize the bug reporting site at this time.
When we changed our internal defect tracking system in June, CCP Tuxford and CCP Paradox from Team Superfriends did some work to ensure that the In Game Bug Reporting tool was compatible with it. For those of you that have never encountered it before, it is located in the Help menu of the client (F12) and is different from the site in a couple of ways.
It automatically collects some logs and settings files from your client, which lets us diagnose the issue much faster and saves time for you. It can also take screenshots which can be annotated by the tool itself. Unfortunately there is no example text or guidelines at the moment, so it‘s going to be more difficult to use for a neophyte bug reporter. Don‘t panic though – we have some guidelines on Evelopedia and I will be writing a third blog shortly on what makes the perfect bug report! Please also note that getting bug reports via the in game tool is generally more useful to us than getting them via the site, so if you can use the tool, please do so.
So, this is what we‘ve been working on. There is more to do - for instance the 2 way communication between the bug reporter and our development team that was in the previous system and is something we will implement soon, but for now please be extra vigilant to include as much information as possible as we will not follow up bug reports with requests for more information. Also there is currently no way for you to see your bug report once it has entered the system (which means no editing either) and this is also scheduled to be added. We hope you find it to your liking, but if you have suggestions or feedback, please leave them in the comments section of this blog and we will analyse them. I‘ll be back with my bug report blog in a couple of weeks. Thanks for reading!
Hello again, CCP Goliath here to show you our new bug reporting tools! I discussed the background of bug reporting and EVE in my previous blog. Today I want to talk about the future, and how you can help us to make EVE an even better experience for everyone.
Let‘s begin by discussing briefly how we go about ensuring EVE is as bug-free as possible (but if you just want to see the new stuff, jump ahead a few paragraphs!). In our various stages of development, the QA Analyst of each feature team will work on creating a test plan, to ensure that everything their team works on goes through an appropriate amount of testing. They look at functional testing, to ensure that the core behaviour of their feature works as intended, then get into some destructive testing, to test the limits of the system when it is used for different purposes than it was built for, and finally round it out with some exploratory testing, checking out the systems that are touched by the new feature to ensure they have not been changed or broken. This sounds like a Herculean task of course, but they are aided by their team in an initiative we call Whole Team Quality – essentially everyone pitching in when there‘s work to be done! The testing doesn‘t end there though – in parallel to this development we have a team of 10 running our regression suite. This is a series of test cases covering all of the core features of EVE (and then some!). It takes 2 weeks for the 10 to run through all of the tests, so it‘s extensive and gives us a lot of confidence in our product.
In further addition to this, our volunteer Bug Hunters will be checking things out even before we go public. They tend to work in a more exploratory fashion and are guided by the ever capable CCP Vertex. Once things are in a state we are happy to move forward with, public testing commences. We deploy our build to Singularity and open the gates to those interested in getting an early look at our new features. These players submit bug reports to us, which we investigate and attempt to reproduce, prioritize and fix before our deployment to Tranquility. We also run mass tests on Singularity to replicate as best we can the conditions of a large fleet fight. This is a great workout for Time Dilation and rewarding for those who participate, as they get 2 million Skill Points on Singularity to let them try out some stuff before skilling for it on Tranquility.
Despite all of this rigorous work that goes into assuring the quality of EVE, some bugs inevitably slip through our net. Typically these are issues that you, our devilishly intelligent players, have discovered through inventive uses of our systems (putting it mildly). When such issues arise, it‘s obviously important to get that information to us so that we can reproduce it, diagnose it and get it fixed as quickly as possible. This is where bug reporting comes in.
Firstly, we want to show off the new bug reporting site itself. Developed from scratch by Team Roundhouse Kick to fit with the new look and feel of the EVE websites, it‘s a sleek, easy to use method of getting information to us. Let‘s take a look at some screenshots!
As you can see, we‘ve improved and explained the category selection system so that you can be sure that the correct people will end up looking at your report.
We‘ve included example text in all of the fields so that you don‘t need to worry if you‘re writing the correct thing in the correct place. We‘ve also automated the collection of build number when you select your server from the dropdown, which lets us pinpoint the defect in our code. Additionally we have added the ability to upload multiple files at once, since invariably each report would ideally include a screenshot or two, a logfile and a dxdiag file. You can also view the progress of the uploads in real time. We‘ve even added a language field when you report a bug in the localization category, so that our Russian, German and Japanese players can get their reports to people that understand them – though do note that there is no plan to localize the bug reporting site at this time.
When we changed our internal defect tracking system in June, CCP Tuxford and CCP Paradox from Team Superfriends did some work to ensure that the In Game Bug Reporting tool was compatible with it. For those of you that have never encountered it before, it is located in the Help menu of the client (F12) and is different from the site in a couple of ways.
It automatically collects some logs and settings files from your client, which lets us diagnose the issue much faster and saves time for you. It can also take screenshots which can be annotated by the tool itself. Unfortunately there is no example text or guidelines at the moment, so it‘s going to be more difficult to use for a neophyte bug reporter. Don‘t panic though – we have some guidelines on Evelopedia and I will be writing a third blog shortly on what makes the perfect bug report! Please also note that getting bug reports via the in game tool is generally more useful to us than getting them via the site, so if you can use the tool, please do so.
So, this is what we‘ve been working on. There is more to do - for instance the 2 way communication between the bug reporter and our development team that was in the previous system and is something we will implement soon, but for now please be extra vigilant to include as much information as possible as we will not follow up bug reports with requests for more information. Also there is currently no way for you to see your bug report once it has entered the system (which means no editing either) and this is also scheduled to be added. We hope you find it to your liking, but if you have suggestions or feedback, please leave them in the comments section of this blog and we will analyse them. I‘ll be back with my bug report blog in a couple of weeks. Thanks for reading!