Lee Jack, a software testing consultant/IT services Sales Manager is tensed about his proposed meeting with Mark Johnson, a Senior IT managers for a large IT group. Mark, after a cold call, is looking forward to sell testing services to Lee’s group and hence to his entire IT group. Here is how the conversation goes.
Lee Jack: What is your problem statement?
Mark Johnson: Not sure. We recently had a cold call – you say you provide Testing services what you can offer me?
LJ: You need to focus on Independent testing
MJ: We have been managing with developers doing testing for last 6 years – have no problems whatsoever. Why do I need to bother now?
LJ: May be your developer testing is not proper or inadequate or it is costly?
MJ: How can you say that?
LJ: Tell us about production defects and quality of your applications in general…
MJ: I told you already. We have no production problems that we worry about. Our site is every now and then goes down. We wait for 30-40 mins and start over again everything works fine. Why bother?
LJ: Do you say your applications are of high quality?
MJ: I would say our applications are of “good enough quality”. I get what I pay for and I have the quality for the application that my users care for. So far so good.
LJ: What are your “business plans” going forward? New growth etc? What you seem to be saying “good enough may not be good in the future?
MJ: May be. I don’t think that far ….
MJ; Have you thought about cost of testing? Do you have any mandate from your boss about reducing IT budget and hence reduced budget for testing?
MJ: Oh!! Yes. We have taken care of the larger part of cost problems. We have nearly replaced all our commercial software stack by open source equivalents. Same is true even for tools. So we can say we run our entire shop on open source tools. That is our achievement.
LJ: Tell us about your development /testing model ?
MJ: We are pretty advanced from that angle. We follow scrum based agile development model – outsourced. Developers do all required testing and development and everything is fine. I am hearing you talking about QA again and again without me asking about it. We do agile model and as such there is no distinction between development and QA. As such we do not think cost is our concern.
LJ: I did not say “QA”, I said “testing” – I suppose you know the difference.
MJ: Fine … here we use these terms interchangeably. Well, does that matter in agile?
LJ: How can we help you?
MJ: You have been saying that you give QA , sorry testing services – Do you provide white box testing services?
LJ: Our Testers are trained well in both black box and white box techniques – that should not be a problem.
MJ: Let me make it very clear – in the name of QA or Testing, I don’t want some so called tester coming and doing “negative” testing and showing 100’s bogus UI level bugs. That is not what I want.
LJ: Our testers are trained in exploratory testing and can flush out hidden bugs
MJ: I said I do not want negative testing – UI level keyboard banging. I want white box testers.
LJ: What according to you is white box testing?
MJ: I am surprised that you are asking this silly question. It is the testing done with the knowledge of product internals. Something that developers do. Don’t you have such testers .. I mean developers?
LJ: Well, the reason I asked this question is – there are no universally accepted meanings of the term white box testing and I wanted to make sure I understand your version. OK Why can’t your current developers do that?
MJ: Our developers are already stressed out. We have very well defined agile processes. Our developers do very smart and JUnit based automated tests are our key to success.
LJ: OK … What can we do for you? It seems that you have everything that you need.
MJ: How about your folks doing an assessment and tell us what our maturity is? How well we are placed with respect to industry standards?
LJ: That appears like a different problem …. Anyway, are you sure You want an assessment?
MJ: Yes. I would like our processes to be best as per industry standards.
LJ: What are your expectations out of this assessment?
MJ: We need to add few white box testers to our existing team as we are weak in that area. We are sure we do not need QA resources that do negative or exploratory testers. We are also sure we do not need testers. We need developers who can do white box testing.
LJ: Oh!!!! That is what you want? I can give you developers who can do white box testing? Why you need assessment?
MJ: You see, we are looking for a consultant to make case for us to get budget for these resources. We really do not worry about testing, QA, White/black box testing etc. Can you get a consultant?
LJ: Great … I will have a top notch consultant at your office tomorrow.
MJ: Thank you. Make sure, you get someone who understands our problem very well. I don’t want to go through all these questions again.
LJ: So, let me recap. What is your problem?
MJ: Damn it …. I need a business case for increasing my team’s headcount and I need white box testers. Are you clear?
LJ: Yes … Sure …
MJ: Thank you very much.
What do you think … what is the problem here?
Update: I did a mistake of swapping the names of MJ and LJ. This is corrected. See how it reads now.
Shrini
Thursday, October 15, 2009
Tuesday, October 06, 2009
Necessary Tester skills ....
Curiosity? Keen and close observation? Skepticism? Creativity? Arguing/debating? Imaginative? Analytical and Holistic? Good at Probing/Investigation?
None – would say many IT testers, I meet on regular basis
Today’s IT testers have forgotten these terms I suppose. When looked at macroscopic and group level – todays Test managers, delivery heads appear to have lost sight on these core skills for testers. When I ask any typical IT testers what they are doing to enhance the skills, typical answers I get are - learning domain, usage of specific tool, or a programming language, or some process/methodologies stuff. That is not wrong but having any of these skills without the basic skills required for testers – we would be developing the armies of robots who are very good at following orders and over a period of time, stop thinking on their own.
NY Police (a profession similar to testers) are taking course in “observing and describing” – What our testers should be doing to sharpen their testing skills? How they are putting their cognitive skills in their day today work?
http://www.smithsonianmag.com/arts-culture/Teaching-Cops-to-See.html
Shrini
A process is what you actually do. A process document describes what someone would like you to do, ideally. They rarely coincide exactly, and sometimes don't overlap at all - Jerry Weinberg
None – would say many IT testers, I meet on regular basis
Today’s IT testers have forgotten these terms I suppose. When looked at macroscopic and group level – todays Test managers, delivery heads appear to have lost sight on these core skills for testers. When I ask any typical IT testers what they are doing to enhance the skills, typical answers I get are - learning domain, usage of specific tool, or a programming language, or some process/methodologies stuff. That is not wrong but having any of these skills without the basic skills required for testers – we would be developing the armies of robots who are very good at following orders and over a period of time, stop thinking on their own.
NY Police (a profession similar to testers) are taking course in “observing and describing” – What our testers should be doing to sharpen their testing skills? How they are putting their cognitive skills in their day today work?
http://www.smithsonianmag.com/arts-culture/Teaching-Cops-to-See.html
Shrini
A process is what you actually do. A process document describes what someone would like you to do, ideally. They rarely coincide exactly, and sometimes don't overlap at all - Jerry Weinberg
Friday, September 11, 2009
Who decides what is a bug and what should be fixed?
I was submitting a proposal for a paper for STC 2009 conference. After completing all fields – about 20 of them , I accidently hit clear button (by practice – “OK” or “submit” button appears first in such forms) and all that I entered is “gone” – worst there is no way to recover.
Is this a bug?
Another catch – for the date field – no format is specified, neither there is a calendar control. How do I find out the required format? Try one and get to know about the format.
Is this a bug?
If you were a tester who would strictly go by “test cases” generated out of “specifications” – you are likely to miss such bugs – remember there is something called “Requirement based testing”. Alternatively, you might argue that these are not bugs or bugs of low severity. Who has the final authority to say which is a bug and which one should be fixed and when?
Shrini
Wednesday, August 19, 2009
Your user acceptance testing is fixed – Is that a problem?
“This troublesome user acceptance testing got to be easier. It is really a pain” said of my colleague quoting an IT manager responsible for release management and user acceptance testing. Major problem according the IT manager was getting the “users” to do their job – testing a defect fix, feature enhancement or a new release. Users are always busy with their “day job” and for most of them doing testing is a low priority activity. I have heard many similar stories regarding this important aspect of “end user” responsibility.
In traditional IT organizations, user acceptance testing is a formal stage in the testing life cycle where business users will test the proposed changes to production software applications. IT organization that delivers the software to meet the needs of business users, require a (formal) approval of proposed changes that come in terms of defect fixes, enhancements and new releases. Since it is business users that ask for changes for their existing and fund the development/testing work – they will have the final say on “acceptance” of proposed changes to production applications.
Primary purpose of user acceptance testing is check (cursory- final) proposed changes to the production software by those users that asked for the changes so that any last minutes changes and surprises are avoided. User acceptance test is the last gate before software goes Live so that it provides last opportunity for all involved (both IT and business) to make corrections if required. Depending upon the nature of business supported, nature of software application, nature of changes proposed UAT may last for few days to few weeks. Software that fails in UAT is typically assumed to be insufficiently tested and is variably returned to IT for fixes and more testing.
The problem of UAT
Business users who hold the responsibility of approving software updates to production systems, typically are not testers and as such do not come with testing skills. Depending upon the nature of software updates, IT team will ask business users to carry out testing to check the features of new system. Most of the user acceptance testing tends to be repetitive and that is what drives “real users” crazy. For development team (IT), their work is not complete until the piece of code is user tested and accepted. Hence they literally chase user group to perform UAT with users complaining about the time crunch and their “important” business work getting impacted. This creates a situation where both parties (IT and business) want UAT to be somehow completed so that each can carry out their business as usual. Hence UAT often gets “fixed” with the consent of both parties having stakes in the activity.
Forms of fixing UAT
Between Business and IT, UAT can be fixed in number of ways – some of these are reasonably business driven models given the constraints.
1. UAT will be performed by the members from IT team where business only reviews the results of the test. If the results are OK then UAT is ought to have been completed
2. UAT will be performed by a third party, such as someone from support staff. Business will review the results and accept the proposed changes if results are OK
3. Business will provide a prescribed set of test scripts that can either be used by anyone IT staff or support staff. Results will be verified by the Business.
4. UAT will be done like a demo to the users where IT staff will execute some pre-approved test scenarios related to proposed changes.
5. IT team will train the business in new features proposed to be introduced. Business users after training, test (use) the proposed software and accept the software
Why fixing user acceptance testing is bad thing?
Why bother to UAT after all? For IT, more often than not, it is a formality to be completed before they push the code to production.
In my opinion, it is the spirit and purpose of UAT that gets compromised. Typically, in spite of all best efforts, the depth and frequency of interactions between IT and business throughout the project remains low. When business users do not participate with full spirit in UAT, lots of things go unnoticed into production. This might results users (non participating ones especially) getting surprised when they see the product.
What has been your experience? Do you bother if you see your UAT is fixed?
Shrini
In traditional IT organizations, user acceptance testing is a formal stage in the testing life cycle where business users will test the proposed changes to production software applications. IT organization that delivers the software to meet the needs of business users, require a (formal) approval of proposed changes that come in terms of defect fixes, enhancements and new releases. Since it is business users that ask for changes for their existing and fund the development/testing work – they will have the final say on “acceptance” of proposed changes to production applications.
Primary purpose of user acceptance testing is check (cursory- final) proposed changes to the production software by those users that asked for the changes so that any last minutes changes and surprises are avoided. User acceptance test is the last gate before software goes Live so that it provides last opportunity for all involved (both IT and business) to make corrections if required. Depending upon the nature of business supported, nature of software application, nature of changes proposed UAT may last for few days to few weeks. Software that fails in UAT is typically assumed to be insufficiently tested and is variably returned to IT for fixes and more testing.
The problem of UAT
Business users who hold the responsibility of approving software updates to production systems, typically are not testers and as such do not come with testing skills. Depending upon the nature of software updates, IT team will ask business users to carry out testing to check the features of new system. Most of the user acceptance testing tends to be repetitive and that is what drives “real users” crazy. For development team (IT), their work is not complete until the piece of code is user tested and accepted. Hence they literally chase user group to perform UAT with users complaining about the time crunch and their “important” business work getting impacted. This creates a situation where both parties (IT and business) want UAT to be somehow completed so that each can carry out their business as usual. Hence UAT often gets “fixed” with the consent of both parties having stakes in the activity.
Forms of fixing UAT
Between Business and IT, UAT can be fixed in number of ways – some of these are reasonably business driven models given the constraints.
1. UAT will be performed by the members from IT team where business only reviews the results of the test. If the results are OK then UAT is ought to have been completed
2. UAT will be performed by a third party, such as someone from support staff. Business will review the results and accept the proposed changes if results are OK
3. Business will provide a prescribed set of test scripts that can either be used by anyone IT staff or support staff. Results will be verified by the Business.
4. UAT will be done like a demo to the users where IT staff will execute some pre-approved test scenarios related to proposed changes.
5. IT team will train the business in new features proposed to be introduced. Business users after training, test (use) the proposed software and accept the software
Why fixing user acceptance testing is bad thing?
Why bother to UAT after all? For IT, more often than not, it is a formality to be completed before they push the code to production.
In my opinion, it is the spirit and purpose of UAT that gets compromised. Typically, in spite of all best efforts, the depth and frequency of interactions between IT and business throughout the project remains low. When business users do not participate with full spirit in UAT, lots of things go unnoticed into production. This might results users (non participating ones especially) getting surprised when they see the product.
What has been your experience? Do you bother if you see your UAT is fixed?
Shrini
Sunday, July 26, 2009
Tom DeMarco’s confession
Confession is probably a harsh to describe what Tom DeMarco, the creator of celebrated punch line of software managers – "You can't control that you can't measure", wrote recently. But, if you read Tom's recent article "Software Engineering – An idea whose time has come and Gone?" in IEEE software, you would probably say something similar to this. In this short 2 Page article, you will find Tom DeMarco in reflective and retrospective mood.
In the book "Controlling Software Projects: Management, Measurement, and Estimation (Prentice Hall/Yourdon Press, 1982) ", Tom talked about "Controlling" and "measuring" in Software Engineering. Nearly 40 years later, he now appears to admit that he pushed the notion of "control" and "measurement" too much.
This "confession" has apparently caught the attention of many. Jeff Atwood writes an obituary to "software engineering". More than his post, the comments for the post of Jeff Atwood are interesting to read. How come, suddenly so many are accepting now that "software" is human centric, people oriented, "engineering" is not the right term to use and so on? Michael Bolton's recent article (three kinds of measurements and two ways of using them) on stickyminds appears have been triggered by Tom's "confession". Matt Heusser writes about metrics here, here and here.
Managing vs controlling
In his own admission, Tom seems to distinguish between controlling and managing. He gives the example of "upbringing of a teenager". As any family therapist would recommend, you manage your teenage kid rather than controlling them. Like in most human endeavors (including software programming and testing), you can manage quite lot things than controlling them. I think we can manage (and also a control a bit) WITHOUT measuring ANYTHING at all. I like Matt Heusser's example of hair cut. Many of us control and manage our hair (style) without measuring.
Manage people and Control Money and timelines
This is Tom's recipe for managing the project without controlling it. While breaking project into human elements and non human elements (such as code, schedule, money) is a welcome change, I am not sure how can you do it – managing people and control money and timelines. To me, these two sets of things are NOT mutually exclusive so that you can treat each in a different way. Controlling time and money impacts people and managing people impacts timelines and money.
Some software is really engineered!!!
Dave Markle makes an interesting point in Jeff Atwood's post - "IMO you can't say that programming languages themselves aren't engineered based on solid computer science. You can't say that something like LINQ hasn't been engineered. Whenever you use a FSM in your software, you are applying computer science, which makes you a software engineer"
Programming languages are engineered, operating systems are engineered and so are software algorithms. It probably the user mass/ size of the group decides engineering vs. crafted.
Dave uses the Paint analogy nicely to drive home the point - the paint artist uses is engineered (developed and mass produced using the principles of chemistry and physics) where as the "art" produced by the artist is NOT. This stirs the nest of debate of what is engineering and what is craftsmanship. Are Engineering and Craftsmanship are mutually exclusive? I am afraid NOT.
Tom, Damage has been done and is still happening. Many still many abuse your punch-line to push loads of documentation, process, approvals, and meetings and of course endless charts/graphs of metrics – all in the name of "control". Probably the time has come to step back and be sensible on "measurements" in software.
Shrini
Tuesday, June 16, 2009
To tweet or to blog ...?
Due to me crazy traveling and work, I have not been able to write as frequently as I would have liked, for the blog. At least half a dozen potential blog posts are waiting to see the light of the day from my side. It is just not working out.
I have been on twitter (for starters - you can take this as a quick blogging or microblogging) quite active in recent days. Happy to see many following me now. It is suiting me for now as I need not feel guilty of not being able to discharge my duties as a blogzen (no!!! this word has not yet been used by someone previously)of software testing blogosphere.
So till, I get to full time blogging - please follow my thoughts on twitter. I have added twitter feed on this blog to facilitate for my readers to catch up with what I am working on ....
Thanks for being my blog reader ....
Shrini
I have been on twitter (for starters - you can take this as a quick blogging or microblogging) quite active in recent days. Happy to see many following me now. It is suiting me for now as I need not feel guilty of not being able to discharge my duties as a blogzen (no!!! this word has not yet been used by someone previously)of software testing blogosphere.
So till, I get to full time blogging - please follow my thoughts on twitter. I have added twitter feed on this blog to facilitate for my readers to catch up with what I am working on ....
Thanks for being my blog reader ....
Shrini
Thursday, May 14, 2009
10 ways to make automation difficult or ineffective
Here goes another 10 items list for automation. If you are in IT or IT services space and manage/deliver automation solutions -- make sure you stay away from these as these items have high likely hood of making automation inefficient/ineffective and difficult.
This list is an extension of a topic and this list (of 10 items again) for test automation outsourcing
10. Wild Desire to automate 100%
9.Attempting to automate existing test cases without scrutinizing them for “suitability” to automate
8. Mapping test case to script 1:1 linear model – falling prey to deceptive traceability and gold plated reporting.
7.Not building automation solution bottom-up , unidentifiable building block of the solution.
6. Trying only one type of automation or attacking only one layer of the application – Farther you go from code, messier it gets.
5. Focusing only test execution related tasks
4. Treating automation as scripting – ignoring “generally accepted good software development practices for hygiene.
3. Failure to involve developers from the beginning – Not attempting to testability or automatability of the application.
2. Jumping to automation to speed up testing or save cost before fixing testing problems – inadequate, inefficient and broken.
1. Failure to arrive (formulate) at the right mix of human testing and automated test execution.
0. Using Automation as solution to testing problems.
I reiterate that these are applicable mostly to COTS driven, GUI functional Testing automation that is typical in IT/IT services environments. WI might have to rewrite some of these for xUnit type formalized unit testing (that is also automation and some call it even as "testing").
Shrini
This list is an extension of a topic and this list (of 10 items again) for test automation outsourcing
10. Wild Desire to automate 100%
9.Attempting to automate existing test cases without scrutinizing them for “suitability” to automate
8. Mapping test case to script 1:1 linear model – falling prey to deceptive traceability and gold plated reporting.
7.Not building automation solution bottom-up , unidentifiable building block of the solution.
6. Trying only one type of automation or attacking only one layer of the application – Farther you go from code, messier it gets.
5. Focusing only test execution related tasks
4. Treating automation as scripting – ignoring “generally accepted good software development practices for hygiene.
3. Failure to involve developers from the beginning – Not attempting to testability or automatability of the application.
2. Jumping to automation to speed up testing or save cost before fixing testing problems – inadequate, inefficient and broken.
1. Failure to arrive (formulate) at the right mix of human testing and automated test execution.
0. Using Automation as solution to testing problems.
I reiterate that these are applicable mostly to COTS driven, GUI functional Testing automation that is typical in IT/IT services environments. WI might have to rewrite some of these for xUnit type formalized unit testing (that is also automation and some call it even as "testing").
Shrini
Subscribe to:
Posts (Atom)
