When assessing new products, whether it’s software or hardware, you should have a clear process on how to asses whether the product is meeting all of your requirements. Below is that very process that we’ve used in our organisations many times and has served us well when assessing technologies. The proof of concept should be broken down into the following stages:
- Requirements
- Research
- Short List
- Trial
- Validation & Recommendation
Requirements
You may be wondering why requirements is before research? Majority of the time you’ll have identified a gap that needs filling and this is normally the justification for a business case as to why you should do a proof of concept in the first place.
When recording your requirements I always find it’s easier to group them into categories and provide a metric or weight against each requirement so that when others read the artefact it becomes clear as to which requirements were crucial in making the decision to acquire a product.. Below is a simple table to use when defining your requirements:
| Category | Requirement | Rating |
|---|---|---|
| Functional | Intuitive/easy to use. | Must have |
| Functional | Integration to other capabilities | Nice to have |
| Non-Functional | Scale horizontally | Must have |
Research
Let’s not reinvent the wheel. There are companies, that their sole purpose is to do technology research and can save you a lot of time in narrowing down vendors and technologies. Gartner and Ovum are examples of these companies. Both are paid for subscriptions but if you do a quick search you’ll find many of the vendors they analyse will actually give you copies of these reports for free.
These are normally great starting points to understand who the players are in the market and also useful in assisting to create a short list. Be diligent in reading the reports to understand how they’ve assessed the vendors and the capabilities are being reviewed. Ensure they align with your requirements.
Short List
You now have two pieces of data to help create a short list. Your list of requirements and a list of vendors that on face value can meet those requirements. So how do you narrow this down further? Which vendor or product should you do a proof of concept with? This is where a vendor questionnaire comes in handy. Take your list of requirements and turn it into a list of open and precise questions that you can email to 5 or so vendors to narrow down the list further.
The questionnaire should ask the vendor to provide examples or links to documentation where they believe they can meet the requirements. Here is a sample vendor questionnaire used to ask Application Performance Management vendors about their product
| Category | Question | Response and References |
|---|---|---|
| Functional | Does your product instrument .Net applications and what versions of .Net does it support? | |
| Functional | Does your product instrument java applications and what versions of java does it support? | |
| Functional | What role-based access controls are available and how granular are they? | |
| Non-Functional | Is there is a limit of how many applications and/or agents the product supports? |
Once you have a list together, use the information from your research to identify which vendors to send it out to and ask for a formal response back. Once they’ve had a chance to digest and reply back to the questions you then need to evaluate their responses and identify which you want to bring in for an interview. You should score each of the questions based on whether the requirements were met.
- For every “must have” requirement the vendor claims to be able to meet score it a 2
- For every “nice to have” requirement the vendor claims to be able to meet score it 1
- For every other requirement that was not met then it’s given a 0
The “must have” should be weighted higher as they are the critical requirements you wish these technologies to have. At the end you should then have a total score for each vendor. Some answers may require validation with the vendors to ensure they’re scored as accurately as possible so you would normally have a follow up meeting with each vendor to validate and understand the response.
You should now have a scorecard for each vendor to use to move onto the next step. At this phase you should start to have an indication from the vendor on the costs to run the technology as this will obviously have a bearing on the selection.
Trial
Because of time and budget you would normally only do one proof of concept with one vendor. This is where the scorecard comes in handy to identify which of the vendors to do the proof of concept with. Before you commence the trial you need to identify where best to run the proof of concept so that you can assess the proof of concept against all of your requirements. Often you won’t be able to run a proof of concept in a production environment and nor should you, so therefore you need to identify a suitable environment which is as close to “prod-like” in order to understand how it would work in your organisation with your assets.
Make sure between the vendor and your business you have:
- Define what success looks like.
- How long the proof of concept will run for?
- And also you will need a test plan that aligns with requirements that were defined in phase 1.
The test cases need to cover off the functional and non functional requirements. Be extremely thorougher in testing, every product has limits and as part of these tests you should look to understand what those limits are and what they mean for you.
Don’t be afraid to share the test cases with the vendor and in fact make sure they’re included when you run through the test cases. It’s with in your interest to ensure you’re using the product correctly whilst doing the test cases so that there is no misjustice done by yourself when going through the test cases.
Validation & Recommendation
Now the proof of concept has completed, you need to validate the results of the proof of concept and asses whether enough of the requirements have been met or do you need to go back to a trial with another vendor? Either way you have evidence now to support your decision.