Criteria for selecting software packages

What kind of criteria are important to consider when selecting a COTS software package?

First the somewhat straight-forward things...

• Current requirements: What do you need right now? How much configuration / customisation would be required?
• Future requirements: What do you anticipate you will definitely need later? What is the vendor's roadmap, including frequency of release? How expensive is it to move away from this technology? How much influence do you have over the roadmap?
• Implementability: What platform does it require? How difficult is it to integrate with existing systems? Does it have hardware scaling issues? Are there single points of failure?
• Supportability: What is available in terms of training? Consulting? Documentation? Community? How stable is the vendor? If the vendor is also intended to be an integration partner, how aligned are you culturally?
• Cost: How much will it cost to implement (license, hosting, customisation), maintain, upgrade, modify?
• Deliverability: (If customisation or extensive configuration is required) Will customisation be done through APIs (good) or will it need to be done by modifying internals (bad)? How difficult is it to test the package (especially in an automated fashion)? How difficult is it to automate installation, configuration setup,and builds (wizards are bad, scripted APIs are good)? How difficult is it to setup version control especially integrated with your existing configuration management system?

Key non-obvious points

Reduce customisation as much as possible... otherwise you will be killed by the cost and effort of upgrades.  This suggests that you would modify the business process to match the package rather than vice versa... which in turn suggests that you should generally not be considering packages for strategic business processes / capabilities. This also suggests that you want to be very clear about and enforce boundaries to prevent package features creeping into strategic areas.

Just because the product offers a feature, doesn't mean you should turn it on or use it.

According to Capers Jones, at 25% customisation, it's cheaper in the long-run to build a custom system instead and 15% customisation is a safer number to use. If the vendor is hostile, the number drops to 5%.

Customisation or extensive configuration highlights the need for deliverability.  If the package is like an appliance (e.g., Microsoft Word), it should just work. Initial selection and upgrades might be more about manual, exploratory testing. However, once we start introducing customisation, then the importance of being able to setup automated testing (as well as other development features) grows.

Configuration, especially if extensive, should not necessarily be treated with any less rigour than custom development. 

Especially with the typical COTS packages in terms of CRM, ERP, financials, the main options tend to have feature parity which means you should generally focus on other aspects.  This could be vendor alignment, testability, modifiability, etc.

It is better to reduce commitment than attempt to commit to a perfectly correct decision.  If the package does not require as much commitment (e.g., hosted service), we've maintained options to change our mind later and don't need to worry as much about making an optimal decision up front.

Don't pick the right technology.  Pick the technology which is cheapest to move away from.

Chris Matts

The factors that increase commitment are primarily the size of the initial investment and the cost of migrating data. The size of initial investment is about sunk cost fallacy which makes it a psychological / political factor, not an economic one. 

Scripted customisation scenarios may be even more important than scripted demos.  In a customisation situation you want to assess the modifiability of a package in a consistent way by providing several customisation scenarios and seeing how vendors are able to respond.

(Thanks to input from Jonny Leroy, Tim Brown, Martin Fowler, John Hume, Ahrum Song, Scott Shaw, Evan Bottcher, Rebecca Parsons, and Aaron Erickson)