KC - 10.2 - Selecting COTS Software - Contd...

The following sections describe details of the social-technical criteria definition, identification of alternatives and evaluation process.

1. Social-technical criteria

The evaluation criteria are parameters against which the COTS product is evaluated and upon which selection decisions are made. The criteria definition process derives these attributes from high level requirements, themselves derived from the requirements elicitation process. For a system to be cost-effective it must be driven by essential requirements, not optional or the "nice to have" requirements. Using some existing standard checklist or template to define the criteria is highly recommended. The social-technical criteria include:

1. technology factors,
2. functionality characteristics,
3. product quality characteristics, and
4. social-economic factors.

Technology factors - A technology is a specification or framework that provides for integrating components. The COTS underlying technology is selected from high-level customer requirements. However, an understanding and evaluation of the underlying assumptions about the technology must be elicited and analysed before commitment to a particular technology. A number of issues may be evaluated when considering selecting a particular technology. They include (but not limited to) the following:

• Functionality -
• Performance
• Framework and architecture style.
• Interface standard.
• Security
• Concepts of evolution and versioning.
• Development environments.

Functionality characteristics – The functions are those that satisfy stated or implied needs. The functional requirements may be represented in the form of scenarios or use-cases and should include essential customer requirements and customer standards.

Product quality characteristics - COTS component quality characteristics are behavioural properties that the product must have and should match the customer's non-functional requirements. The product quality characteristics do not necessary change from application to application. Examples include, maintainability, reliability, portability and performance (Kontio, 1996). However, this should be reviewed and adapted in accordance to customer requirements and priorities.

Socio-economic factors - these are non-technical factors that should be included in the evaluation and selection of COTS components such as costs, business issues, vendor performance and reliability.A number of social-economic issues must be considered when selecting a product or technology. They include:

• Business issues - the financial case for buying a particular product or technology. Business issues include cost of product/technology, licensing arrangements, additional cost of adapting and integrating the products, training and support costs, cost of maintenance or replacement with upgrades.
• Customer capability - examples of customer capability include customer experience with product/technology, customer expectations, internal organisational politics, and customer/organisation policies or preferences.
• Marketplace variables examples of marketplace variables include product or technology reputation, maturity and stability of a product, product or technology restrictions, market trends and viability of products over long period.
• Vendor capability variables - the performance and capability of the vendor and examples include vendor profile, reputation, certification, stability, available training and support.

2. Search for alternatives

The search for candidate COTS components is conducted through market surveys; Internet search; product publications and sales promotions; and computer fairs and shows. The identified components are screened to reduce them to reasonable number so that they can be evaluated in details.

The next step in this identification process is to obtain information about the COTS products or obtain the COTS products or both. This information may consist of evaluations by independent evaluators, reports from vendors, a demonstration of the product capabilities by the vendor and information obtained directly from users. The alternative identification depends on customer requirements, social-technical criteria and the initial screening (evaluation). However the adopted search techniques, customer participation and availability of products have an impact on the success of this process and the whole COTS selection process.

3. Evaluation (assessment)

The evaluation involves contacting vendor technical support for evaluation information, review of vendor documentation and product testing for quality and functionality. It includes evaluating COTS performance, interfaces and ease of integration, comparing short-term and long-term licensing costs against integration costs. The weighting and ranking scheme for measuring suitability for instance the use of a 1-to-5 scale is defined during this process. The reasons for selecting each component and the reasons for rejecting others should be recorded.

The evaluation criteria definition and identification of alternatives have a significant impact on the evaluation success. The following factors also impact on the evaluation process success:

1) Evaluation strategy – It is always better to employ the keystone identification strategy with the technology as the keystone issue (In keystone selection strategy, a keystone characteristic such as vendor or type of technology is selected first before selecting the COTS products. Often, interoperability with the keystone will become an overriding concern, effectively eliminating a large number of other products from consideration.). The separation of COTS underlying technology from COTS products during evaluation allows fair comparisons between products. The other advantage of separating products from technology is that useful literature is available on technology comparisons, since technology change is not as fast as product change.

2) Data collection techniques - there are a number of data collection techniques such as examining the products and vendor supplied documentation, viewing demonstration and interviewing demonstrators, executing test cases and applying the products in pilot projects. It is advisable to select appropriate techniques depending on resources and experience. In addition data collection may include interviewing actual users of the products, and examining sample outputs from projects that have used the products. Other techniques include cards sorting and laddering, outranking methods and feature analysis technique.

3) Data analysis techniques - The use of the Analytic Hierarchy Process (AHP) is best to consolidate evaluation data in order to select the "best" components among alternatives. There are tools available to support the AHP techniques. AHP was developed by Saaty (Saaty, 1990) for multiple criteria decision making and has been successfully used in software selection (Kontio, 1996) (Zviran, 1993). The AHP technique is based on pair-wise comparison between alternatives. The result of this pair-wise comparison is converted to a normalised ranking by calculating the eigenvector from the comparison matrix's largest eigenvalue. The advantages of the AHP technique are (Kontio, 1996, Zviran, 1993):

• a systematic approach for consolidating information about alternatives using multiplecriteria;
• an objective weighing technique for setting the weighing scale for qualitative and quantitative data, and
• it allows for consistency checking.

4. Customer Participation

Customer participation refers to the behaviours and activities of the customers during information system development and customer involvement refers to the participation in the system development process by representatives of the target user group. Participation may lead to increased user cceptance by developing realistic expectations about the systems capabilities, providing an arena for bargaining and conflict resolution about the selected product and leading to system ownership by users.

Customer participation may vary from direct, where all parties are affected by the system are involved, to indirect, where employee representatives serve on decision-making committees. Mumford proposes three types of participation in the selection process of the COTS product and its underlying technology.

• consultative,
• representative and
• consensus participation