Lifecycle Management

Work Completed By: Tim West, Sukhpal Shergill and Winston Castle

DESCRIPTION

Within any architecture there needs to be a standard method by which changes can be introduced in a controlled fashion and where development can take place in such a way that there is freedom to test new approaches and solutions without adversely impacting the live environment. Frameworks, such as ITIL, provide guidance around roles, responsibilities and processes underpinned by an infrastructure that is capable of supporting this approach.

This project is concerned with the impact a lifecycle approach has on the underlying infrastructure.

If we define the lifecycle stages (see below) we can start to consider how the infrastructure needs to operate in order to meet the needs of each one.

Lifecycle Stages
*Development
*User Acceptance testing

  • Pre-release
  • Live
  • Retirement

This project will investigate lifecycle management and what role infrastructure plays within each stage, what issues exist and how they could be overcome.

DETAILS/LOCATION OF PRACTICAL DEMONSTRABLE OUTPUT

Details of where code, web pages etc can be found.

Introduction

This resume is generalised and needs to be considered against what is being tested in each individual case. A small change may not need full or exhaustive testing against the complete pre-release environment. So each test needs to be seen on its merits. It is not possible to be prescriptive in every case and these notes are merely a guide.

The Lifecycle Stages were considered and the conclusion was that separate environments are required for some of the various lifecycle stages. Obviously it is very expensive to provide these so the number was reduced to the bare minimum.

Development

This stage has the widest remit and tests range from patch testing to new software systems. The main thing though is that it is a Sandbox where testers can make what changes they like and the changes are pretty much in isolation to other things. At this point we are looking at how feature rich a proposed change is and to demonstrate it working. The earlier work done to create A Private Cloud is a demonstration of this.

User Acceptance Testing

This stage also requires a test structure. Here we try to see how tough the change is and try to break it. We test it to see how it works under stress and in an area which is more like the live environment. Do other things that happen in WCC affect it? This is where we find out. This environment will also be used to develop and test new internal applications and infrastructure environments. Proof of Concept and Model Office work would be carried out in this space.

Pre-release

At this stage the changes have been done and the product should be ready. A testing area is required which must mimic the live environment as closely as possible. We are now testing that the introduction of the change will have little impact on LIVE apart of course from the desired effect. This is in effect a rubber stamp where we prove the change will work and there should be no issues going to Live. If we have done the job properly we can have confidence that we can do the roll-out at night at a quiet time automatically as we know it will be a success. Having a team standing by to fix it is overkill.

Live

By definition this is up and in use. Nothing is required.

Retirement

Removing a product should not be an issue as a replacement should have been found if necessary and in use before the removal. The only issue could be some side effects. For example the old issue of .dlls being shared by other programs. Also your data may be being used by other programs e.g. data on a website may be being fed to a mashup which then stops working after retirement. However prior to removal users and Developers will have been warned so others who use the data will have notice of the change and can appeal if they wish.
Therefore no real Business case for a test 'Retirement' environment could be made.

Requirements for each Testbed considered in more detail

Development wip

Development is a large and diverse area of testing.
At the Research end of the scale a change can be considered as in a lab. It is in isolation from the rest of the world. So the environment can be as small as possible. Many labs could be effected on top end laptops which could probably run a couple of servers and a client. Where a product may need say a DNS server this could just be a HOSTS file entry to minimse the environment. However even here consideration must be given to the running of the testbed on a works laptop. Will it affect Live? Should it be on the corporate LAN? Should it be on a test LAN with no or limited access to Live? Should the VMs Networked cards be bridged or kept within the confines of the laptop? Should there be a clearance procedure for bridged VMs being visible on LIVE? Should the server names used go through the request procedure and the files kept and logged or just maintained by the developer?
As the phase continues and the work becomes more 'Development' orientated then the work

Moving to the Proof of Concept Phase where a larger environment might be needed then more management would be needed. A developer will request kit, use it and then return it to store. The best way to return it would be formatted clean for the next user. The important thing is you book out what you want. You check everything is in the box when you start and you put it back in the same state you took it out.

When Development equipment is required then it needs to be kept in a

In reality the equipment would probably be kept in a dedicated rack. So there would be a hardware booking system and a library of software and pre-prepared VMs. Control of Virtual machines has been considered in earlier projects such as A Private Cloud At the end of the booking period a short report of what had been done plus any created VMs to be considered being added to the library for later tests. There would also be someone responsible for it so that the users of the equipment would report losses and failures.

After a successful Proof of Concept and a project has been given the go ahead. Many projects would then go to a limited pilot stage to iron out any teething troubles on a representative section of users before going 'Live'

More involved projects may then need a full testing environment which is in effect a Model Office.
The change is now to be checked to see how it would work in the real world and to assess user acceptance. Therefore the setup must be as close to the Live environment as possible. Generally the testing will be in a stable state.

Following the Proof of Concept/Model Office the test system needs to be left 'as you found it'. Therefore the equipment needs to be put back into a usable state. Generally this would be the responsibility of the tester who has a procedure to follow to return the kit to its normal state.

A sign off procedure similar to the one for Develoment would be required plus the proviso of leaving the equipment as described.

Proof of Concept and Model Office Testing equipment would also be presented in a rack or racks. The equipment would be booked out as required. The obvious mechanism for this would be on an internal wiki so users of the system could easily book out equipment in advance.

Some of the Requirements for a Model Office

TIME
SAN
AD
Clustered Novell Nodes
Clustered Microsoft Nodes
TSM
ESX
Border Manager (or it's replacement)
Firewall
DMZ
SQL
Oracle
IDM Connectors (Identity Management synchronisation)
Clients
Licensed servers and software

Notes
Where clustering is required as a process to test against for example when you want to test that a piece of software can be setup as a clusterable service then virtual servers using for example iSCSI which can prove the concept would be acceptable

Pre-release

The pre-release area has to mimic the live environment as accurately as possible. As a result the environment needs to be stable and a Change Control environment similar to the live Change Control is required. The equipment would also need to be booked to Developers as per the Proof of Concept above.

Data refreshes from the Live Environment will be required. Of course data in a Pre-release area may need to be anonymised so that actual sensitive data in software such as Carefirst could not be accessed. Again this needs to be considered on a case by case basis.

The firmware and software versions in Live should be the same as those in Pre-release as they are tested first in Pre-release. A system of checking this should be devised so that the two are kept in unison.

Some of the Requirements for a Pre-release Environment

TIME
SAN
AD
Clustered Novell Nodes
Clustered Microsoft Nodes
TSM
ESX
Border Manager (or it's replacement)
Firewall
DMZ
SQL
Oracle
IDM Connectors (Identity Management synchronisation)
Clients
Licensed servers and software

Notes
Where the test is against for example firmware upgrades on SANS and Clustered nodes. Then the test kit will need to be similar to Live with the same cards at the same firmware levels.

Some equipment e.g. hardware appliances (Aventail and foundry switches) may be able to be used across multiple environments, as it will not be needed full time in any one, and the cost of purchasing an appliance for each environment is too costly.

VMWares approach: Software LifeCycle Automation

VMWare have written a whitepaper which explains how Lifecycle management can be implemented using VMWare Workstation, ESX Server, LabManager and VirtualCenter.

flickr:3923314656

R & D work can be carried out on workstations using VMWare Workstation, VirtualCenter enables you to clone live servers and Lab Manager can be used to isolate them in a test environment that can be shared with developers, multiple cloned images can be used, each isolated from the other. The benefit of this approach is that it is fairly quick to clone and set up a replica of the live environment and just as quick to flatten it and start again, it would require dedicated ESX host servers as well as SAN disk space on which the environments will be built.

flickr:3923314822

PROJECT OUTCOME

The project showed how large a field of study creating test environments is and that a lot of preparative work is needed both in obtaining and designing equipment for testing as well as the back office systems to keep it ready for use. The creation of new data centres in the relatively near future must take these extra costs into account.

SHORT TERM BENEFITS

To an extent we are already using and benefitting from this approach, for example the packaging team are using VMWare workstation when packaging scripted installs of applications. The test environment at Saltisford is being used mainly by the Lotus Notes developers on testing new updates before being applied to the live environment and the Intel team are using the environment on updates to the Active Directory infrastructure which is where the groundwork was carried out before implementation to the live environment. However these environments represent pockets of the live environment but can be used as a starting point.

As discussed above purchase of software and hardware now for new projects needs to consider the various environments. Discussions with software vendors as to costs of 'test' licenses for software and servers needs to be considered so the environment is kept within a legal framework

STRATEGIC IMPLICATIONS

Strategically as at least some of the equipment in test needs to be the same as in Live, thought must be given to purchasing test and live at the same time rather than on a rolling basis. Obviously there will be some years where little kit will be purchased and then major amounts in other years. Thus annual sinking funds for equipment purchase at say three or four year intervals must be considered.

Presently many of the latest versions of software will only run on 64 bit architecture e.g. Windows 2008 server release 2 and VMWare Vsphere, this creates the problem where putting older live equipment 'out to grass' as test servers may not always work in the future.

OTHER NOTES ETC

Communications Environment

The Communications Environment at WCC is a constantly changing diverse structure and is difficult to reproduce. Therefore at this stage slow links can be replicated in Test quite easily for example. The full environment would be more difficult. However as so much depends on the Communications Environment it may become necessary to create a better test model in the future.

VMWare Whitepaper on software lifecycle automation:
http://www.vmware.com/pdf/vsla_whitepaper.pdf

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License