Data Center Automation

Intro

I was tasked with automating the deployment of the infrastructure at a customer’s site, to run a specific application on top of it.

There were 2 main goals in this project:

  1. Automate the initial provisioning of the infrastructure as much as possible
  2. Automate the recovery of the infrastructure in case of failure

Environment components

  • Hardware
    • HP DL G9 Series Servers
    • HP Z Workstations
    • Cisco Switches
    • NetApp Storage
  • Software
    • Windows Server 2012 R2
    • VMware vSphere 6
    • Windows 7 Professional SP1
    • Other 3rd party software (Office, vCenter, etc.)
  • The environment isn’t connected to any network. No Internet connection.

Initial Provisioning

Well, we start with nothing, and we have to start somewhere, right? So this is what we have –

  • Manually install the Cisco switches & NetApp storage
  • Manually install the 1st HP server with VMware vSphere ESXi 6 software
  • Install a Windows Server 2012 R2 VM
    • Build a domain controller (automagically using powershell DSC)
    • Build the domain database – OUs, Users, Groups (using a CSV file as input source, documented and versioned in source control)
  • Build a “utility” server to perform IT tasks:
    • Deploy other servers (using Microsoft Windows Deployment Services – WDS)
    • Distribute IP addresses (including reservations)
    • Print services
    • Software activation service (KMS)
  • Pre-configure the environment:
    • Build Group Policy Objects (GPOs) with various configurations (hardening, customization, etc.)
    • Create a folder in the NetApp to hold all software that’s needed on the servers
    • Deploy software using batch script connected to GPOs or Software Deployment GPOs (for .msi installations)
    • Build machine database in the utility server. The unique ID is the machine MAC address:
      • WDS pre-staged devices (with proper OU location)
      • DHCP reservations (for server management, a.k.a iLO IP, as well as data IP)

Server provisioning process

Once the above components are in place, the process to deploy a server is as follows:

For all server types

  1. Configure server iLO & BIOS settings using powershell script and HP powershell modules
  2. PXE boot the server and install the base software (either Windows Server 2012 R2 base image or VMware vSphere 6 ESXi)

Windows Specific Configurations

  1. Windows machines (Servers & Workstations) automatically get basic information during deployment, such as:
    1. Time Zone
    2. Domain join
    3. KMS Client key
  2. Further information is applied to the Windows machines post-deployment from the GPOs –
    1. Software needed for operation
    2. Configuration (ports, permissions, restricted groups, etc.)
    3. Customizations (Wallpaper, automatic login, etc.)
  3. Specific servers are installed in a semi-automation fashion – manually running scripts instead of using GPOs (this was done mainly as a precaution step) –
    1. VMware vCenter (using a pre-configured JSON file holding all setup configuration)
    2. Powershell script builds the vCenter after initial installation –
      1. Creates a data center and joins ESXi servers to the datacenter
      2. Deploy VM templates (all with base configuration – CPU, RAM, Disk and no OS installed)
    3. Microsoft SQL Server (including complete DB restoration)

 

ESXi Specific Configurations

ESXi servers are deployed from WDS using kickstart configuration file.

Powershell script is used post-deployment to complete the setup and configuration of the ESXi server:

  1. Time Zone
  2. Domain Join
  3. DNS configuration
  4. Security (SSH, NTP, etc.)

Software Activation

Software activation is a 2-step process

  1. Manually activating the IT utility server using phone activation with Microsoft. The utility server is now a KMS server capable of activating Windows Server OS, Windows 7 OS and Microsoft Office software
  2. Install all machine with client KMS keys and point the clients using DNS configuration to the KMS server for activation

 

At this point the environment is ready to run the application it’s intended to run. All servers and clients are installed with the necessary prerequisites to run code.


 

Server Recovery process

In case a server fails, the following steps are taken to bring back the server to an operational state:

  • If changed, MAC address needs to get updated in the IT utility server database (both WDS and DHCP)
  • Server installation logs needs to be cleared
  • Repeat of the above steps:
    • BIOS configuration via Powershell
    • PXE boot and installation
    • Server-specific post-deployment configuration (ESXi Powershell script / DB script / etc.)
    • Software activation

 

Conclusion

The above process is not a “one-click-process”, but is rather straightforward and automated process. As such, one can deploy such environment in a standard and automated fashion.

In real life, setting up such environment from scratch, from nothing to a fully functional environment, should take around 2-4 days (Assuming you have ~30+ physichal servers and ~50+ more virtual machines)

Further automation is covered in-depth in my blog posts:

In addition, many of the scripts can be found in my github repo


Future Improvements

There is room for improvements, of course. Key areas that should be covered in future iterations are:

  • Using Microsoft Deployment Toolkit (MDT) for the deployment process instead of a combination of Windows Deployment Services, GPOs & Scripts
  • Automate Cisco & NetApp provisioning and recovery
  • Further enhance client automation – BIOS configuration, remote boot, etc.

 

If you’d like to know more about specific steps or areas covered in this project, feel free to reach out and I’ll be happy to discuss

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s