Black login screen on Windows Server 2003
This was passed to me by Robert C.
Issue: An application server ran out of disk space on the “C” volume. We could connect to file shares but RDP sessions and the console session showed a black screen so no one could not log in.
Fix: Since the server ran out of disk space the colors for the default user are all reset to black. To correct the issue I took a similar OS and exported the following reg key “HKEY_USERS\.DEFAULT\Control Panel\Colors”. Since I could not login, I connected via network registry, saw all of the color values were set to “0” and imported the export from a valid source taken previously. Rebooted and issue was resolved. Below is an example of that that hive should look like.
[HKEY_USERS\.DEFAULT\Control Panel\Colors]
“ActiveBorder”=”212 208 200″
“ActiveTitle”=”10 36 106″
“AppWorkSpace”=”128 128 128″
“Background”=”58 110 165″
“ButtonAlternateFace”=”180 180 180″
“ButtonDkShadow”=”64 64 64″
“ButtonFace”=”212 208 200″
“ButtonHilight”=”255 255 255″
“ButtonLight”=”212 208 200″
“ButtonShadow”=”128 128 128″
“ButtonText”=”0 0 0″
“GradientActiveTitle”=”166 202 240″
“GradientInactiveTitle”=”192 192 192″
“GrayText”=”128 128 128″
“Hilight”=”10 36 106″
“HilightText”=”255 255 255″
“HotTrackingColor”=”0 0 255″
“InactiveBorder”=”212 208 200″
“InactiveTitle”=”128 128 128″
“InactiveTitleText”=”212 208 200″
“InfoText”=”0 0 0″
“InfoWindow”=”255 255 225″
“Menu”=”212 208 200″
“MenuText”=”0 0 0″
“Scrollbar”=”212 208 200″
“TitleText”=”255 255 255″
“Window”=”255 255 255″
“WindowFrame”=”0 0 0″
“WindowText”=”0 0 0″
“MenuHilight”=”210 210 255″
“MenuBar”=”212 208 200″
Anyone else run into this one?
HP BL460c Boot from EVA 5000 Notes
Booting from SAN isn’t really anything special these days, but it’s still not the easiest thing in the world to do. Of course, having older storage hardware as your boot source can complicate things. So, here’s some notes and guidance developed in my environment to successfully get HP BL460c blades booting from a Generation 1 EVA 5000.
Disclaimer: Not every, exact, specific step you have to perform is listed here. It’s assumed that you’ve fully read the HP Booting from SAN document listed below. Furthermore, it’s assumed that you’re familiar with installing all of the HBA firmware, drivers and other software listed here. If you have existing knowledge of working with BL460c blades, Emulex LPe1105-HP HBAs and EVA 5000 storage systems, then you should be OK. Your mileage may very. Always keep your local HP support team engaged in changes in your environment.
Applicable documents:
- HP StorageWorks Booting Windows Server 2003 and Windows Serverx64 Edition systems from a storage area network application notes
- Rapid Deployment Pack: How To Perform a Boot From SAN Installation (not used in this procedure though, discussed below)
Equipment list:
- HP c7000 blade system chassis
- Brocade 4/24 4GB FC switches for c-Class blade system
- HP BL460c servers
- Emulex LPe1105-HP 4GB dual-port HBAs
- EVA 5000 with HSV110 controllers (Generation one)
- (Optional) Other FC switches upstream of c-Class chassis, commonly the case when you add the c-Class chassis into an existing EVA environment
- Fabric environment that includes two independent zones, A and B or 1 and 2, or something such similar
Software list:
- Windows Server 2003 R2 SP2 64-bit (could use 32-bit as well, but not Windows Server 2008 as it is not supported in Active/Passive controller mode with VCS 3.x code on the EVA 5000)
- Most current versions of firmware, drivers and management software for the LPe1105-HP HBA.
- Start here to obtain these software items
- Emulex HBAnyware management software (v 3.3a14 as of this post)
- HP branded firmware for the LPe1105-HP (v 2.72a2 as of this post)
- HP branded multi-boot firmware for the LPe1105-HP (v 6.00a5 as of this post)
- HP StorPort driver for LPe1105-HP (v 7-2.00a12 as of this post)
- MPIO Basic package from HP (v 1.2 as of this post)
- LP6DUTIL command line HBA maintenance utility (found in v 2.70a5 of firmware)
- Microsoft StorPort hotfix for HP systems (currently MS KB 946448 as far as I know)
So, the basic process goes like this:
- Create and enable a single zone that includes HBA port #1 and a single EVA controller port on Fabric A.
- Boot to a DOS prompt and use the the LP6DUTIL to update the multi-boot code on the LPe1105-HP HBAs. See below for more information on creating a bootable USB key.
- On the BL460c, enter the RBSU (ROM Based Setup Utility) and move HBAs up to #1 and #2 in boot order list. Disable the on board E200i adapter from PC device list.
- Reboot the BL460c and enter Emulex BIOS.
- Enable the BIOS on HBA port #1.
- Set HBA port #1 fabric topology to “Fabric Point to Point” by choosing option 4 from the menu and then selection option 4 again (see Image 1 and 2 below).
- Set HBA port #1 “Start Unit Command” to enabled, which is only needed on Generation 1 EVAs, by choosing Option 8 from the menu.
- Create the server object in Command View.
- Create a boot from SAN LUN in Command View.
- Assign the new LUN to the server in Command View.
- Configure boot options on HBA (1 port with 1 path), follow the directions in the HP document.
- Send the Boot from SAN specific RDP job to server (this is a modified standard job with one change to “Distribute Disk Image” part (see Image 3 below).
- Complete Windows configuration using standard guidelines.
- Install all HBA items in this order:
- HBAnyware
- HBA firmware via HBAnyware
- HP StorPort driver
- Reboot
- Microsoft StorPort hotfix
- Reboot
- MPIO Basic
- Reboot
- HBAnyware (yes, again, it gets uninstalled when you install the HP StorPort driver for whatever reason)
- Change first zone to include both EVA controller ports on Fabric A.
- Add the second HBA port to Boot from SAN LUN in EVA Command View.
- Add the second zone with both EVA controller ports and HBA port #2 in Fabric B.
- Reboot, enter Emulex BIOS again.
- Add second boot path to HBA port #1 (this would be other EVA controller in Fabric A).
- Enable the BIOS on HBA port #2 and add both boot paths to it.
- Make sure that both HBAs are setup in the same order (i.e. Primary to Controller B, Secondary to controller A…see what controller “owns” the BoS volume in Command View).
- Set HBA port #2 fabric topology to “Fabric Point to Point”.
- Set HBA port #2 “Start Unit Command” to enabled (only needed on Gen 1 EVA).
- Reboot the server.
- Complete monitoring application configuration.
- Complete application software installation and configuration.
Image 1: The Configure This Adapter’s Parameters menu
Image 2: Configuring the fabric topology type
Image 3: Changes made to Distribute Disk Image step in RDP installation job
Making this change to the RDP job was a suggestion of an HP support engineer who’s done quite a few Boot from SAN installations using RDP. Furthermore, he also stated there was no reason to perform the actions outlined in RDP KB article 127 if you’re following along step for step with the Boot from SAN document listed above.
You can create a bootable USB key by downloading and using the HP USB Disk Storage Format Tool (v2.1.8), which is pretty hard to find. Download it from a Chinese language HP page here. Just click the button to the right of the file name. You’ll also need the Windows 98 or Me media to make your boot disk.
Image 4: The HP USB Disk Storage Format Tool
Run the utility and be sure to select Create a DOS startup disk and you’ll have a formatted, bootable, USB key in no time. Place your LP6DUTIL file and your multi-boot code file (.PRG) on the USB key and you’re good to go on that.
Booting BL460c blades from an EVA 5000…did this help?
Print server migration script
We’ve got a project upcoming where we’ll be moving all of the Windows print queues from four existing print servers (they’re also file servers) to a a single print server. This is a good thing since it will get the load of having the print queues (and all of the various drivers) off of the file servers onto a dedicated server.
Migrating the 500 or so print queues is not so bad, even if done by hand to ensure that the there is only one version (the most current) of each printer driver installed. But how would one go about updating the mappings on all of the client computers? That’s a big task for any organization, especially when you get into the thousands of computers and thousands of users involved.
The attached script will do just that. It should be run as a startup script and will query the local PC to determine what network print queues it is mapped to and remap them accordingly to the new server…just change the server names and off you go.
RemapPrinters2.zip (728 bytes, 1,248 hits)
“More Available” DHCP
Last year I upgraded our production AD environment from Windows 2000 Server SP4 based DCs to Windows Server 2003 64-bit R2 SP2 based DCs…yeah, a little bit late.
One of the issues with the previous environment was there was no redundancy in the DHCP implementation. Natively, within what Windows provides, your only real choice for highly available DHCP is to have a DHCP cluster. Since most admins, myself included, prefer to keep all of the network infrastructure services on the DCs (DNS, DHCP, WINS), that makes clustering a no-go…you cannot cluster DCs (more specifically, you cannot cluster DCs and still remain in a supportable configuration should you need assistance from PSS).
There are a few really good hardware based products out there for IP Address Management, such the devices from Bluecat Networks. These appliances have fail-over clustering capabilities and provide DNS and DHCP. But, as was the case with our organization, the desire to stay away from adding extra layers of complexity to the core infrastructure won out over an obviously attractive solution for creating highly available DHCP.
Enter the concept of what I like to call “more available”, or just MA for short. Using a combination of freely available utilities and built-in functionality in Windows Server 2003, you can create a MA solution for DHCP at no cost. A couple of assumptions must be made at this point though:
- You have at least two Domain Controllers in the domain.
- You are willing to install and authorize the DHCP service one at least two of the Domain Controllers.
- You are willing to create a service account that will be a member of the Domain Admins group.
- You are willing to have some divergence (i.e. difference) between the active copy of the DHCP database and the standby copy/copies of the DHCP database on the non-active DHCP servers.
The basic process works like this:
- A wrapper .BAT file is called by a scheduled task on the Domain Controller that is providing DHCP. This scheduled task must be run with credentials that have Domain Admin group membership.
- The wrapper .BAT file first runs a second .BAT file that uses the netsh command to export the DHCP database to a text file. Neat!
- The wrapper .BAT file next runs a third .BAT file that uses the great robocopy utility to copy the entire contents of a certain folder from the Domain Controller that is running DHCP to one or more other Domain Controllers that have DHCP installed and authorized, but not running.
- On each of the target Domain Controllers, a fourth .BAT file is run by a scheduled task (using the same account with Domain Admin credentials). This .BAT file makes a backup copy of the DHCP database on that target Domain Controller and then runs a fifth .BAT file to use netsh to import the DHCP database that was exported from the source Domain Controller. (This task should run later than the one on the source Domain Controller, say 5 or 10 minutes later.)
You can get Robocopy by downloading the Windows Server 2003 Resource Kit tools.
In the attached ZIP file are all of the .BAT files you’ll need to make this work.
- DHCP_processing.bat: The first wrapper file that performs the DHCP database export on the source Domain Controller and then copies the files to the destination Domain Controller(s).
- NETSH_export.bat: Called by the first wrapper file, it performs the export of the DHCP database on the source Domain Controller.
- DHCP_copy.bat: Called by the first wrapper file, it uses robocopy to copy the contents of the folder containing all of the script files and the exported DHCP database to the source Domain Controller(s).
- DHCP_import.bat: Another wrapper file, this one manages the state of the DHCP service on the target Domain Controller and creates a copy of the existing DHCP database before calling the last file to import the DHCP database.
- NETSH_import.bat: Called by the wrapper file on the target Domain Controllers, this one imports the DHCP database.
A few notes to keep in mind to make this work:
- All DHCP servers must be authorized in AD.
- You’ll want to edit the file paths in the provided .BAT files to match your environment.
- The DHCP service on the source Domain Controllers should be changed to Manual start up mode (to ensure that the DHCP service is never accidentally started, which would be a bad thing since you’d have multiple DHCP servers issuing addresses and no single point of reference).
- The service account to be used must be a Domain Admin.
- You should put all of the script files and the robocopy executable in a single folder, such “E:\DHCP\IMPORT_EXPORT” in my environment. Let robocopy copy all of files in the folder to the source Domain Controllers, even though they are not needed (this provides protection for your script files if you lose the source Domain Controller).
- Make sure your network group has configured the network infrastructure (routers and/or switches as required) to allow the IP address of any servers you’re setting this up on as “DCHP helpers”…that will be one less thing to do during a failure event.
In our environment, I have the scheduled tasks running twice daily at 10 AM and 7 PM as this was adequate to capture the vast majority of new DHCP lease issues (i.e. they happen on first and second shift before these times). You can certainly run the tasks as often as you want to reduce the differential between the contents of the two databases.
To put one of the standby DHCP servers in operation, simply stop the DHCP running on the original DHCP server (if it’s available) and start the DHCP service on the standby service. More Available DHCP has now been achieved.
More_Available_DHCP.zip (962 bytes, 988 hits)
Granting permissions on Event Logs
I was asked a few days ago by our application developers to grant them read-only access to the Application logs on a couple of Web servers. Some of their Web applications write custom entries to the Application log, and they wanted to be able to connect to the Web servers via Computer Management and see those log entries.
Easy enough request…but I’ll never understand why Microsoft doesn’t allow you to configure Event Log security from the Properties dialog box like you can for literally everything else (file, folders, registry hives and keys, directory services objects, the list goes on…). No, we get stuck using SDDL (Security Descriptor Definition Language)…not exactly the funnest thing in the world.
Fortunately, you can grant permissions on the Event Logs easily enough to either well-known SIDs or even to individual groups or users. First things first…you’ll manage the security settings for the Event Logs via the CustomSD string value in the Registry. Look at the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Eventlog key. Under this key, you’ll see entries for the Application Log, Security Log, etc.
Looking at the CustomSD String key, you see the lovely SDDL entries. Joy!
So, basically, there are three parts:
- Allowing (A) or Denying (D)?
- What level of access?
- Who to?
Allowing or Denying access is pretty straight-forward.
The levels of access are as follows (and are cumulative to reach combinations):
- 1 = Read
- 2 = Write
- 4 = Clear
The last part is the SID that the permissions are being set for. This can either be a well-known SID (in that case a SID string can be used such as IU in the third to last enty in the key above or an actual SID like that seen in the last entry in the key above…or even a REAL user’s or group’s SID. Cool!
So, since I wanted to grant read access to the Application Log only for the developers, I added the following entry to the end of the CustomSD String key for the Application log. Of course, your SID values will be different!
(A;;0×1;;;S-1-5-21-1605523419-404293322-1556899496-26114)
Be sure to get all of the parts of the entry correct. That would be the opening parenthesis, the A or D, the two semi-colons, the Hexadecimal value representation of the cumulative permissions desired, three more semi-colons, the SID and lastly a closing parenthesis.
Need a quick way to get the SID for a security principal in your domain? Try the attached script file. Save it as a .VBS file and place the sAMAccountName of the security principal you’re interested in where the **SAM** placeholder is.
You can get some more information about making these changes in KB323076, and on MSDN.
getSIDv2.zip (419 bytes, 1,169 hits)
The Case of the Suddenly Dead Server
I had recently built and turned over a Windows Server 2003 SP2 server to the applications team to be used a platform and hardware upgrade for the employee time clocking and reporting system. The build went smooth as did the subsequent application installation and configuration. Then, of course, you know what happened next: disaster.
The defining moment, as improbable as it seems, was the stopping of the core service that the application used. The service is stopped and started periodically for various reasons, so stopping it should not have had any negative impact on the system…but it appears to in some way. Soon after the service was stopped, almost every other running service on the server stopped. The server never crashed though, which was odd. Rebooting in Last Known Good didn’t offer any relief either.
When I finally got a chance to examine the Suddenly Dead Server, the first thing I wanted to do was to try and get the RPC service running as it’s a critical core service that many things depend on. Not to be, as it tossed the dreaded Error 5 message back at me.
Could not start the Remote Procedure Call (RPC) service on Local Computer. Error 5: Access is denied.
Not too many things are worse than a generic Error 5 message. So, permissions are the issue…but how do we go from stopping a service (a simple and innocent enough act) to rewriting permissions (in bulk) throughout the Registry and possibly elsewhere. That question still remains to be answered, but there was a way to fix the problem even if I didn’t know what the root cause was. Enter the secedit command. Since it hadn’t even been two days since I finished the build on the server, I felt confident that setting the security configuration back to the installation values would get things moving again.
Using the command secedit /configure /cfg %windir%\repair\secsetup.inf /db secsetup.sdb /verbose from the command line (see KB 313222 for some more background on the secedit command) was the fix I needed to at least get things moving again on the server. After importing the setup security template back into the server, things were looking up. Rather than trying to start all of the stopped services by hand (or using a script to do it), I opted for a reboot. Upon the subsequent log in, nearly everything was normal.
Even though things appeared to be running smoothly again, we still had no idea of what the root cause of the issue was. After gathering all of the relevant logs to be had, the server was rebuilt again. I’ll be going over the logs to try and find a needle in that haystack that leads to a root cause, but I doubt I’ll find it. Since the server never crashed out, there’s no dump to analyze and I didn’t find anything that screamed out to me “I’m the reason the server became Suddenly Dead” when I looked through all the logs.
At any rate, should you find yourself in the bad situation of dealing with the dreaded Error 5, and not being able to get your services running, consider the secedit command. It might just get your Suddenly Dead Server up and running again enough to at least try to collect logs as to what happened and why.
-
Translator
