Replacing a Failed Disk(SVM)

Step-by-Step Procedure for replacing a failed disk:

1. Insert the new disk in place of the failed disk. Create the whole disk for
Solaris partition.

2. copy the VTOC of the secondary disk to the primary disk.
Example: prtvtoc /dev/rdsk/c1t1d0s2 | fmthard -s - /dev/rdsk/c1t0d0s2

3. Then re-attach the failed mirrors using the metareplace command.
Example: metareplace -e d0 < new disk >

Once the resynchronisation has completed, the primary disk is successfully
mirrored back online.

NOTE: The above procedure can be followed for replacing a secondary disk
failure.

Solaris VMSTAT

A.   CPU issues:

Following columns has to be watched to determine if there is any cpu issue

1. Processes in the run queue (procs r)
2. User time (cpu us)
3. System time (cpu sy)
4. Idle time (cpu id)

5.     Problem symptoms:

6.    

1.)   If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .

2.)    if  this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.

3.)    If  the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us)  system is facing shortage of CPU resources.

 

Resolution :

         Resolution to these kind of issues involves tuning of application procedures  to make efficient use of cpu  and as a last resort increasing the cpu power or adding more cpu to the system.

B.   Memory Issues:

Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage.

Resolution :

1. Tune the applications & servers to make  efficient use of memory and cache.

2. Increase system memory .

3. Implement priority paging in s in pre solaris 8 versions by adding line "set priority paging=1" in /etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file

iostat

Solaris 
1)IOSTAT:

The values to look from the iostat output  are:

• Reads/writes  per second (r/s , w/s)
• Percentage busy (%b)
• Service time (svc_t)

If a disk shows consistently high reads/writes along with , the percentage busy (%b) of the disks is greater than 5 percent, and the average service time  (svc_t) is greater than 30 milliseconds, then  one of the following action needs to be taken

1.)Tune the application to use disk i/o more efficiently  by modifying the disk queries and using available cache facilities of application servers .

2.) Spread the file system of the disk on to two or more disk  using disk striping feature of volume manager /disksuite  etc.

3.) Increase the system parameter values for  inode cache  , ufs_ninode ,  which is  Number of inodes to be held in memory. Inodes are cached globally (for UFS), not on a per-file system basis 

4.) Move the file system to another faster disk /controller  or replace existing disk/controller to a faster  one.

Performance Monitoring

iostat , vmstat and netstat are three most commonly used tools for performance monitoring . These comes built in with the operating system and are easy to use .iostat stands for input output statistics and reports statistics for i/o devices such as disk drives . vmstat gives the statistics for virtual Memory and netstat gives the network statstics


Iostat

Vmstat

Netstat

Making a bootable ZFS Root Mirror

Making a bootable ZFS Root Mirror

1.     Installed ZFS OS to Disk A (c0t0d0s0)

2.     Format Disk B (c0t1d0s0) properly.

3.     Overwrite the Disk format properly:

Testserver# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard –s - /dev/rdsk/c0t1d0s2

4.     Attach Disk B to the ZFS Root Pool:        

          Testserver#zpool attach –f rpool c0t0d0s0 c0t1d0s0

5.     Install installboot on Disk B

# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0

6.     Test the worst Case: Disk A fails, you can still boot into Disk B

-        Remove Disk A

-        Replace it with an emtpy Disk C

-        Start the System: Choose to boot from Disk B in the Bios

-        Repeat Steps 2. till 5. for Disk C (swap c0t0d0s0 and c0t1d0s0)

-        Start the System like before (from Disk C)