Tuesday, July 24, 2018

How Resize CFS filesystem


1. Find the CVM Master server. 
# vxdctl -c mode 

Find the CFS Primary for the file system you want to resize. 
# fsclustadm -v showprimary </mount_point> 

If the CFS Primary and CVM Master are not the same, change the CFS Primary designation to be the same as the CVM Master. 
# fsclustadm setprimary </mount_point> 

2. Freeze the cluster to protect against monitor timeout issues: 
# hagrp -freeze cvm 

3. Resize the file system and volume using 'vxresize'. 

Verify the resize operation can complete successfully. 
# vxresize -x -F vxfs -g [disk group] -o verify [volume] [ new size] 

If no errors are returned proceed with the resize operation. 
# vxresize -bx -F vxfs -g [disk group] [volume] [ new size] 
* -x is not necessary however it confirms the valued specified will actually increate the volume length. 
-x Requires that the operation represent an increase 
in the volume length. Fail the operation other- 
wise. 

4. At the end unfreeze the cvm servicegroup 
#hagrp -unfreeze cvm 

Sunday, February 12, 2017

CVM and CFS-Basic



Cluster Volume Manager (CVM)

====================


CVM is an extension of Veritas Volume Manager, the industry-standard storage virtualization platform. CVM extends the concepts of VxVM across multiple nodes. Each node recognizes the same logical volume layout, and more importantly, the same state of all volume resources.
CVM supports performance-enhancing capabilities, such as striping, mirroring, and mirror break-off (snapshot) for off-host backup. You can use standard VxVM commands from one node in the cluster to manage all storage. All other nodes immediately recognize any changes in disk group and volume configuration with no user interaction.

CVM architecture
==============
CVM is designed with a "master and slave" architecture. One node in the cluster acts as the configuration master for logical volume management, and all other nodes are slaves. Any node can take over as master if the existing master fails. The CVM master exists on a per-cluster basis and uses GAB and LLT to transport its configuration data.

Just as with VxVM, the Volume Manager configuration daemon, vxconfigd, maintains the configuration of logical volumes. This daemon handles changes to the volumes by updating the operating system at the kernel level. For example, if a mirror of a volume fails, the mirror detaches from the volume and vxconfigd determines the proper course of action, updates the new volume layout, and informs the kernel of a new volume layout. CVM extends this behavior across multiple nodes and propagates volume changes to the master vxconfigd.

Note:
You must perform operator-initiated changes on the master node.
The vxconfigd process on the master pushes these changes out to slave vxconfigd processes, each of which updates the local kernel. The kernel module for CVM is kmsg.

CVM does not impose any write locking between nodes. Each node is free to update any area of the storage. All data integrity is the responsibility of the upper application. From an application perspective, standalone systems access logical volumes in the same way as CVM systems.


CVM imposes a "Uniform Shared Storage" model. All nodes must connect to the same disk sets for a given disk group. Any node unable to detect the entire set of physical disks for a given disk group cannot import the group. If a node loses contact with a specific disk, CVM excludes the node from participating in the use of that disk.

CVM communication
===============
CVM communication involves various GAB ports for different types of communication. For an illustration of these ports:

CVM communication involves the following GAB ports:

Port w

Most CVM communication uses port w for vxconfigd communications. During any change in volume configuration, such as volume creation, plex attachment or detachment, and volume resizing, vxconfigd on the master node uses port w to share this information with slave nodes.

When all slaves use port w to acknowledge the new configuration as the next active configuration, the master updates this record to the disk headers in the VxVM private region for the disk group as the next configuration.

Port v

CVM uses port v for kernel-to-kernel communication. During specific configuration events, certain actions require coordination across all nodes. An example of synchronizing events is a resize operation. CVM must ensure all nodes see the new or old size, but never a mix of size among members.

CVM also uses this port to obtain cluster membership from GAB and determine the status of other CVM members in the cluster.

Port u



CVM uses the group atomic broadcast (GAB) transport mechanism of VCS to ship the commands from the slave node to the master node. CVM uses group atomic broadcast (GAB) port u.


CVM processes one node joining the cluster at a time. If multiple nodes want to join the cluster simultaneously, each node attempts to open port u in exclusive mode. (GAB only allows one node to open a port in exclusive mode). As each node joins the cluster, GAB releases the port. The next node can then open the port and join the cluster. In a case of multiple nodes, each node continuously retries at pseudo-random intervals until it wins the port.


CVM recovery
============
When a node leaves a cluster, the new membership is delivered by GAB, to CVM on existing cluster nodes. The fencing driver (VXFEN) ensures that split-brain scenarios are taken care of before CVM is notified. CVM then initiates recovery of mirrors of shared volumes that might have been in an inconsistent state following the exit of the node.


For database files, when ODM is enabled with SmartSync option, Oracle Resilvering handles recovery of mirrored volumes. For non-database files, this recovery is optimized using Dirty Region Logging (DRL). The DRL is a map stored in a special purpose VxVM sub-disk and attached as an additional plex to the mirrored volume. When a DRL subdisk is created for a shared volume, the length of the sub-disk is automatically evaluated so as to cater to the number of cluster nodes. If the shared volume has Fast Mirror Resync (FlashSnap) enabled, the DCO (Data Change Object) log volume created automatically has DRL embedded in it. In the absence of DRL or DCO, CVM does a full mirror resynchronization.


Configuration differences with VxVM
=======================
CVM configuration differs from VxVM configuration in the following areas:


  1. Configuration commands occur on the master node.
  2. Disk groups are created (could be private) and imported as shared disk groups.
  3. Disk groups are activated per node.
  4. Shared disk groups are automatically imported when CVM starts.

Cluster File System (CFS)
==================



CFS enables you to simultaneously mount the same file system on multiple nodes and is an extension of the industry-standard Veritas File System. Unlike other file systems which send data through another node to the storage, CFS is a true SAN file system. All data traffic takes place over the storage area network (SAN), and only the metadata traverses the cluster interconnect.



In addition to using the SAN fabric for reading and writing data, CFS offers storage checkpoints and rollback for backup and recovery.




Access to cluster storage in typical SF Oracle RAC configurations use CFS. Raw access to CVM volumes is also possible but not part of a common configuration.

CFS architecture
===========

SF Oracle RAC uses CFS to manage a file system in a large database environment. Since CFS is an extension of VxFS, it operates in a similar fashion and caches metadata and data in memory (typically called buffer cache or vnode cache). CFS uses a distributed locking mechanism called Global Lock Manager (GLM) to ensure all nodes have a consistent view of the file system. GLM provides metadata and cache coherency across multiple nodes by coordinating access to file system metadata, such as inodes and free lists. The role of GLM is set on a per-file system basis to enable load balancing.


CFS involves a primary/secondary architecture. One of the nodes in the cluster is the primary node for a file system. Though any node can initiate an operation to create, delete, or resize data, the GLM master node carries out the actual operation. After creating a file, the GLM master node grants locks for data coherency across nodes. For example, if a node tries to modify a block in a file, it must obtain an exclusive lock to ensure other nodes that may have the same file cached have this cached copy invalidated.

SF Oracle RAC configurations minimize the use of GLM locking. Oracle RAC accesses the file system through the ODM interface and handles its own locking; only Oracle (and not GLM) buffers data and coordinates write operations to files. A single point of locking and buffering ensures maximum performance. GLM locking is only involved when metadata for a file changes, such as during create and resize operations.

CFS communication
=============
CFS uses port f for GLM lock and metadata communication. SF Oracle RAC configurations minimize the use of GLM locking except when metadata for a file changes.

CFS file system benefits
===============

Many features available in VxFS do not come into play in an SF Oracle RAC environment because ODM handles such features. CFS adds such features as high availability, consistency and scalability, and centralized management to VxFS. Using CFS in an SF Oracle RAC environment provides the following benefits:
Increased manageability, including easy creation and expansion of files

In the absence of CFS, you must provide Oracle with fixed-size partitions. With CFS, you can grow file systems dynamically to meet future requirements.

Less prone to user error

Raw partitions are not visible and administrators can compromise them by mistakenly putting file systems over the partitions. Nothing exists in Oracle to prevent you from making such a mistake.

Data center consistency

If you have raw partitions, you are limited to a RAC-specific backup strategy. CFS enables you to implement your backup strategy across the data center.

CFS recovery
=========
The vxfsckd daemon is responsible for ensuring file system consistency when a node crashes that was a primary node for a shared file system. If the local node is a secondary node for a given file system and a reconfiguration occurs in which this node becomes the primary node, the kernel requests vxfsckd on the new primary node to initiate a replay of the intent log of the underlying volume. The vxfsckd daemon forks a special call to fsck that ignores the volume reservation protection normally respected by fsck and other VxFS utilities. The vxfsckd can check several volumes at once if the node takes on the primary role for multiple file systems.

After a secondary node crash, no action is required to recover file system integrity. As with any crash on a file system, internal consistency of application data for applications running at the time of the crash is the responsibility of the applications.

Comparing raw volumes and CFS for data files
=============================
Keep these points in mind about raw volumes and CFS for data files:
If you use file-system-based data files, the file systems containing these files must be located on shared disks. Create the same file system mount point on each node.

If you use raw devices, such as VxVM volumes, set the permissions for the volumes to be owned permanently by the database account.

VxVM sets volume permissions on import. The VxVM volume, and any file system that is created in it, must be owned by the Oracle database user.

Friday, February 3, 2017

Symantec Cluster Server VCS Moving Resource from One Service Group to Another





Unfortunately our Veritas Clusters are rather new, and when setting them up we didn't have a clear plan as to how things where to be laid out. This means that we have far to many Resource Groups making the linking of the resources difficult. Some details discovered did cover some of the things, but not everything, such as the clean up and removal of those redundant RGs. So I decided to include that extra information here, for my reference later. First a note; if you have access to the GUI. This job is 100% easier, as it's just a move and copy affair, in this case we don't have this access due to resources, change requirements and firewall permissions.
Using the tools that VCS gives you, you need to take snap shot of the completed cluster:
# cd /etc/VRTSvcs/conf/config; hacf -verify .
This will create a main.cmd file in the local directory, from this collect the info we need about the RG, so list them:
# hagrp -state
#Group         Attribute             System     Value
cfs_backup     State                 server1    |ONLINE|
cfs_other      State                 server1    |ONLINE|
cvm            State                 server1    |ONLINE|
sg-ip          State                 server1    |OFFLINE|
sg-weblogic    State                 server1    |OFFLINE|
vxfen          State                 server1    |ONLINE|
> hagrp -resources cfs_backup
backup
In the example above we need to move cfs_backup and sg-ip into the sg_weblogic group, and we can see the resource 'backup' belongs to the group 'cfs_backup'. The next command shows us the out put we are going to be using:
# grep "^hares.* backup*" main.cmd
hares -add backup CFSMount cfs_backup
hares -modify backup Critical 0
hares -modify backup MountPoint "/backup"
hares -modify backup BlockDevice "/dev/vx/dsk/backupvg/backupvol"
hares -local backup MountOpt
hares -modify backup MountOpt "cluster" -sys server1
hares -modify backup NodeList  server1
hares -modify backup CloneSkip no
hares -modify backup Policy -delete -keys
hares -modify backup Enabled 1
hares -modify backup_vol CVMDiskGroup backupvg
hares -modify vxfsckd ActivationMode  backupvg sw othervg sw -sys server1
hares -link backup backup_vol
# grep "^hares.* backup*" main.cmd > backup.cmd
Now that we have the captured info, we need to create a delete script
# awk '/^hares -add backup/ {print "hares -delete "$3}' backup.cmd > backup_del.cmd
Once created, edit the backup.cmd and we replace the RED cfs_backup with the new group name 'sg-weblogic'.
# cat backup_del.cmd
hares -delete backup
Update the permissions of the new scripts:
# chmod a+x backup*.cmd
Make the Cluster ready for the edit:
# haconf -makerw
Then run the scripts:
# ./backup_del.cmd
# ./backup.cmd
In our case, we also did the same for the IP group, using the same method above. Once complete you will need to set the cluster back to read-only:
# haconf -dump -makero
Then we stop and start the cluster to make sure it looks all okay?
# hagrp -state
#Group         Attribute             System     Value
cfs_backup     State                 server1    |OFFLINE|
cfs_other      State                 server1    |ONLINE|
cvm            State                 server1    |ONLINE|
sg-ip          State                 server1    |OFFLINE|
sg-weblogic    State                 server1    |ONLINE|
vxfen          State                 server1    |ONLINE|
Once it looks okay, we need to clean up any left parts of these now old defunct RGs, so list there information and dependencies:
# hagrp -resources cfs_backup
<empty>
# hagrp -dep cfs_backup
#Parent      Child      Relationship
cfs_backup   cvm        online local firm
# hagrp -unlink cfs_backup cvm
Finally delete the old RG
# hagrp -delete cfs_backup

Remember to repeat this for any other RGs we cleaned up, in this case the sg-ip group

Symantec Cluster Server. VCS Moving a Mount Point




Today after I getting all the filesystem mount points configured and set up in a cluster I was was asked change them. Annoying but not something I was initially sure how to do. Little bit of searching and clarification of the VCS commands this is what I was able to figure out.

First lets take a look at the mount we are about to move:

# hares -display cfsmount1 | grep -i MountPoint
cfsmount1    ArgListValues         server1   MountPoint        1       /mount/path  BlockDevice     1       /dev/vx/dsk/mountdg/mount_vol MountOpt        1       cluster CloneSkip       1       no      Primary 1       server2        AMFMountType    1       vxfs
cfsmount1    ArgListValues         server2   MountPoint        1       /mount/path  BlockDevice     1       /dev/vx/dsk/mountdg/mount_vol MountOpt        1       cluster CloneSkip       1       no      Primary 1       server2        AMFMountType    1       vxfs

The plan is that we just move the /mount/path to our new path of /mount. So first we need to un-mount our mounts from the nodes in the cluster:

# cfsumount /mount/path
  Unmounting...
  /mount/path got successfully unmounted from server1
  /mount/path got successfully unmounted from server2

Once this is done we need to update and clean up the filesystems and old/new paths:
# ls -dl /mount/path
drwxr-xr-x    2 oracle   dba             256 06 Jun 16:26 /mount/path
# ls -dl /mount
drwxrwxr-x    4 root   dba             256 08 Jun 14:41 /mount

In this example the path needs to be owned by the Oracle user:
# chown -R oracle:dba /mount

Then we are going to check that the old mount point is empty and delete it.
NOTE* Make sure you've unmounted the FS here, else you could destroy all your data. Ours is empty, a good sign that nothing is sitting under the mount point or that nothing is mounted.
NOTE** These filesystem updates need to be actioned on ALL the nodes in the cluster!
# ls -l /mount/path
total 0
# rmdir /mount/path
# ls -dl /mount
drwxrwxr-x    3 oracle   dba             256 14 Jun 10:41 /mount

Our new mount point looks good, so we can now make the required change:
# hares -modify cfsmount1 MountPoint "/mount"
# hares -display cfsmount1 | grep -i MountPoint
cfsmount1    ArgListValues         server1   MountPoint        1       /mount BlockDevice     1       /dev/vx/dsk/mountdg/mount_vol MountOpt        1       cluster CloneSkip       1       no      Primary 1       server2        AMFMountType    1       vxfs
cfsmount1    ArgListValues         server2   MountPoint        1       /mount BlockDevice     1       /dev/vx/dsk/mountdg/mount_vol MountOpt        1       cluster CloneSkip       1       no      Primary 1       server2        AMFMountType    1       vxfs
cfsmount1    MountPoint            global     /mount

The information above looks perfect, so go ahead and remount it all.
# cfsmount /mount
  Mounting...
  [/dev/vx/dsk/mountdg/mount_vol] mounted successfully at /mount on server1
  [/dev/vx/dsk/mountdg/mount_vol] mounted successfully at /mount on server2
# df -g /mount
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/vx/dsk/mountdg/mount_vol     99.00     92.66    7%      339     1% /mount

How to expand Solaris Volume Manager filesystem which is exported to zones from Global Zone

How to expand Solaris Volume Manager filesystem which is exported to zones from Global Zone
Ok, it's been long that no updates on blog... Anyways, Today I'm having some good information on - "How to expand Solaris Volume Manager (Metadevice) filesystem which is exported to zones from Global Zone"

I've a uniqe system which is having little diffrent configuration than other systems. I've SPARC-Enterprise M4000 system and having 2 zones running on it. Here is the zone configuration example for one of them.

# zonecfg -z zone1 info
zonename: zone1
zonepath: /zone1/zonepath
brand: native
autoboot: true
bootargs:
pool: oracpu_pool
limitpriv: default,dtrace_proc,dtrace_user
scheduling-class:
ip-type: shared
[cpu-shares: 32]
fs:
dir: /oracle
special: /dev/md/dsk/d56
raw: /dev/md/rdsk/d56
type: ufs
options: []
fs:
dir: /oradata1
special: /dev/md/dsk/d59
raw: /dev/md/rdsk/d59
type: ufs
options: []
fs:
dir: /oradata2
special: /dev/md/dsk/d62
raw: /dev/md/rdsk/d62
type: ufs
options: []
fs:
dir: /oradata3
special: /dev/md/dsk/d63
raw: /dev/md/rdsk/d63
type: ufs
options: []
[...]

Ok, So here you can see that I've metadevices which are exported to the zone from the global zone. I need to expand one of the filesystem say /oaradata1 by XXG so how am I going to perform this? Take a look at below procedure to understand on how we can do it.

global:/
# zonecfg -z zone1 info fs dir=/oradata1
fs:
dir: /oradata1
special: /dev/md/dsk/d59
raw: /dev/md/rdsk/d59
type: ufs
options: []
global:/
# metattach d59 Storage_LUN_ID

global:/
# growfs -M /zone1/zonepath/root/oradata1 /dev/md/rdsk/d59

This all operation needs to be performed from global zone.

How to Increase Cluster File System - CVM Cluster



In this post I would like to discuss and demonstrate about increasing the Cluster File System (CFS) in CVM environment.

    - The requirement is to increase the filesystem by 1TB.

As a best practice, in a CVM/CFS environment, volume should be grown on CVM master and file system should be grown on a CFS Primary. Please note that the CVM Master and the SF CFS node can be two different nodes.

Just off the topic, to know how to grow volume on CVM master & then grow the filesystem on CFS Primary -

To increase the size of the file system, execute the following on CVM master -

# vxassist –g shared_disk_group growto volume_name newlength

And then on CFS node, execute-

# fsadm –F vxfs –b newsize –r device_name mount_point

On other hand, if the system is both CVM master and CFS primary, then "vxresize" command could be executed on the system without any issues.

The above mentioned statement is true below VERITAS version 3.5 but from and above VERITAS version 3.5 vxresize can be run on any nodes within the cluster provided that attribute named 'HacliUserLevel' is set to value "COMMANDROOT". The default value of this attribute is 'NONE' which prevents user to run vxresize command from all the nodes in the cluster.

In my case it is set to "COMMANDROOT"

# /opt/VRTSvcs/bin/haclus -display | grep Hacli
HacliUserLevel         COMMANDROOT

But if the value of attribute 'HacliUserLevel' is set to "NONE" then below is method can be used to change the value.

To change the value to COMMANDROOT, run:

# /opt/VRTSvcs/bin/haconf -makerw
# /opt/VRTSvcs/bin/haclus -modify HacliUserLevel COMMANDROOT
# /opt/VRTSvcs/bin/haconf -dump -makero

This change allows vxresize to call hacli command, which then allows any command to be run on any system within the cluster.

Okay, back to our requirement -

So first let's find out the CFS primary node -

# fsclustadm -v showprimary /u01-zz/oradata
XXXXX

Now let's find out the master CVM node in the cluster by -

# vxdctl -c mode
mode: enabled: cluster active - SLAVE
master: XXXXX

Now that I need to increase the filesystem by 1TB hence I'm checking if disk group has enough space in it.

# vxassist -g dde1ZZGO0 maxsize
Maximum volume size: 2136743936 (1043332Mb)

Well, we have enough space available under disk group dde1ZZGO0.

Let increase the FS by 1TB now -

# vxresize -b -F vxfs -g dde1ZZGO0 ZZGO0_v0 +1000g

BEFORE:

# df -kh /u01-zz/oradata
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/dde1ZZGO0/ZZGO0_v0
                       5.0T   5.0T    17G   100%    /u01-zz/oradata

AFTER:

# df -kh /u01-zz/oradata
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/dde1ZZGO0/ZZGO0_v0

                       6.0T   5.0T   954G    85%    /u01-zz/oradata

Changing the CVM master manually




You can change the CVM master manually from one node in the cluster to another node, while the cluster is online. CVM migrates the master node, and reconfigures the cluster.

Symantec recommends that you switch the master when the cluster is not handling VxVM configuration changes or cluster reconfiguration operations. In most cases, CVM aborts the operation to change the master, if CVM detects that any configuration changes are occurring in the VxVM or the cluster. After the master change operation starts reconfiguring the cluster, other commands that require configuration changes will fail.

See Errors during CVM master switching.

To change the master online, the cluster must be cluster protocol version 100 or greater.

To change the CVM master manually

To view the current master, use one of the following commands:

# vxclustadm nidmap
Name              CVM Nid    CM Nid    State
system01            0        0         Joined: Slave
system02            1          1         Joined: Master
# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: system02
In this example, the CVM master is system02.

From any node on the cluster, run the following command to change the CVM master:

# vxclustadm setmaster nodename
where nodename specifies the name of the new CVM master.

The following example shows changing the master on a cluster from system02 to system01:

# vxclustadm setmaster system01
To monitor the master switching, use the following command:

# vxclustadm -v nodestate
 state: cluster member
        nodeId=0
        masterId=0
        neighborId=1
        members[0]=0xf
        joiners[0]=0x0
        leavers[0]=0x0
        members[1]=0x0
        joiners[1]=0x0
        leavers[1]=0x0
        reconfig_seqnum=0x9f9767
        vxfen=off
state: master switching in progress
reconfig: vxconfigd in join
In this example, the state indicates that master is being changed.

To verify whether the master has successfully changed, use one of the following commands:

# vxclustadm nidmap
Name              CVM Nid    CM Nid    State
system01            0        0         Joined: Master
system02            1         1         Joined: Slave
# vxdctl -c mode
mode: enabled: cluster active - MASTER

master: system01

Migrating VCS clusters on Solaris 10 systems- Solaris 11 branded zone

Configuring VCS/SF in a branded zone environment

You must perform the following steps on the Solaris 11 systems.

To configure VCS/SF in a branded zone environment
1.       Install VCS, SF, or SFHA as required in the global zone.
See the Symantec Cluster Server Installation Guide.
See the Symantec Storage Foundation and High Availability Installation Guide.
2.       Configure a solaris10 branded zone. For example, this step configures a solaris10 zone.
•        Run the following command in the global zone as the global administrator:
•        # zonecfg -z sol10-zone
•        sol10-zone: No such zone configured
Use 'create' to begin configuring a new zone.
•        Create the solaris10 branded zone using the SYSsolaris10 template.
zonecfg:sol10-zone> create -t SYSsolaris10
•        Set the zone path. For example:
zonecfg:sol10-zone> set zonepath=/zones/sol10-zone
Note that zone root for the branded zone can either be on the local storage or the shared storage VxFS, UFS, or ZFS.
•        Add a virtual network interface.
•        zonecfg:sol10-zone> add net
•        zonecfg:sol10-zone:net> set physical=net1
•        zonecfg:sol10-zone:net> set address=192.168.1.20
zonecfg:sol10-zone:net> end
•        Verify the zone configuration for the zone and exit the zonecfg command prompt.
•        zonecfg:sol10-zone> verify
zonecfg:sol10-zone> exit
The zone configuration is committed.
3.       Verify the zone information for the solaris10 zone you configured.
# zonecfg -z sol10-zone info
Review the output to make sure the configuration is correct.
4.       Install the solaris10 zone that you created using the flash archive you created previously.
See Preparing to migrate a VCS cluster.
# zoneadm -z sol10-zone install -p -a /tmp/sol10image.flar
After the zone installation is complete, run the following command to list the installed zones and to verify the status of the zones.
# zoneadm list -iv
5.       Boot the solaris10 branded zone.
6.       # /usr/lib/brand/solaris10/p2v sol10-zone 
# zoneadm -z sol10-zone boot
After the zone booting is complete, run the following command to verify the status of the zones.
# zoneadm list -v
7.       Configure the zone with following command:
# zlogin -C sol10-zone
8.       Install VCS in the branded zone:
•        Install only the following VCS 6.1 packages:
•        VRTSperl
•        VRTSvlic
•        VRTSvcs
•        VRTSvcsag
9.       If you configured Oracle to run in the branded zone, then install the VCS agent for Oracle packages (VRTSvcsea) and the patch in the branded zone.
See the Symantec Cluster Server Agent for Oracle Installation and Configuration Guide for installation instructions.
10.     For ODM support, install the following additional packages and patches in the branded zone:
•        Install the following 6.1 packages:
•        VRTSvlic
•        VRTSvxfs
•        VRTSodm
11.     If using ODM support, relink Oracle ODM library in solaris10 branded zones:
•        Log into Oracle instance.
•        Relink Oracle ODM library.
If you are running Oracle 10gR2:
$ rm $ORACLE_HOME/lib/libodm10.so
$ ln -s /opt/VRTSodm/lib/sparcv9/libodm.so \
$ORACLE_HOME/lib/libodm10.so
If you are running Oracle 11gR1:
$ rm $ORACLE_HOME/lib/libodm11.so
$ ln -s /opt/VRTSodm/lib/sparcv9/libodm.so \ 
$ORACLE_HOME/lib/libodm11.so
•        To enable odm inside branded zone, enable odm in global zone using smf scripts as described as below:
•        global# svcadm enable vxfsldlic
global# svcadm enable vxodm
To use ODM inside branded zone, export /dev/odm, /dev/fdd, /dev/vxportal devices and /etc/vx/licenses/lic directory.
global# zoneadm -z myzone halt 
global# zonecfg -z myzone 
zonecfg:myzone> add device 
zonecfg:myzone:device> set match=/dev/vxportal 
zonecfg:myzone:device> end 
zonecfg:myzone> add device 
zonecfg:myzone:device> set match=/dev/fdd 
zonecfg:myzone:device> end 
zonecfg:myzone> add device 
zonecfg:myzone:device> set match=/dev/odm 
zonecfg:myzone:device> end 
zonecfg:myzone> add device 
zonecfg:myzone:device> set match=/dev/vx/rdsk/dg_name/vol_name 
zonecfg:myzone:device> end 
zonecfg:myzone> add device 
zonecfg:myzone:device> set match=/dev/vx/dsk/dg_name/vol_name 
zonecfg:myzone:device> end 
zonecfg:myzone> add fs 
zonecfg:myzone:fs> set dir=/etc/vx/licenses/lic 
zonecfg:myzone:fs> set special=/etc/vx/licenses/lic 
zonecfg:myzone:fs> set type=lofs 
zonecfg:myzone:fs> end 
zonecfg:myzone> set fs-allowed=vxfs,odm 
zonecfg:myzone> verify 
zonecfg:myzone> commit 
zonecfg:myzone> exit 
global# zoneadm -z myzone boot
          Configure the resources in the VCS configuration file in the global zone. The following example shows the VCS configuration when VxVM volumes are exported to zones via zone configuration file:
          group zone-grp (
                   SystemList = { vcs_sol1 = 0, vcs_sol2 = 1 }
                   ContainterInfo@vcs_sol1 {Name = sol10-zone, Type = Zone,Enabled = 1 }
                   ContainterInfo@vcs_sol2 {Name = sol10-zone, Type = Zone, Enabled = 1 }
                   AutoStartList = { vcs_sol1 }
                   Administrators = { "z_z1@vcs_lzs@vcs_sol2.symantecexample.com" }
          )
         
                    DiskGroup zone-oracle-dg (
                          DiskGroup = zone_ora_dg
                          )
                 Volume zone-oracle-vol (
                          Volume = zone_ora_vol
                          DiskGroup = zone_ora_dg
                          )
         
                  Netlsnr zone-listener (
                          Owner = oracle
                          Home = "/u01/oraHome"
                          )
         
                  Oracle zone-oracle (
                          Owner = oracle
                          Home = "/u01/oraHome"
                          Sid = test1
                             )
                  Zone zone-res (
                          )
                 zone-res requires zone-oracle-vol
                zone-oracle-vol requires zone-oracle-dg
      zone-oracle requires zone-res


Migrating VCS clusters on Solaris 10 systems


You can migrate VCS clusters that run on Solaris 10 systems to solaris10 branded zones on Solaris 11 systems. For example, with branded zones you can emulate a Solaris 10 operating system as Solaris 10 container in the Solaris 11 branded zone. This Solaris 10 non-global zone acts as a complete runtime environment for Solaris 10 applications on Solaris 11 systems. You can directly migrate an existing Solaris 10 system into a Solaris 10 container.
Figure: Workflow to migrate VCS cluster to branded zones illustrates the workflow to migrate a VCS cluster on Solaris 10 systems to branded zones on Solaris 11 systems.




Assign the storage used for the VxVM volumes and VxFS file systems used by application inside the Solaris 10 physical system to the new system. Mount the VxFS file system using loop back mount or direct mount.

Thursday, December 15, 2016

Veritas Cluster Server service in Solaris 10 reports "maintenance"

Problem

Attempting to enable the svc:/system/vcs:default service using svcadm enable vcs, the service transitions to maintenance.

Error Message

svc:/system/vcs:default (Veritas Cluster Server (VCS) Init service)
 State: maintenance since Thu Oct 14 12:15:57 2010
Reason: Start method failed repeatedly, last exited with status 2.
   See: http://sun.com/msg/SMF-8000-KS
   See: man -M /opt/VRTS/man/man1m/ -s 1M vcsconfig
   See: /var/svc/log/system-vcs:default.log
Impact: This service is not running.

Cause

This situation can occur if the '/etc/default/vcs' file is present, but the VCS_START variable has been set to '0' (i.e., do not start or stop VCS via svc:/system/vcs:default) explicitly in /etc/default/vcs.

Solution


Set VCS_START variable to '1' in /etc/default/vcs

LLT service in SMF is showing in maintenance state






After installing VCS on new node and trying to add it to existing cluster llt service may not start through SMF on new node. It will show LLT service is in maintenance status. Due to this rest of cluster services will not start during system boot
However, a manual start on LLT and GAB working fine.
Error Message

LLT service in SMF is showing in maintenance status.
# svcs -xv

svc:/system/llt:default (Veritas Low Latency Transport (LLT) Init service)

 State: maintenance since Thu Jul 29 13:40:32 2010

Reason: Start method failed repeatedly, last exited with status 2.

   See: http://sun.com/msg/SMF-8000-KS

   See: man -M /opt/VRTSllt/man/man1m/ -s 1M lltconfig

   See: /var/svc/log/system-llt:default.log

Impact: 3 dependent services are not running:

        svc:/system/gab:default

        svc:/system/vcs:default

        svc:/system/vxfen:default



LLT is in maintenance state even its dependencies are in Online states.
#svcs -l llt

fmri         svc:/system/llt:default

name         Veritas Low Latency Transport (LLT) Init service

enabled      true

state        maintenance

next_state   none

state_time   Fri Jul 30 02:09:03 2010

logfile      /var/svc/log/system-llt:default.log

restarter    svc:/system/svc/restarter:default

dependency   require_all/none svc:/system/filesystem/local (online)

dependency   optional_all/none svc:/network/initial (online)


Log file /var/svc/log/system-llt:default.log  is showing below errors.

[ Jul 30 02:13:17 Executing start method ("/lib/svc/method/llt start") ]
silent failure
[ Jul 30 02:13:17 Method "start" exited with status 2 ]
Cause

SMF uses /lib/svc/method/llt script to start LLT at boot time and same to stop at shutdown and it needs environment set in "/etc/default/llt"

# cat /etc/default/llt
#
# This file is sourced :
#       from /etc/init.d/llt            for Solaris < 2.10
#       from /lib/svc/method/llt        for Solaris 2.10
#
# Set the two environment variables below as follows:
#
#       1 = start or stop llt
#       0 = do not start or stop llt
#

LLT_START=0
LLT_STOP=0


If LLT_START is set equal to zero, SMF service does not start LLT service at boot time.

Solution

Modified /etc/default/llt and set LLT_START and LLT_STOP to ONE

# cat /etc/default/llt
#
# This file is sourced :
#       from /etc/init.d/llt            for Solaris < 2.10
#       from /lib/svc/method/llt        for Solaris 2.10
#
# Set the two environment variables below as follows:
#
#       1 = start or stop llt
#       0 = do not start or stop llt
#

LLT_START=1
LLT_STOP=1

Wednesday, November 16, 2016

How to Increasing the size of the root file system (slice) on an encapsulated root disk-VXvm


This technical document describes a procedure to increase the root file system (slice), where the root disk is encapsulated under VERITAS Volume Manager (tm). It is acknowledged that there may be other ways to achieve this objective. It is also possible that the procedure described may not fully apply to every system running Volume Manager, but is likely that the procedure may be adopted with some changes to suit most of the common Volume Manager configurations and installations.


Assumptions:

The boot (root) disk is encapsulated.
The root diskgroup (rootdg) may consist of more than one disk.
One free disk with at least as much capacity as the current root disk.


Preliminary details:

If the free disk required for this procedure is only as big as the current root disk, then there must be enough free space on the root disk to allow for the increased root file system size. If this free space is not available on the disk, then the proposed increase to the root file system size must come from a corresponding reduction in the size, or omission of some other file system (or partition) on the root disk.
The free disk described in this procedure is used to prepare a new root disk with the necessary file system (slice) sizes. The original root disk can be preserved at least until the system boots from the new root disk successfully.
The general procedure to accomplish this task is as follows:


  1. Initialize new root disk and add to rootdg diskgroup
  2. Mirror root volume onto new disk from current root disk
  3. Increase size of root volume and file system
  4. Mirror other volumes from current root disk
  5. Break mirrors on new root disk
  6. Create underlying physical partitions on disk
  7. Create swap partition on disk
  8. Reboot system off slices from new root disk
  9. Remove volumes from old root disk
  10. Encapsulate new root disk and add to rootdg disk group
  11. Reboot off new root disk using volumes
  12. Mirror new root disk to another equivalent disk


If the new root disk is the same size as the original root disk, then upon successful encapsulation of the new root disk, it is possible to mirror back to the original root disk, thereby having two identical mirrors. If the new root disk is of larger capacity, then another disk of equivalent size will be necessary for the root mirror.
In the detailed procedure listed below, the rootdg initially consists of the boot disk (rootdisk=c2t8d0) and one additional disk (root0=c2t3d0) containing Volume Manager volumes. The procedure would be unchanged even if the boot disk has a mirror. The new disk that will eventually become the boot disk is c2t2d0.


Detailed procedure

1. Initialize new disk:
Is new disk same size as current boot disk?
Yes
Is Volume Manager version 3.2?
Yes
Check private region length for boot disk using prtvtoc command. Here, c2t8d0 is the root disk.
# prtvtoc /dev/rdsk/c2t8d0


Slice 7 (tag 15) is the vtoc entry representing the private region. Note the sector count of 2744. This will be the public region offset for the new disk.
Initialize the new disk (c2t2d0) using the vxdisksetup command.
# /etc/vx/bin/vxdisksetup -i c2t2d0 puboffset=2744
Go to step 2.
No (Volume Manager version < 3.2)
Initialize the new disk (c2t2d0) using the vxdisksetup command.
# /etc/vx/bin/vxdisksetup -i c2t2d0
Go to step 2.
No (new disk of higher capacity)
Initialize the new disk (c2t2d0) using the vxdisksetup command.
# /etc/vx/bin/vxdisksetup -i c2t2d0

2. Add new disk to rootdg. In this example, the disk is named newroot (c2t2d0).
# vxdg -g rootdg adddisk newroot=c2t2d0

The rootdg configuration now looks as follows.
# vxdisk list


# vxprint -Qqhtg rootdg


3. Mirror the root slice to disk newroot.
# /etc/vx/bin/vxrootmir newroot

Check rootvol details.
# vxprint -Qqhtr rootvol


4. A series of steps to increase size of the root slice on the new disk.
Disassociate the newly created plex rootvol-02 from rootvol volume.
# vxplex dis rootvol-02

Make a volume named rootalt with usage type "gen" and associate plex rootvol-02.
# vxmake -U gen vol rootalt plex=rootvol-02

Make the volume active.
# vxvol init active rootalt

Volume rootalt, as it appears now.
# vxprint -Qqhtr rootalt


Run a file system check on the "ufs" file system contained within volume rootalt. Fix the file system state in the super block and any other errors encountered. This will clear the file system super-block flag and enable us to mount the file system.
# fsck -F ufs /dev/vx/rdsk/rootdg/rootalt

Mount the rootalt volume on /mnt (or /a)
# mount -F ufs /dev/vx/dsk/rootdg/rootalt /mnt

Estimate the new size for the root slice (rootalt). The new size must be such that the sub-disk ends on a disk cylinder boundary. In this example, the rootvol (and the newly created mirror volume, rootalt) is 4195576 disk sectors long (1 disk sector = 512 bytes). This translates to a size that is marginally over 2 GB. This size will be increased by 500 MB to a new size that is roughly 2.5 GB.
# prtvtoc /dev/rdsk/c2t2d0s2


From the above, it is clear that the root partition size of 4195576 sectors corresponds to 3058 disk cylinders (4195576 sectors / 1372 sectors per cylinder = 3058 cylinders).
The proposed increase is 500 MB. This translates to approximately 746 disk cylinders. The calculation for this being as follows:
500 MB = 500 * 1024 KB = 512000 KB = 512000 * 2 disk sectors = 1024000 disk sectors
1024000 sectors / 1372 sectors per cylinder = 746.35 cylinders.
In this discussion, the figure has been rounded down to 745 cylinders. This means that the new size for the root volume will be 3803 cylinders (3058 + 745). This translates to a size of 5217716 sectors (3803 cylinders * 1372 sectors per cylinder).

Increase size of volume rootalt and the file system to the new size.
# vxassist growto rootalt 5217716

Increase the size of the "ufs" file system to the new size.
# /usr/sbin/growfs -M /mnt /dev/vx/rdsk/rootdg/rootalt

Verify the size of the rootalt volume
# vxprint -Qqhtr rootalt


The new root file system (mounted on /mnt) shows increased file system size.
# df -k


5. Re-write the disk slice information that represents the sub-disk (newroot-01) for volume rootalt.
The current entry (see the prtvtoc command output at step 4, slice 0) contains the old size.
# /etc/vx/bin/vxmksdpart -g rootdg newroot-01 0 0x2 0x0
(command usage: vxmksdpart -g disk-group sub-disk disk-partition tag-in-hexadecimal flag-in-hexadecimal)

Now, the slice information for disk newroot (c2t2d0) is:
# prtvtoc -s /dev/rdsk/c2t2d0s2


6. Delete the rootalt volume from the new root disk.
# cd /
# umount /mnt

Stop the volume.
# vxvol stop rootalt

Disassociate the plex from the volume and remove the volume.
# vxplex dis rootvol-02
# vxedit rm rootalt

Disassociate the sub-disk from the plex and remove the plex.
# vxsd dis newroot-01
# vxedit rm rootvol-02

7. Mirror all the other volumes from the current root disk to the new root disk.
Do not mirror swap volumes. Swap slices will be created on the new disk manually.
In this example, the volumes to mirror are var and opt.

# vxassist -g rootdg mirror var newroot
# vxassist -g rootdg mirror opt newroot

The rootdg diskgroup now looks like this.
# vxprint -Qqhtg rootdg


8. Create the underlying physical partition for each of these volumes, using the vxmksdpart command.

From the sub-disks for the var and opt volumes on the new root disk, create the vtoc entries.
# /etc/vx/bin/vxmksdpart -g rootdg newroot-02 5 0x7 0x0
# /etc/vx/bin/vxmksdpart -g rootdg newroot-03 6 0x0 0x0

9. Remove all plexes and sub-disks from the new root disk.
Remove the disk from the rootdg disk group and take the disk out of Volume Manager control.

Remove the plexes first.
# vxplex -o rm dis var-02
# vxplex -o rm dis opt-02

Remove the only remaining sub-disk next.
# vxedit rm newroot-01

Remove the disk from rootdg.
# vxdg -g rootdg rmdisk newroot

Take the disk out of Volume Manager control.
# /etc/vx/bin/vxdiskunsetup c2t2d0

10. Using the Solaris format command, create a disk partition for swap on the new root disk.
The size of this partition may be adjusted according to the requirements of the system.
In this example, slice 1 has been used for the swap partition.
In case the new root disk has unallocated space at the end of the disk, create a dummy slice that uses up that space. If this is not done, then the private region uses space from the end of the disk when the disk is encapsulated at step 14. Since we already have free space at the beginning of the disk that was being used for the private region when this disk was under Volume Manager control (see prtvtoc output of disk at step 5), we may not want this space to be wasted. Please note that adding this dummy slice is not essential. The only purpose is to make use of the space already available at the beginning of the disk for the private region. The dummy volume created due to the encapsulation at step 14 may be deleted. The volume table of contents (VTOC) on the new root disk looks as shown below:
# prtvtoc -s /dev/rdsk/c2t2d0s2


Slice 0=root slice, slice 1=swap, slice 5=var, slice 6=opt, slice 7=dummy slice

11. Perform a file system check on all partitions containing file systems. In this example, there are three "ufs" file systems.
Clear the super-block flag for the file systems and any other errors encountered.
# fsck -F ufs /dev/rdsk/c2t2d0s0
# fsck -F ufs /dev/rdsk/c2t2d0s5
# fsck -F ufs /dev/rdsk/c2t2d0s6

12. Mount the root slice from the new disk on to /mnt and edit the /etc/system and /etc/vfstab files.
# mount -F ufs /dev/dsk/c2t2d0s0 /mnt

Edit the etc/system file and comment out the lines relevant to Volume Manager boot off encapsulated boot disks.
# vi /mnt/etc/system

Comment out the following two lines by placing an asterisk in the first column and then save the file.



Make a copy of etc/vfstab file and then edit the file. For all the file systems and swap volumes on the current root disk, change volume devices to disk partition devices (slices on new boot disk). Save the edited file.
# cd /mnt/etc
# cp vfstab vfstab.orig
# vi vfstab

In this example, the vfstab looks like this:



Unmount the root slice on the new boot disk.
# cd /
# umount /mnt

13. Take system to "ok" prompt and then boot off the new boot disk.
# init 0

ok> devalias

From the devalias output, select the new boot disk device alias. In our case, this is represented by the vx-newroot alias (a reset command may be required for the alias to be visible).

ok> boot vx-newroot

After the system has rebooted off the slices on the new boot disk, the file system mounts look as follows:
# df -k


Please note that although the system has used the new boot disk to boot off slices, Volume Manager is still able to start up due to the presence of the rootdg diskgroup. All rootdg volumes are started. However, due to the changes made to the vfstab file (at step 13), none of the file systems on the volumes in the old root disk should be mounted.

14. Encapsulate the new root disk and add it to the rootdg diskgroup.
Before we remove the original root disk from the rootdg diskgroup, it may be best to ensure that the underlying physical partitions exist for the volumes on that disk. By ensuring this, we can revert back to using that disk if necessary. Use the prtvtoc command to check for current slice information for the old root disk. Missing slices may be added to the disk VTOC using the vxmksdpart command (please see previous examples at steps 5 and 9).

Remove all volumes from the old root disk (and its mirrors, if any).
# vxassist -g rootdg remove volume rootvol
# vxassist -g rootdg remove volume swapvol
# vxassist -g rootdg remove volume var
# vxassist -g rootdg remove volume opt

Rename the old root disk. In this example, rootdisk is being renamed as rootold.
# vxedit -g rootdg rename rootdisk rootold

Encapsulate the new root disk (c2t2d0).
# /etc/vx/bin/vxencap rootdisk=c2t2d0

Set the default system boot device to be the new root disk. In our example, a device alias called vx-newroot already exists for the new disk.
# /etc/vx/bin/vxeeprom boot-device vx-newroot

Reboot the system. It will reboot twice, and come up on volumes on the new root disk.
# shutdown -g0 -y -i6

Examining the file systems mounted after the reboot, we can see volumes being used instead of slices.
# df -k


Remove the old root disk (and mirror disk, if applicable) from the rootdg diskgroup.
# vxdg -g rootdg rmdisk rootold

The disks on the system now appear as follows:
# vxdisk list


The rootdg diskgroup now consists of all the original volumes.
# vxprint -Qqhtg rootdg


In the output, note the volume rootdisk7vol. This volume has been created on account of the dummy slice that was added at step 10. This volume may be deleted as it does not contain any useful information.
# vxassist remove volume rootdisk7vol

The new root disk's VTOC looks as follows.
# prtvtoc -s /dev/rdsk/c2t2d0s2


15. If the new root disk is to be mirrored, ensure that a disk of at least similar capacity is available for the mirror.
You could, for example, replace the old boot disk with a new disk that is the same size as the new root disk.
In case a new disk will be used for the mirror, follow the steps described at step 1 (substitute the literal "new disk" with "mirror disk") to initialize the disk. Add the disk to the rootdg diskgroup.
Mirror all volumes on the boot disk to mirror disk (rootmir).
# /etc/vx/bin/vxmirror rootdisk rootmir  

At the conclusion of the above steps, we are left with a running Volume Manager configuration with an expanded root volume.x