Linux Scroll

Just another site

Category Archives: veritas

VCS WARNING V-16-1-10263 Cannot clear resource of Operations type None

Cannot clear resource of Operations type ‘None’

Description: For a persistent resource, which cannot be controlled by VCS,
faults cannot be cleared through VCS. Persistent resources are designated
with “No” (None) operations from VCS perspective. Examples of such resource
types are NIC, MultiNICA, MultiNICB, Proxy. VCS will only MONITOR those
resources. It will not attempt to bring the resource online or offline.

Solution: Please rectify the fault, for example, if the NIC pointed to by this
resource is malfunctioning, replace the NIC. VCS will detect the state of this
resource to be online if the fault has been rectified (outside VCS control).


Disabling Dynamic Multipathing (DMP) on VxVM

If planned maintenance on external infrastructure is scheduled, then disabling the affected paths on the hosts will avoid this issue.
Paths can be disabled using:

# vxdmpadm disable ctlr=affected_controllers

Once planned maintenance is finished, the affected paths can be enabled again using:

# vxdmpadm enable ctlr=affected_controllers

If the affected infrastructure goes down outside of scheduled maintenance, then disabling and enabling the controllers using the above commands also clears the QUIESCED state of the paths.

Failed disk in Solaris

To diagnose the problem:
From Solaris OS :

# format

# iostat -En
# iostat -en
-e Display device error summary statistics. The
total errors, hard errors, soft errors, and
transport errors are displayed.
-E Display all device error statistics.
-n Display names in descriptive format. For example,
cXtYdZ, rmt/N, server:/export/path.

Check messages file (/var/adm/messages) .

Is this disk managed by any Volume Manager? (VxVM, SDS/SVM)?

For disks running under Solstice Disk Suite/Solaris Volume Manager:

# metadb -i

# metastat -t

For disks running under Veritas Volume Manager (VxVM):

# vxprint -Ath

# vxdisk list

Troubleshooting Veritas Cluster Server (VCS)

Simple steps to troubleshoting vcs problem:

1. Check the vcs status:

$ /opt/VRTSvcs/bin/hastatus -summary

# hasum

2. Check for cluster communication status.
To determine any heartbeat failure.

$ gabconfig -a

If memberships were present (0 and 1 above), this could indicate a problem with gab or llt.
Start gab.

$ /etc/init.d/gab start

$ /sbin/gabconfig -x (Force start gab)

3. Check on LLT

$ lltconfig

$ lltstat -vvn
$ /etc/init.d/llt.rc stop
$ /etc/init.d/llt.rc start

4. Check VCS log

$ ls -altr /var/VRTSvcs/log

VERITAS Cluster Server

This application clustering solution provides application high availability for the most important applications in your data center. If it detects a fault, it takes action, recovering applications automatically. If your whole site goes down, it restarts those applications at another data center, so even during a disaster, your IT services keep running.

Term/Cmd/Pkg   Description                              Command / File                                    

VRTSvcs VERITAS Cluster Server
VRTSvcswz VERITAS Cluster Server Wizard
VRTScsga VERITAS Cluster Server Graphical Administrator
VRTSgab VERITAS Group Membership and Atomic Broadcast
VRTSllt VERITAS Low Latency Transport
VRTSvcsor VERITAS Cluster Server Oracle Enterprise Extension
VRTSvcssy VERITAS Cluster Server Sybase Enterprise Extension
VRTSperl VERITAS Perl for VRTSvcs

Cluster Name of your HA environment
Nodes Physical systems that make up the cluster
Service group Abstract container of related resources
Resource Cluster components (i.e. NICs, IPs, disk groups, volumes, mounts, processes, etc...)
Attributes Parameter values that define the resources
Dependencies Links between resources or service groups

Cluster Mgr Cluster Monitor : Log in, add clusters, change preferences
Cluster Mgr Cluster Explorer: Monitor systems, service grps, resources, attributes & dependencies
Cluster Mgr Log Desk : Monitor log messages received from engine, view GUI commands
Cluster Mgr Command Center : Build VCS commands and send to engine
LLT Low Latency transport provides fast kernel-kernel comm. & monitors network connx.
GAB Grp membership & Atomic Broadcast maintains a synch. state & monitors disk comm.

Config files VCS etc directory $VCSETC=/etc/VRTSvcs
Config files VCS configuration directories $VCSCONF=/etc/VRTSvcs/conf/config
Binary files VCS opt directory $VCSOPT=/opt/VRTSvcs
Binary files VCS binary path $VCSBIN=/opt/VRTSvcs/bin
Log files VCS log path $VCSLOG=/var/VRTSvcs/log
Config files VCS configuration file /etc/VRTSvcs/conf/config/
LLT tab file LLT configuration file /etc/llttab
LLT hosts file LLT host name database /etc/llthosts
GAB file Grp membership & Atomic Broadcast file /etc/gabtab

quick-start VCS Quick-start wizard # $VCS_HOME/wizards/config/quick_start
quick-NFS VCS Quick-NFS wizard # $VCS_HOME/wizards/services/quick_nfs
llt Verify LLT # /sbin/llstat -n
llt Get interface MAC Address # /opt/VRTSllt/getmac device_name
llt Check network connectivity # /opt/VRTSllt/dlpiping -s|-c -v device_name
gab Verify GAB # /sbin/gabconfig -a ; /sbin/gabconfig -l
hasys List systems in cluster # /opt/VRTSvcs/bin/hasys -list
hasys Detailed info on each cluster node # /opt/VRTSvcs/bin/hasys -display (sysname)
hasys Increase system count in gabtab startup # /opt/VRTSvcs/bin/hasys -add (sysname)
hasys Delete a system # /opt/VRTSvcs/bin/hasys -delete (sysname)
hastart Start VCS cluster # /opt/VRTSvcs/bin/hastart
hastart Force start a stale VCS cluster # /opt/VRTSvcs/bin/hastart -force -stale
hastop Stop VCS on all systems # /opt/VRTSvcs/bin/hastop -all
hastop Stop VCS had, keep srvc-groups running # /opt/VRTSvcs/bin/hastop -local -force
hastop Stop VCS, migrate srvc-groups to sysname # /opt/VRTSvcs/bin/hastop -sys (sysname) -evacuate
hastatus Provide continual status of service grps # /opt/VRTSvcs/bin/hastatus
hastatus Verify status of service groups # /opt/VRTSvcs/bin/hastatus -summary
hacf Check for syntax errors in # /opt/VRTSvcs/bin/hacf -verify /etc/VRTSvcs/conf/config/
hacf Generate dependency tree in # /opt/VRTSvcs/bin/hacf -generate /etc/VRTSvcs/conf/config/

hares List all resources # /opt/VRTSvcs/bin/hares -list
hares List a resource's dependencies # /opt/VRTSvcs/bin/hares -dep (resource_name)
hares Get detailed info on a resource # /opt/VRTSvcs/bin/hares -display (resource)
hares Add a resource # /opt/VRTSvcs/bin/hares -add (resource_name (resource_type (service_group)
hares Modify attributes of the new resource # /opt/VRTSvcs/bin/hares -modify (resource_name (attribute_name (value)
hares Delete a resource # /opt/VRTSvcs/bin/hares -delete (resource_name)
hares Online a resource # /opt/VRTSvcs/bin/hares -online (resource_name) -sys (system_name)
hares Offline a resource # /opt/VRTSvcs/bin/hares -offline (resource_name) -sys (system_name)
hares Monitor resource on a system # /opt/VRTSvcs/bin/hares -probe (resource_name) -sys (system_name)
hares Clear a faulted resource # /opt/VRTSvcs/bin/hares -clear (resource_name) [-sys system_name]
hares Make a resource's attribute value local # /opt/VRTSvcs/bin/hares -local (resource_name) (attribute_name) value)
hares Make a resource's attribute value global # /opt/VRTSvcs/bin/hares -global (resource_name) (attribute_name) value)
hares Specify a dependency between 2 resources # /opt/VRTSvcs/bin/hares -link (parent_res) (child_res)
hares Remove dependency between 2 resources # /opt/VRTSvcs/bin/hares -unlink (parent_res) (child_res)
hares Modify a Share res. by adding options # /opt/VRTSvcs/bin/hares Share_cicgt-as4-p_apps Options "%-o rw,root=dcsa-cln1"

hagrp List all service groups # /opt/VRTSvcs/bin/hagrp -list
hagrp List a service group's resources # /opt/VRTSvcs/bin/hagrp -resources [service_group]
hagrp List a service group's dependencies # /opt/VRTSvcs/bin/hagrp -dep [service_group]
hagrp Detailed info about a service group # /opt/VRTSvcs/bin/hagrp -display [service_group]
hagrp Start service group, bring res. online # /opt/VRTSvcs/bin/hagrp -online (service_group) -sys (system_name)
hagrp Stop service group, bring res. offline # /opt/VRTSvcs/bin/hagrp -offline (service_group) -sys (system_name)
hagrp Switch service group between nodes # /opt/VRTSvcs/bin/hagrp -switch (service_group) -to (system_name)
hagrp Freeze svcgroup, (disable onl. & offl.) # /opt/VRTSvcs/bin/hagrp -freeze (service_group) [-persistent]
hagrp Thaw a svcgroup, (enable onl. & offl.) # /opt/VRTSvcs/bin/hagrp -unfreeze (service_group) [-persistent]
hagrp Enable a service group # /opt/VRTSvcs/bin/hagrp -enable (service_group) [-sys system_name]
hagrp Disable a service group # /opt/VRTSvcs/bin/hagrp -disable (service_group) [-sys system_name]
hagrp Enable all resources in a service group # /opt/VRTSvcs/bin/hagrp -enableresources (service_group)
hagrp Disable all resources in a service group # /opt/VRTSvcs/bin/hagrp -disableresources (service_group)
hagrp Specify dependenciy between 2 svc groups # /opt/VRTSvcs/bin/hagrp -link (parent_group) (child_group) (relationship)
hagrp Remove dependenciy between 2 svc groups # /opt/VRTSvcs/bin/hagrp -unlink (parent_group) (child_group)
hagrp Auto-Enable a servicegroup marked # /opt/VRTSvcs/bin/hagrp -autoenable (service_group) [-sys system_name]
disabled due to prob with system_name.

hatype List resource types # /opt/VRTSvcs/bin/hatype -list
hatype Detailed info on a resource type # /opt/VRTSvcs/bin/hatype -display (resource_type)
hatype List all resources of a part. type # /opt/VRTSvcs/bin/hatype -resources (resource_type)
hatype Add a resource type # /opt/VRTSvcs/bin/hatype -add (resource_type)
hatype Set static attribute values # /opt/VRTSvcs/bin/hatype -modify ...
hatype Delete a resource type # /opt/VRTSvcs/bin/hatype -delete (resource_type)
haattr Add Attribute to a Type definition # /opt/VRTSvcs/bin/haattr -add (resource_type) (attribute_name) (attribute_type -integer, -string, -vector)
haattr Delete a Entry in a Type definition # /opt/VRTSvcs/bin/haattr -delete (resource_type) (attribute_name)
haconf Set VCS configuration file to r/w mode # /opt/VRTSvcs/bin/haconf -makerw
haconf Set VCS configuration file to read mode # /opt/VRTSvcs/bin/haconf -dump -makero
hauser Add a user with r/w access to VCS # /opt/VRTSvcs/bin/hauser -add (user_name)
hauser Add a user with read access only to VCS # /opt/VRTSvcs/bin/hauser -add VCSGuest
hauser Update a user # /opt/VRTSvcs/bin/hauser -update (user_name)
hauser Delete a user # /opt/VRTSvcs/bin/hauser -delete (user_name)
hauser Display all users # /opt/VRTSvcs/bin/hauser -display
haagent Start agents manually # haagent -start (agent_name) -sys (system_name)
haagent Stop agents manually # haagent -stop (agent_name) -sys (system_name)
hagui Start Cluster Manager # /opt/VRTSvcs/bin/hagui
hagui Start Cluster Manager in debug mode # /opt/VRTSvcs/bin/hagui -D