Debian-facile

/run/rpcbind.sock

Can't connect to local MySQL server through socket '/var/mysql/mysql.sock' (2)

node1 IPaddr::192.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2

node1 IPaddr::192.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql

node1 IPaddr::192.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2

Sep 14 10:15:14 node1 heartbeat: [3275]: info: No log entry found in ha.cf -- use logd
Sep 14 10:15:14 node1 heartbeat: [3275]: info: Enabling logging daemon
Sep 14 10:15:14 node1 heartbeat: [3275]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 10:15:14 node1 heartbeat: [3275]: info: **************************
Sep 14 10:15:14 node1 heartbeat: [3275]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 10:15:14 node1 heartbeat: [3276]: info: heartbeat: version 3.0.5
Sep 14 10:15:14 node1 heartbeat: [3276]: WARN: No Previous generation - starting at 1442218515
Sep 14 10:15:14 node1 heartbeat: [3276]: info: Heartbeat generation: 1442218515
Sep 14 10:15:14 node1 heartbeat: [3276]: info: No uuid found for current node - generating a new uuid.
Sep 14 10:15:14 node1 heartbeat: [3276]: info: Creating FIFO /var/lib/heartbeat/fifo.
Sep 14 10:15:14 node1 heartbeat: [3276]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 10:15:14 node1 heartbeat: [3276]: info: Local status now set to: 'up'
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Link node2:eth0 up.
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Status update for node node2: status up
Sep 14 10:15:26 node1 heartbeat: [3286]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:15:26 node1 harc[3286]: [3293]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Comm_now_up(): updating status to active
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Local status now set to: 'active'
Sep 14 10:15:26 node1 heartbeat: [3276]: debug: get_delnodelist: delnodelist=
Sep 14 10:15:27 node1 heartbeat: [3276]: info: Status update for node node2: status active
Sep 14 10:15:27 node1 heartbeat: [3300]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:15:27 node1 harc[3300]: [3306]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:15:37 node1 heartbeat: [3276]: info: remote resource transition completed.
Sep 14 10:15:37 node1 heartbeat: [3276]: info: remote resource transition completed.
Sep 14 10:15:37 node1 heartbeat: [3276]: info: Initial resource acquisition complete (T_RESOURCES(us))
Sep 14 10:15:37 node1 IPaddr[3346]: [3378]: INFO:  Resource is stopped
Sep 14 10:15:37 node1 heartbeat: [3311]: info: Local Resource acquisition completed.
Sep 14 10:15:37 node1 heartbeat: [3276]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:15:37 node1 heartbeat: [3382]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:15:37 node1 harc[3382]: [3388]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 10:15:37 node1 ip-request-resp[3382]: [3394]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 10:15:37 node1 ResourceManager[3395]: [3406]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 10:15:37 node1 IPaddr[3418]: [3450]: INFO:  Resource is stopped
Sep 14 10:15:37 node1 ResourceManager[3395]: [3465]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:15:37 node1 IPaddr[3490]: [3518]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:15:37 node1 IPaddr[3490]: [3540]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:15:37 node1 IPaddr[3466]: [3559]: INFO:  Success
Sep 14 10:15:37 node1 Filesystem[3586]: [3625]: INFO:  Running OK
Sep 14 10:15:37 node1 drbdlinks[3632]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:15:37 node1 drbdlinks[3632]: Status mode returning stopped
Sep 14 10:15:37 node1 ResourceManager[3395]: [3642]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 10:15:37 node1 drbdlinks[3643]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:15:37 node1 drbdlinks[3643]: Exiting with no errors
Sep 14 10:15:37 node1 ResourceManager[3395]: [3664]: info: Running /etc/init.d/apache2  start
Sep 14 10:17:01 node1 /USR/SBIN/CRON[3696]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 14 10:19:17 node1 heartbeat: [3276]: info: Heartbeat shutdown in progress. (3276)
Sep 14 10:19:17 node1 heartbeat: [3715]: info: Giving up all HA resources.
Sep 14 10:19:17 node1 ResourceManager[3729]: [3740]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:19:17 node1 ResourceManager[3729]: [3751]: info: Running /etc/init.d/mysql  stop
Sep 14 10:19:17 node1 ResourceManager[3729]: [3789]: info: Running /etc/init.d/apache2  stop
Sep 14 10:19:18 node1 ResourceManager[3729]: [3822]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:19:18 node1 drbdlinks[3823]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:19:18 node1 drbdlinks[3823]: Exiting with no errors
Sep 14 10:19:18 node1 ResourceManager[3729]: [3838]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:19:18 node1 Filesystem[3845]: [3873]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:19:18 node1 Filesystem[3845]: [3888]: INFO: Trying to unmount /cluster
Sep 14 10:19:18 node1 Filesystem[3845]: [3896]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:18 node1 Filesystem[3845]: [3899]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:19 node1 Filesystem[3845]: [3908]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:19 node1 Filesystem[3845]: [3911]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:20 node1 Filesystem[3845]: [3920]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:20 node1 Filesystem[3845]: [3923]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:21 node1 Filesystem[3845]: [3932]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:21 node1 Filesystem[3845]: [3935]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:22 node1 Filesystem[3845]: [3944]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:22 node1 Filesystem[3845]: [3947]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:23 node1 Filesystem[3845]: [3956]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:24 node1 Filesystem[3845]: [3959]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:25 node1 Filesystem[3845]: [3968]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:25 node1 Filesystem[3845]: [3971]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:26 node1 Filesystem[3845]: [3980]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:26 node1 Filesystem[3845]: [3983]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:27 node1 Filesystem[3845]: [3992]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:27 node1 Filesystem[3845]: [3995]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:28 node1 Filesystem[3845]: [4004]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:28 node1 Filesystem[3845]: [4007]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:29 node1 Filesystem[3845]: [4016]: ERROR: Couldn't unmount /cluster; trying cleanup with KILL
Sep 14 10:19:29 node1 Filesystem[3845]: [4019]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:30 node1 Filesystem[3845]: [4028]: INFO: unmounted /cluster successfully
Sep 14 10:19:30 node1 Filesystem[3839]: [4035]: INFO:  Success
Sep 14 10:19:30 node1 ResourceManager[3729]: [4050]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:19:30 node1 kernel: [ 3532.543969] block drbd0: role( Primary -> Secondary )
Sep 14 10:19:30 node1 kernel: [ 3532.543979] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:19:30 node1 kernel: [ 3532.545998] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:19:30 node1 ResourceManager[3729]: [4075]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:19:30 node1 IPaddr[4100]: [4111]: INFO: ifconfig eth0:0 down
Sep 14 10:19:30 node1 IPaddr[4076]: [4115]: INFO:  Success
Sep 14 10:19:30 node1 heartbeat: [3715]: info: All HA resources relinquished.
Sep 14 10:19:30 node1 kernel: [ 3532.792118] block drbd0: peer( Secondary -> Primary )
Sep 14 10:19:30 node1 heartbeat: [3276]: WARN: 1 lost packet(s) for [node2] [134:136]
Sep 14 10:19:30 node1 heartbeat: [3276]: info: No pkts missing from node2!
Sep 14 10:19:32 node1 heartbeat: [3276]: info: killing HBREAD process 3283 with signal 15
Sep 14 10:19:32 node1 heartbeat: [3276]: info: killing HBFIFO process 3281 with signal 15
Sep 14 10:19:32 node1 heartbeat: [3276]: info: killing HBWRITE process 3282 with signal 15
Sep 14 10:19:32 node1 heartbeat: [3276]: info: Core process 3282 exited. 3 remaining
Sep 14 10:19:32 node1 heartbeat: [3276]: info: Core process 3281 exited. 2 remaining
Sep 14 10:19:32 node1 heartbeat: [3276]: info: Core process 3283 exited. 1 remaining
Sep 14 10:19:32 node1 heartbeat: [3276]: info: node1 Heartbeat shutdown complete.
Sep 14 10:19:46 node1 kernel: [ 3548.450222] block drbd0: peer( Primary -> Secondary )
Sep 14 10:21:06 node1 heartbeat: [4267]: info: No log entry found in ha.cf -- use logd
Sep 14 10:21:06 node1 heartbeat: [4267]: info: Enabling logging daemon
Sep 14 10:21:06 node1 heartbeat: [4267]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 10:21:06 node1 heartbeat: [4267]: info: **************************
Sep 14 10:21:06 node1 heartbeat: [4267]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 10:21:06 node1 heartbeat: [4268]: info: heartbeat: version 3.0.5
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Heartbeat generation: 1442218516
Sep 14 10:21:06 node1 heartbeat: [4268]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Local status now set to: 'up'
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Link node2:eth0 up.
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Status update for node node2: status active
Sep 14 10:21:06 node1 heartbeat: [4276]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:21:06 node1 harc[4276]: [4283]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Comm_now_up(): updating status to active
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Local status now set to: 'active'
Sep 14 10:21:07 node1 heartbeat: [4268]: info: remote resource transition completed.
Sep 14 10:21:07 node1 heartbeat: [4268]: info: remote resource transition completed.
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Local Resource acquisition completed. (none)
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Initial resource acquisition complete (T_RESOURCES(them))
Sep 14 10:21:34 node1 heartbeat: [4268]: info: Received shutdown notice from 'node2'.
Sep 14 10:21:34 node1 heartbeat: [4268]: info: Resources being acquired from node2.
Sep 14 10:21:34 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:21:34 node1 heartbeat: [4292]: info: acquire all HA resources (standby).
Sep 14 10:21:34 node1 ResourceManager[4321]: [4340]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:34 node1 IPaddr[4365]: [4428]: INFO:  Resource is stopped
Sep 14 10:21:34 node1 IPaddr[4364]: [4429]: INFO:  Resource is stopped
Sep 14 10:21:34 node1 heartbeat: [4293]: info: Local Resource acquisition completed.
Sep 14 10:21:34 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 2
Sep 14 10:21:34 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:21:34 node1 ResourceManager[4321]: [4447]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:21:34 node1 IPaddr[4472]: [4500]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:21:34 node1 IPaddr[4472]: [4522]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:21:34 node1 IPaddr[4448]: [4541]: INFO:  Success
Sep 14 10:21:34 node1 ResourceManager[4321]: [4571]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 10:21:34 node1 kernel: [ 3657.051757] block drbd0: role( Secondary -> Primary )
Sep 14 10:21:34 node1 Filesystem[4588]: [4632]: INFO:  Resource is stopped
Sep 14 10:21:34 node1 ResourceManager[4321]: [4647]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 10:21:34 node1 Filesystem[4654]: [4682]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 10:21:34 node1 Filesystem[4648]: [4702]: INFO:  Success
Sep 14 10:21:34 node1 kernel: [ 3657.120578] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 10:21:34 node1 drbdlinks[4709]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:34 node1 drbdlinks[4709]: Status mode returning stopped
Sep 14 10:21:34 node1 ResourceManager[4321]: [4719]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 10:21:34 node1 drbdlinks[4720]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:34 node1 drbdlinks[4720]: Exiting with no errors
Sep 14 10:21:34 node1 ResourceManager[4321]: [4741]: info: Running /etc/init.d/apache2  start
Sep 14 10:21:34 node1 ResourceManager[4321]: [4792]: info: Running /etc/init.d/mysql  start
Sep 14 10:21:35 node1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0+deb7u1) starting as process 5149 ...
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Note] Plugin 'FEDERATED' is disabled.
Sep 14 10:21:35 node1 mysqld: #007/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: The InnoDB memory heap is disabled
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Compressed tables use zlib 1.2.7
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Using Linux native AIO
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Initializing buffer pool, size = 128.0M
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Completed initialization of buffer pool
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35  InnoDB: Operating system error number 13 in a file operation.
Sep 14 10:21:35 node1 mysqld: InnoDB: The error means mysqld does not have the access rights to
Sep 14 10:21:35 node1 mysqld: InnoDB: the directory.
Sep 14 10:21:35 node1 mysqld: InnoDB: File name ./ibdata1
Sep 14 10:21:35 node1 mysqld: InnoDB: File operation call: 'create'.
Sep 14 10:21:35 node1 mysqld: InnoDB: Cannot continue operation.
Sep 14 10:21:35 node1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Sep 14 10:21:40 node1 heartbeat: [4268]: WARN: node node2: is dead
Sep 14 10:21:40 node1 heartbeat: [4268]: info: Cancelling pending standby operation
Sep 14 10:21:40 node1 heartbeat: [4268]: info: Dead node node2 gave up resources.
Sep 14 10:21:41 node1 heartbeat: [4268]: info: Link node2:eth0 dead.
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]:
Sep 14 10:21:49 node1 ResourceManager[4321]: [5346]: ERROR: Return code 1 from /etc/init.d/mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5348]: CRIT: Giving up resources due to failure of mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5350]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5361]: info: Running /etc/init.d/mysql  stop
Sep 14 10:21:49 node1 ResourceManager[4321]: [5399]: info: Running /etc/init.d/apache2  stop
Sep 14 10:21:50 node1 ResourceManager[4321]: [5432]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:21:50 node1 drbdlinks[5433]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:50 node1 drbdlinks[5433]: Exiting with no errors
Sep 14 10:21:50 node1 ResourceManager[4321]: [5448]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:21:50 node1 Filesystem[5455]: [5483]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:21:50 node1 Filesystem[5455]: [5498]: INFO: Trying to unmount /cluster
Sep 14 10:21:50 node1 Filesystem[5455]: [5506]: INFO: unmounted /cluster successfully
Sep 14 10:21:50 node1 Filesystem[5449]: [5513]: INFO:  Success
Sep 14 10:21:50 node1 ResourceManager[4321]: [5528]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:21:50 node1 kernel: [ 3672.661265] block drbd0: role( Primary -> Secondary )
Sep 14 10:21:50 node1 kernel: [ 3672.661273] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:21:50 node1 kernel: [ 3672.661831] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:21:50 node1 ResourceManager[4321]: [5553]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:21:50 node1 IPaddr[5578]: [5589]: INFO: ifconfig eth0:0 down
Sep 14 10:21:50 node1 IPaddr[5554]: [5593]: INFO:  Success
Sep 14 10:21:50 node1 heartbeat: [4292]: info: all HA resource acquisition completed (standby).
Sep 14 10:21:50 node1 heartbeat: [4268]: ERROR: Ignored standby message 'done' from node1 in state 0
Sep 14 10:21:50 node1 heartbeat: [5595]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:21:50 node1 harc[5595]: [5602]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:21:50 node1 mach_down[5607]: [5627]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 10:21:50 node1 mach_down[5607]: [5632]: info: mach_down takeover complete for node node2.
Sep 14 10:21:50 node1 heartbeat: [4268]: info: mach_down takeover complete.
Sep 14 10:21:50 node1 heartbeat: [5633]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:21:50 node1 harc[5633]: [5639]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 10:21:50 node1 ip-request-resp[5633]: [5645]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 10:21:50 node1 ResourceManager[5646]: [5657]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:50 node1 IPaddr[5669]: [5701]: INFO:  Resource is stopped
Sep 14 10:21:50 node1 ResourceManager[5646]: [5716]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:21:50 node1 IPaddr[5741]: [5769]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:21:50 node1 IPaddr[5741]: [5791]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:21:50 node1 IPaddr[5717]: [5810]: INFO:  Success
Sep 14 10:21:50 node1 ResourceManager[5646]: [5840]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 10:21:50 node1 kernel: [ 3672.795117] block drbd0: role( Secondary -> Primary )
Sep 14 10:21:50 node1 Filesystem[5857]: [5901]: INFO:  Resource is stopped
Sep 14 10:21:50 node1 ResourceManager[5646]: [5916]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 10:21:50 node1 Filesystem[5923]: [5951]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 10:21:50 node1 kernel: [ 3672.882601] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 10:21:50 node1 Filesystem[5917]: [5971]: INFO:  Success
Sep 14 10:21:50 node1 drbdlinks[5978]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:50 node1 drbdlinks[5978]: Status mode returning stopped
Sep 14 10:21:50 node1 ResourceManager[5646]: [5988]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 10:21:50 node1 drbdlinks[5989]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:50 node1 drbdlinks[5989]: Exiting with no errors
Sep 14 10:21:50 node1 ResourceManager[5646]: [6010]: info: Running /etc/init.d/apache2  start
Sep 14 10:21:50 node1 ResourceManager[5646]: [6061]: info: Running /etc/init.d/mysql  start
Sep 14 10:21:50 node1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0+deb7u1) starting as process 6417 ...
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Note] Plugin 'FEDERATED' is disabled.
Sep 14 10:21:50 node1 mysqld: #007/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: The InnoDB memory heap is disabled
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Compressed tables use zlib 1.2.7
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Using Linux native AIO
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Initializing buffer pool, size = 128.0M
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Completed initialization of buffer pool
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50  InnoDB: Operating system error number 13 in a file operation.
Sep 14 10:21:50 node1 mysqld: InnoDB: The error means mysqld does not have the access rights to
Sep 14 10:21:50 node1 mysqld: InnoDB: the directory.
Sep 14 10:21:50 node1 mysqld: InnoDB: File name ./ibdata1
Sep 14 10:21:50 node1 mysqld: InnoDB: File operation call: 'create'.
Sep 14 10:21:50 node1 mysqld: InnoDB: Cannot continue operation.
Sep 14 10:21:50 node1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]:
Sep 14 10:22:04 node1 ResourceManager[5646]: [6638]: ERROR: Return code 1 from /etc/init.d/mysql
Sep 14 10:22:04 node1 ResourceManager[5646]: [6640]: CRIT: Giving up resources due to failure of mysql
Sep 14 10:22:04 node1 ResourceManager[5646]: [6642]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:22:04 node1 ResourceManager[5646]: [6653]: info: Running /etc/init.d/mysql  stop
Sep 14 10:22:04 node1 ResourceManager[5646]: [6691]: info: Running /etc/init.d/apache2  stop
Sep 14 10:22:06 node1 ResourceManager[5646]: [6724]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:22:06 node1 drbdlinks[6725]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:22:06 node1 drbdlinks[6725]: Exiting with no errors
Sep 14 10:22:06 node1 ResourceManager[5646]: [6740]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:22:06 node1 Filesystem[6747]: [6775]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:22:06 node1 Filesystem[6747]: [6790]: INFO: Trying to unmount /cluster
Sep 14 10:22:06 node1 Filesystem[6747]: [6798]: INFO: unmounted /cluster successfully
Sep 14 10:22:06 node1 Filesystem[6741]: [6805]: INFO:  Success
Sep 14 10:22:06 node1 ResourceManager[5646]: [6820]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:22:06 node1 kernel: [ 3688.435641] block drbd0: role( Primary -> Secondary )
Sep 14 10:22:06 node1 kernel: [ 3688.435651] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:22:06 node1 kernel: [ 3688.436451] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:22:06 node1 ResourceManager[5646]: [6845]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:22:06 node1 IPaddr[6870]: [6881]: INFO: ifconfig eth0:0 down
Sep 14 10:22:06 node1 IPaddr[6846]: [6885]: INFO:  Success
Sep 14 10:22:20 node1 hb_standby[5594]: [6903]: Going standby [foreign].
Sep 14 10:22:20 node1 heartbeat: [4268]: info: node1 wants to go standby [foreign]
Sep 14 10:22:32 node1 heartbeat: [4268]: WARN: No reply to standby request.  Standby request cancelled.
Sep 14 10:22:36 node1 hb_standby[6886]: [6921]: Going standby [foreign].
Sep 14 10:22:36 node1 heartbeat: [4268]: info: node1 wants to go standby [foreign]
Sep 14 10:22:48 node1 heartbeat: [4268]: WARN: No reply to standby request.  Standby request cancelled.
Sep 14 10:24:23 node1 kernel: [ 3826.174202] block drbd0: role( Secondary -> Primary )
Sep 14 10:24:38 node1 kernel: [ 3840.546456] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Heartbeat restart on node node2
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Link node2:eth0 up.
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Status update for node node2: status init
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Status update for node node2: status up
Sep 14 10:24:51 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:24:51 node1 heartbeat: [6988]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:24:51 node1 harc[6988]: [6994]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:24:51 node1 heartbeat: [6999]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:24:51 node1 harc[6999]: [7005]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:24:52 node1 heartbeat: [4268]: debug: get_delnodelist: delnodelist=
Sep 14 10:24:53 node1 heartbeat: [4268]: info: Status update for node node2: status active
Sep 14 10:24:53 node1 heartbeat: [7010]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:24:53 node1 harc[7010]: [7016]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:24:53 node1 heartbeat: [4268]: info: remote resource transition completed.
Sep 14 10:25:06 node1 heartbeat: [4268]: info: Heartbeat shutdown in progress. (4268)
Sep 14 10:25:06 node1 heartbeat: [7036]: info: Giving up all HA resources.
Sep 14 10:25:06 node1 ResourceManager[7050]: [7061]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 10:25:06 node1 ResourceManager[7050]: [7072]: info: Running /etc/init.d/apache2  stop
Sep 14 10:25:06 node1 ResourceManager[7050]: [7100]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:25:06 node1 drbdlinks[7101]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:25:06 node1 drbdlinks[7101]: Exiting with no errors
Sep 14 10:25:06 node1 ResourceManager[7050]: [7116]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:25:06 node1 Filesystem[7123]: [7151]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:25:06 node1 Filesystem[7123]: [7166]: INFO: Trying to unmount /cluster
Sep 14 10:25:06 node1 Filesystem[7123]: [7174]: INFO: unmounted /cluster successfully
Sep 14 10:25:06 node1 Filesystem[7117]: [7181]: INFO:  Success
Sep 14 10:25:06 node1 ResourceManager[7050]: [7196]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:25:06 node1 kernel: [ 3868.984325] block drbd0: role( Primary -> Secondary )
Sep 14 10:25:06 node1 kernel: [ 3868.984335] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:25:06 node1 kernel: [ 3868.984975] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:25:06 node1 ResourceManager[7050]: [7221]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:25:06 node1 IPaddr[7222]: [7254]: INFO:  Success
Sep 14 10:25:06 node1 heartbeat: [7036]: info: All HA resources relinquished.
Sep 14 10:25:07 node1 kernel: [ 3869.439181] block drbd0: peer( Secondary -> Primary )
Sep 14 10:25:07 node1 heartbeat: [4268]: WARN: 1 lost packet(s) for [node2] [18:20]
Sep 14 10:25:07 node1 heartbeat: [4268]: info: No pkts missing from node2!
Sep 14 10:25:08 node1 heartbeat: [4268]: info: killing HBFIFO process 4272 with signal 15
Sep 14 10:25:08 node1 heartbeat: [4268]: info: killing HBWRITE process 4273 with signal 15
Sep 14 10:25:08 node1 heartbeat: [4268]: info: killing HBREAD process 4274 with signal 15
Sep 14 10:25:08 node1 heartbeat: [4268]: info: Core process 4273 exited. 3 remaining
Sep 14 10:25:08 node1 heartbeat: [4268]: info: Core process 4272 exited. 2 remaining
Sep 14 10:25:08 node1 heartbeat: [4268]: info: Core process 4274 exited. 1 remaining
Sep 14 10:25:08 node1 heartbeat: [4268]: info: node1 Heartbeat shutdown complete.
Sep 14 10:25:48 node1 heartbeat: [7356]: info: No log entry found in ha.cf -- use logd
Sep 14 10:25:48 node1 heartbeat: [7356]: info: Enabling logging daemon
Sep 14 10:25:48 node1 heartbeat: [7356]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 10:25:48 node1 heartbeat: [7356]: info: **************************
Sep 14 10:25:48 node1 heartbeat: [7356]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 10:25:48 node1 heartbeat: [7357]: info: heartbeat: version 3.0.5
Sep 14 10:25:48 node1 heartbeat: [7357]: info: Heartbeat generation: 1442218517
Sep 14 10:25:48 node1 heartbeat: [7357]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 10:25:48 node1 heartbeat: [7357]: info: Local status now set to: 'up'
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Link node2:eth0 up.
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Status update for node node2: status active
Sep 14 10:25:49 node1 heartbeat: [7365]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:25:49 node1 harc[7365]: [7372]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Comm_now_up(): updating status to active
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Local status now set to: 'active'
Sep 14 10:25:50 node1 heartbeat: [7357]: info: remote resource transition completed.
Sep 14 10:25:50 node1 heartbeat: [7357]: info: remote resource transition completed.
Sep 14 10:25:50 node1 heartbeat: [7357]: info: Local Resource acquisition completed. (none)
Sep 14 10:25:50 node1 heartbeat: [7357]: info: Initial resource acquisition complete (T_RESOURCES(them))
Sep 14 10:36:22 node1 kernel: [ 4544.524360] e1000: eth0 NIC Link is Down
Sep 14 10:36:26 node1 heartbeat: [7357]: WARN: node node2: is dead
Sep 14 10:36:26 node1 heartbeat: [7357]: WARN: No STONITH device configured.
Sep 14 10:36:26 node1 heartbeat: [7357]: WARN: Shared disks are not protected.
Sep 14 10:36:26 node1 heartbeat: [7357]: info: Resources being acquired from node2.
Sep 14 10:36:26 node1 heartbeat: [7357]: info: Link node2:eth0 dead.
Sep 14 10:36:26 node1 heartbeat: [7379]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:36:26 node1 harc[7379]: [7392]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:36:26 node1 mach_down[7405]: [7442]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 10:36:26 node1 mach_down[7405]: [7452]: info: mach_down takeover complete for node node2.
Sep 14 10:36:26 node1 heartbeat: [7357]: info: mach_down takeover complete.
Sep 14 10:36:26 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:36:26 node1 IPaddr[7448]: [7483]: INFO:  Resource is stopped
Sep 14 10:36:26 node1 heartbeat: [7380]: info: Local Resource acquisition completed.
Sep 14 10:36:26 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:36:26 node1 heartbeat: [7487]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:36:26 node1 harc[7487]: [7493]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 10:36:26 node1 ip-request-resp[7487]: [7499]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 10:36:26 node1 ResourceManager[7500]: [7511]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 10:36:26 node1 IPaddr[7523]: [7555]: INFO:  Resource is stopped
Sep 14 10:36:26 node1 ResourceManager[7500]: [7570]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:36:26 node1 IPaddr[7595]: [7623]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:36:26 node1 IPaddr[7595]: [7645]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:36:26 node1 IPaddr[7571]: [7664]: INFO:  Success
Sep 14 10:36:26 node1 ResourceManager[7500]: [7694]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 10:36:28 node1 kernel: [ 4550.533774] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 14 10:36:29 node1 heartbeat: [7357]: CRIT: Cluster node node2 returning after partition.
Sep 14 10:36:29 node1 heartbeat: [7357]: info: For information on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain
Sep 14 10:36:29 node1 heartbeat: [7357]: WARN: Deadtime value may be too small.
Sep 14 10:36:29 node1 heartbeat: [7357]: info: See FAQ for information on tuning deadtime.
Sep 14 10:36:29 node1 heartbeat: [7357]: info: URL: http://linux-ha.org/wiki/FAQ#Heavy_Load
Sep 14 10:36:29 node1 heartbeat: [7357]: info: Link node2:eth0 up.
Sep 14 10:36:29 node1 heartbeat: [7357]: WARN: Late heartbeat: Node node2: interval 8010 ms
Sep 14 10:36:29 node1 heartbeat: [7357]: info: Status update for node node2: status active
Sep 14 10:36:29 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:12:57 node1 heartbeat: [7357]: WARN: Shutdown delayed until current resource activity finishes.
Sep 14 11:12:57 node1 kernel: [ 4554.433090] block drbd0: peer( Primary -> Secondary )
Sep 14 11:12:58 node1 heartbeat: [7357]: info: Received shutdown notice from 'node2'.
Sep 14 11:12:58 node1 heartbeat: [7357]: info: Resources being acquired from node2.
Sep 14 11:12:58 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:12:58 node1 IPaddr[7744]: [7782]: INFO:  Running OK
Sep 14 11:12:58 node1 heartbeat: [7709]: info: Local Resource acquisition completed.
Sep 14 11:12:58 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:12:58 node1 kernel: [ 4555.399627] block drbd0: role( Secondary -> Primary )
Sep 14 11:12:58 node1 Filesystem[7798]: [7842]: INFO:  Resource is stopped
Sep 14 11:12:58 node1 ResourceManager[7500]: [7857]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 11:12:58 node1 Filesystem[7864]: [7892]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 11:12:58 node1 Filesystem[7858]: [7912]: INFO:  Success
Sep 14 11:12:58 node1 kernel: [ 4555.472030] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 11:12:58 node1 drbdlinks[7919]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:12:58 node1 drbdlinks[7919]: Status mode returning stopped
Sep 14 11:12:58 node1 ResourceManager[7500]: [7929]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 11:12:58 node1 drbdlinks[7930]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:12:58 node1 drbdlinks[7930]: Exiting with no errors
Sep 14 11:12:58 node1 ResourceManager[7500]: [7951]: info: Running /etc/init.d/apache2  start
Sep 14 11:12:58 node1 heartbeat: [7965]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:12:58 node1 harc[7965]: [7971]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:12:58 node1 heartbeat: [7980]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:12:58 node1 harc[7980]: [7987]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:12:58 node1 mach_down[7992]: [8012]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 11:12:58 node1 mach_down[7992]: [8017]: info: mach_down takeover complete for node node2.
Sep 14 11:12:58 node1 heartbeat: [7357]: info: mach_down takeover complete.
Sep 14 11:12:58 node1 heartbeat: [7357]: info: Heartbeat shutdown in progress. (7357)
Sep 14 11:12:58 node1 heartbeat: [8018]: info: Giving up all HA resources.
Sep 14 11:12:58 node1 ResourceManager[8032]: [8043]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:12:58 node1 ResourceManager[8032]: [8054]: info: Running /etc/init.d/apache2  stop
Sep 14 11:13:00 node1 ResourceManager[8032]: [8087]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 11:13:00 node1 drbdlinks[8088]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:13:00 node1 drbdlinks[8088]: Exiting with no errors
Sep 14 11:13:00 node1 ResourceManager[8032]: [8103]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 11:13:00 node1 Filesystem[8110]: [8138]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 11:13:00 node1 Filesystem[8110]: [8153]: INFO: Trying to unmount /cluster
Sep 14 11:13:00 node1 Filesystem[8110]: [8162]: INFO: unmounted /cluster successfully
Sep 14 11:13:00 node1 Filesystem[8104]: [8169]: INFO:  Success
Sep 14 11:13:00 node1 ResourceManager[8032]: [8184]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 11:13:00 node1 kernel: [ 4556.758351] block drbd0: role( Primary -> Secondary )
Sep 14 11:13:00 node1 kernel: [ 4556.758362] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 11:13:00 node1 kernel: [ 4556.758997] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 11:13:00 node1 ResourceManager[8032]: [8209]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 11:13:00 node1 IPaddr[8234]: [8245]: INFO: ifconfig eth0:0 down
Sep 14 11:13:00 node1 IPaddr[8210]: [8249]: INFO:  Success
Sep 14 11:13:00 node1 heartbeat: [8018]: info: All HA resources relinquished.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: killing HBFIFO process 7361 with signal 15
Sep 14 11:13:02 node1 heartbeat: [7357]: info: killing HBWRITE process 7362 with signal 15
Sep 14 11:13:02 node1 heartbeat: [7357]: info: killing HBREAD process 7363 with signal 15
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Core process 7362 exited. 3 remaining
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Core process 7361 exited. 2 remaining
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Core process 7363 exited. 1 remaining
Sep 14 11:13:02 node1 heartbeat: [7357]: info: node1 Heartbeat shutdown complete.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Heartbeat restart triggered.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Restarting heartbeat.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Performing heartbeat restart exec.
Sep 14 11:13:08 node1 heartbeat: [7357]: info: No log entry found in ha.cf -- use logd
Sep 14 11:13:08 node1 heartbeat: [7357]: info: Enabling logging daemon
Sep 14 11:13:08 node1 heartbeat: [7357]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 11:13:08 node1 heartbeat: [7357]: info: **************************
Sep 14 11:13:08 node1 heartbeat: [7357]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 11:13:08 node1 heartbeat: [8250]: info: heartbeat: version 3.0.5
Sep 14 11:13:08 node1 heartbeat: [8250]: info: Heartbeat generation: 1442218518
Sep 14 11:13:08 node1 heartbeat: [8250]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 11:13:08 node1 heartbeat: [8250]: info: Local status now set to: 'up'
Sep 14 11:13:08 node1 heartbeat: [8250]: info: Link node2:eth0 up.
Sep 14 11:13:08 node1 heartbeat: [8250]: debug: get_delnodelist: delnodelist=
Sep 14 11:13:09 node1 heartbeat: [8250]: info: Status update for node node2: status active
Sep 14 11:13:09 node1 heartbeat: [8250]: info: Comm_now_up(): updating status to active
Sep 14 11:13:09 node1 heartbeat: [8250]: info: Local status now set to: 'active'
Sep 14 11:13:09 node1 heartbeat: [8258]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:13:09 node1 harc[8258]: [8266]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:13:19 node1 heartbeat: [8250]: info: local resource transition completed.
Sep 14 11:13:19 node1 heartbeat: [8250]: info: Initial resource acquisition complete (T_RESOURCES(us))
Sep 14 11:13:19 node1 IPaddr[8306]: [8338]: INFO:  Resource is stopped
Sep 14 11:13:19 node1 heartbeat: [8271]: info: Local Resource acquisition completed.
Sep 14 11:13:19 node1 heartbeat: [8250]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:13:19 node1 heartbeat: [8342]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:13:19 node1 harc[8342]: [8348]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 11:13:19 node1 ip-request-resp[8342]: [8354]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 11:13:19 node1 ResourceManager[8355]: [8366]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:13:19 node1 IPaddr[8378]: [8410]: INFO:  Resource is stopped
Sep 14 11:13:19 node1 ResourceManager[8355]: [8425]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 11:13:19 node1 IPaddr[8450]: [8478]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 11:13:19 node1 IPaddr[8450]: [8500]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 11:13:19 node1 IPaddr[8426]: [8519]: INFO:  Success
Sep 14 11:13:19 node1 ResourceManager[8355]: [8549]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 11:13:19 node1 kernel: [ 4576.447471] block drbd0: role( Secondary -> Primary )
Sep 14 11:13:19 node1 Filesystem[8566]: [8610]: INFO:  Resource is stopped
Sep 14 11:13:19 node1 ResourceManager[8355]: [8625]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 11:13:19 node1 Filesystem[8632]: [8660]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 11:13:19 node1 kernel: [ 4576.519118] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 11:13:19 node1 Filesystem[8626]: [8680]: INFO:  Success
Sep 14 11:13:19 node1 drbdlinks[8687]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:13:19 node1 drbdlinks[8687]: Status mode returning stopped
Sep 14 11:13:19 node1 ResourceManager[8355]: [8697]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 11:13:19 node1 drbdlinks[8698]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:13:19 node1 drbdlinks[8698]: Exiting with no errors
Sep 14 11:13:19 node1 ResourceManager[8355]: [8719]: info: Running /etc/init.d/apache2  start
Sep 14 11:13:19 node1 heartbeat: [8250]: info: remote resource transition completed.
Sep 14 11:17:01 node1 /USR/SBIN/CRON[8741]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 14 11:32:05 node1 heartbeat: [8250]: info: Heartbeat shutdown in progress. (8250)
Sep 14 11:32:05 node1 heartbeat: [8776]: info: Giving up all HA resources.
Sep 14 11:32:05 node1 ResourceManager[8790]: [8801]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 11:32:05 node1 ResourceManager[8790]: [8812]: info: Running /etc/init.d/mysql  stop
Sep 14 11:32:05 node1 ResourceManager[8790]: [8850]: info: Running /etc/init.d/apache2  stop
Sep 14 11:32:06 node1 ResourceManager[8790]: [8883]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 11:32:06 node1 drbdlinks[8884]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:32:06 node1 drbdlinks[8884]: Exiting with no errors
Sep 14 11:32:06 node1 ResourceManager[8790]: [8899]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 11:32:06 node1 Filesystem[8906]: [8934]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 11:32:06 node1 Filesystem[8906]: [8949]: INFO: Trying to unmount /cluster
Sep 14 11:32:06 node1 Filesystem[8906]: [8958]: INFO: unmounted /cluster successfully
Sep 14 11:32:06 node1 Filesystem[8900]: [8965]: INFO:  Success
Sep 14 11:32:06 node1 ResourceManager[8790]: [8980]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 11:32:06 node1 kernel: [ 5703.480968] block drbd0: role( Primary -> Secondary )
Sep 14 11:32:06 node1 kernel: [ 5703.480978] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 11:32:06 node1 kernel: [ 5703.481631] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 11:32:06 node1 ResourceManager[8790]: [9005]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 11:32:06 node1 IPaddr[9030]: [9041]: INFO: ifconfig eth0:0 down
Sep 14 11:32:06 node1 IPaddr[9006]: [9045]: INFO:  Success
Sep 14 11:32:06 node1 heartbeat: [8776]: info: All HA resources relinquished.
Sep 14 11:32:07 node1 kernel: [ 5703.819546] block drbd0: peer( Secondary -> Primary )
Sep 14 11:32:07 node1 heartbeat: [8250]: WARN: 1 lost packet(s) for [node2] [583:585]
Sep 14 11:32:07 node1 heartbeat: [8250]: info: No pkts missing from node2!
Sep 14 11:32:08 node1 heartbeat: [8250]: info: killing HBFIFO process 8253 with signal 15
Sep 14 11:32:08 node1 heartbeat: [8250]: info: killing HBWRITE process 8254 with signal 15
Sep 14 11:32:08 node1 heartbeat: [8250]: info: killing HBREAD process 8255 with signal 15
Sep 14 11:32:08 node1 heartbeat: [8250]: info: Core process 8254 exited. 3 remaining
Sep 14 11:32:08 node1 heartbeat: [8250]: info: Core process 8253 exited. 2 remaining
Sep 14 11:32:08 node1 heartbeat: [8250]: info: Core process 8255 exited. 1 remaining
Sep 14 11:32:08 node1 heartbeat: [8250]: info: node1 Heartbeat shutdown complete.
Sep 14 11:32:22 node1 kernel: [ 5719.439494] block drbd0: peer( Primary -> Secondary )
Sep 14 11:34:02 node1 heartbeat: [9150]: info: No log entry found in ha.cf -- use logd
Sep 14 11:34:02 node1 heartbeat: [9150]: info: Enabling logging daemon
Sep 14 11:34:02 node1 heartbeat: [9150]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 11:34:02 node1 heartbeat: [9150]: info: **************************
Sep 14 11:34:02 node1 heartbeat: [9150]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 11:34:02 node1 heartbeat: [9151]: info: heartbeat: version 3.0.5
Sep 14 11:34:02 node1 heartbeat: [9151]: info: Heartbeat generation: 1442218519
Sep 14 11:34:02 node1 heartbeat: [9151]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 11:34:02 node1 heartbeat: [9151]: info: Local status now set to: 'up'
Sep 14 11:34:03 node1 heartbeat: [9151]: info: Link node2:eth0 up.
Sep 14 11:34:03 node1 heartbeat: [9151]: info: Comm_now_up(): updating status to active
Sep 14 11:34:03 node1 heartbeat: [9151]: info: Local status now set to: 'active'
Sep 14 11:34:04 node1 heartbeat: [9151]: info: remote resource transition completed.
Sep 14 11:34:04 node1 heartbeat: [9151]: info: remote resource transition completed.
Sep 14 11:34:04 node1 heartbeat: [9151]: info: Local Resource acquisition completed. (none)
Sep 14 11:34:04 node1 heartbeat: [9151]: info: Initial resource acquisition complete (T_RESOURCES(them))
Sep 14 11:34:04 node1 heartbeat: [9151]: info: Status update for node node2: status active
Sep 14 11:34:04 node1 heartbeat: [9162]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:04 node1 harc[9162]: [9168]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:13 node1 heartbeat: [9151]: info: Received shutdown notice from 'node2'.
Sep 14 11:34:13 node1 heartbeat: [9151]: info: Resources being acquired from node2.
Sep 14 11:34:13 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:34:13 node1 heartbeat: [9173]: info: acquire all HA resources (standby).
Sep 14 11:34:13 node1 ResourceManager[9202]: [9221]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:34:13 node1 IPaddr[9246]: [9309]: INFO:  Resource is stopped
Sep 14 11:34:13 node1 IPaddr[9245]: [9310]: INFO:  Resource is stopped
Sep 14 11:34:13 node1 heartbeat: [9174]: info: Local Resource acquisition completed.
Sep 14 11:34:13 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 2
Sep 14 11:34:13 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:34:13 node1 ResourceManager[9202]: [9328]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 11:34:13 node1 IPaddr[9353]: [9381]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 11:34:13 node1 IPaddr[9353]: [9403]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 11:34:13 node1 IPaddr[9329]: [9422]: INFO:  Success
Sep 14 11:34:13 node1 ResourceManager[9202]: [9452]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 11:34:13 node1 kernel: [ 5830.121541] block drbd0: role( Secondary -> Primary )
Sep 14 11:34:13 node1 Filesystem[9469]: [9513]: INFO:  Resource is stopped
Sep 14 11:34:13 node1 ResourceManager[9202]: [9528]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 11:34:13 node1 Filesystem[9535]: [9563]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 11:34:13 node1 Filesystem[9529]: [9583]: INFO:  Success
Sep 14 11:34:13 node1 kernel: [ 5830.189046] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 11:34:13 node1 drbdlinks[9590]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:34:13 node1 drbdlinks[9590]: Status mode returning stopped
Sep 14 11:34:13 node1 ResourceManager[9202]: [9600]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 11:34:13 node1 drbdlinks[9601]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:34:13 node1 drbdlinks[9601]: Exiting with no errors
Sep 14 11:34:13 node1 ResourceManager[9202]: [9622]: info: Running /etc/init.d/apache2  start
Sep 14 11:34:13 node1 heartbeat: [9173]: info: all HA resource acquisition completed (standby).
Sep 14 11:34:13 node1 heartbeat: [9151]: info: Standby resource acquisition done [all].
Sep 14 11:34:13 node1 heartbeat: [9636]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:13 node1 harc[9636]: [9642]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:13 node1 mach_down[9652]: [9672]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 11:34:13 node1 mach_down[9652]: [9677]: info: mach_down takeover complete for node node2.
Sep 14 11:34:13 node1 heartbeat: [9151]: info: mach_down takeover complete.
Sep 14 11:34:13 node1 heartbeat: [9678]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:13 node1 harc[9678]: [9684]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 11:34:13 node1 ip-request-resp[9678]: [9690]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 11:34:13 node1 ResourceManager[9691]: [9702]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:34:13 node1 IPaddr[9714]: [9752]: INFO:  Running OK
Sep 14 11:34:13 node1 Filesystem[9779]: [9818]: INFO:  Running OK
Sep 14 11:34:13 node1 drbdlinks[9825]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:34:13 node1 drbdlinks[9825]: Status mode returning ok
Sep 14 11:34:19 node1 heartbeat: [9151]: WARN: node node2: is dead
Sep 14 11:34:19 node1 heartbeat: [9151]: info: Dead node node2 gave up resources.
Sep 14 11:34:19 node1 heartbeat: [9151]: info: Link node2:eth0 dead.
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Heartbeat restart on node node2
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Link node2:eth0 up.
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Status update for node node2: status init
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Status update for node node2: status up
Sep 14 11:34:30 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:34:30 node1 heartbeat: [9841]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:30 node1 harc[9841]: [9847]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:30 node1 heartbeat: [9852]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:30 node1 harc[9852]: [9858]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:31 node1 heartbeat: [9151]: debug: get_delnodelist: delnodelist=
Sep 14 11:34:31 node1 heartbeat: [9151]: info: Status update for node node2: status active
Sep 14 11:34:31 node1 heartbeat: [9863]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:31 node1 harc[9863]: [9869]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:32 node1 heartbeat: [9151]: info: remote resource transition completed.