Debian-facile

Bienvenue sur Debian-Facile, site d'aide pour les nouveaux utilisateurs de Debian.

Vous n'êtes pas identifié(e).

#1 14-09-2015 10:55:22

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

cluster 2 noeuds actif/passif la ressource mysql plante

Bonjour @ tous,

je suis en train de faire un petit labo en vue de mettre cela en prod

- j'ai installé les paquets suivants vim apache2 php5 mysql-server drbd8-utils drbdlinks heartbeat et owncloud
- la configuration drbd est ok....

- premiers tests de basculement avect heartbeat :

- l'ip virtuelle bascule correctement
- si j'ajoute drbd cela fonctionne aussi
- si j'ajoute drbdlinks aussi
- si j'ajoute apache aussi

- j'ajoute la resource mysql et cela part en live
- plus d'ip virtuelle et drbd passe de primary/secondary en secondary/secondary

j'a copier /var/lib/mysql sur le share drbd ainsi que /etc/mysql.... (les liens symboliques sont gérés par drbdlinks)

est ce que quelqu'un a une idée pourquoi mysql fait tout planter

merci

Dernière modification par thomas (14-09-2015 10:56:28)

Hors ligne

#2 14-09-2015 11:01:53

milou
Modo ... e
Lieu : Sur une autre planète....
Distrib. : Jessie - Stretch/Sid
Noyau : 3.16.0-4-amd64
(G)UI : Lxde
Inscription : 12-02-2015
Site Web

Re : cluster 2 noeuds actif/passif la ressource mysql plante

Bonjour thomas et bienvenue smile

Je déplace la discussion dans le forum réseau qui est plus adapté à ton problème.
Tu auras plus de chances d'obtenir des réponses.

Afin que tu puisses obtenir de l'aide plus facilement,  peux-tu mettre à jour les infos sur ton système
Voir le tuto : Trop cool d'indiquer son installation dans son profil !  wink

J’adorerais changer le monde, mais ils ne veulent pas me fournir le code source
Un vrai geek, c'est un mec qui croit que dans 1km, il y a 1024 mètres
Dans le doute, rebootes. Si tu te tâtes, formates.
1453651422.jpg

Hors ligne

#3 14-09-2015 11:18:12

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

Merci,
l'update du profil vient d'être faites smile

Hors ligne

#4 14-09-2015 11:19:16

milou
Modo ... e
Lieu : Sur une autre planète....
Distrib. : Jessie - Stretch/Sid
Noyau : 3.16.0-4-amd64
(G)UI : Lxde
Inscription : 12-02-2015
Site Web

Re : cluster 2 noeuds actif/passif la ressource mysql plante

cool    smile

J’adorerais changer le monde, mais ils ne veulent pas me fournir le code source
Un vrai geek, c'est un mec qui croit que dans 1km, il y a 1024 mètres
Dans le doute, rebootes. Si tu te tâtes, formates.
1453651422.jpg

Hors ligne

#5 14-09-2015 12:20:30

unaM
Membre
Distrib. : Debian Stretch
Noyau : Linux 4.5.0-2-amd64
(G)UI : Gnome 3.20.2
Inscription : 06-01-2012

Re : cluster 2 noeuds actif/passif la ressource mysql plante

Salut, aurais tu des logs générés suite au plantage ?

Que se passe il si tu ajoute un autre processus/service dans la ha ? Par exemple si tu mets apache au lieu de mysql, le cluster tombe il ?

Hors ligne

#6 14-09-2015 13:00:54

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

actuellement il tourne parfaitement avec les ressources suivantes cumulées

contenu du fichier /etc/ha.d/haresources

node1 IPaddr::192.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2



si j'ajoute mysql baboum

node1 IPaddr::192.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql



si je le retire => reprise à la normale

node1 IPaddr::192.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2



Sep 14 10:15:14 node1 heartbeat: [3275]: info: No log entry found in ha.cf -- use logd
Sep 14 10:15:14 node1 heartbeat: [3275]: info: Enabling logging daemon
Sep 14 10:15:14 node1 heartbeat: [3275]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 10:15:14 node1 heartbeat: [3275]: info: **************************
Sep 14 10:15:14 node1 heartbeat: [3275]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 10:15:14 node1 heartbeat: [3276]: info: heartbeat: version 3.0.5
Sep 14 10:15:14 node1 heartbeat: [3276]: WARN: No Previous generation - starting at 1442218515
Sep 14 10:15:14 node1 heartbeat: [3276]: info: Heartbeat generation: 1442218515
Sep 14 10:15:14 node1 heartbeat: [3276]: info: No uuid found for current node - generating a new uuid.
Sep 14 10:15:14 node1 heartbeat: [3276]: info: Creating FIFO /var/lib/heartbeat/fifo.
Sep 14 10:15:14 node1 heartbeat: [3276]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 10:15:14 node1 heartbeat: [3276]: info: Local status now set to: 'up'
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Link node2:eth0 up.
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Status update for node node2: status up
Sep 14 10:15:26 node1 heartbeat: [3286]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:15:26 node1 harc[3286]: [3293]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Comm_now_up(): updating status to active
Sep 14 10:15:26 node1 heartbeat: [3276]: info: Local status now set to: 'active'
Sep 14 10:15:26 node1 heartbeat: [3276]: debug: get_delnodelist: delnodelist=
Sep 14 10:15:27 node1 heartbeat: [3276]: info: Status update for node node2: status active
Sep 14 10:15:27 node1 heartbeat: [3300]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:15:27 node1 harc[3300]: [3306]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:15:37 node1 heartbeat: [3276]: info: remote resource transition completed.
Sep 14 10:15:37 node1 heartbeat: [3276]: info: remote resource transition completed.
Sep 14 10:15:37 node1 heartbeat: [3276]: info: Initial resource acquisition complete (T_RESOURCES(us))
Sep 14 10:15:37 node1 IPaddr[3346]: [3378]: INFO:  Resource is stopped
Sep 14 10:15:37 node1 heartbeat: [3311]: info: Local Resource acquisition completed.
Sep 14 10:15:37 node1 heartbeat: [3276]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:15:37 node1 heartbeat: [3382]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:15:37 node1 harc[3382]: [3388]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 10:15:37 node1 ip-request-resp[3382]: [3394]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 10:15:37 node1 ResourceManager[3395]: [3406]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 10:15:37 node1 IPaddr[3418]: [3450]: INFO:  Resource is stopped
Sep 14 10:15:37 node1 ResourceManager[3395]: [3465]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:15:37 node1 IPaddr[3490]: [3518]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:15:37 node1 IPaddr[3490]: [3540]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:15:37 node1 IPaddr[3466]: [3559]: INFO:  Success
Sep 14 10:15:37 node1 Filesystem[3586]: [3625]: INFO:  Running OK
Sep 14 10:15:37 node1 drbdlinks[3632]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:15:37 node1 drbdlinks[3632]: Status mode returning stopped
Sep 14 10:15:37 node1 ResourceManager[3395]: [3642]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 10:15:37 node1 drbdlinks[3643]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:15:37 node1 drbdlinks[3643]: Exiting with no errors
Sep 14 10:15:37 node1 ResourceManager[3395]: [3664]: info: Running /etc/init.d/apache2  start
Sep 14 10:17:01 node1 /USR/SBIN/CRON[3696]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 14 10:19:17 node1 heartbeat: [3276]: info: Heartbeat shutdown in progress. (3276)
Sep 14 10:19:17 node1 heartbeat: [3715]: info: Giving up all HA resources.
Sep 14 10:19:17 node1 ResourceManager[3729]: [3740]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:19:17 node1 ResourceManager[3729]: [3751]: info: Running /etc/init.d/mysql  stop
Sep 14 10:19:17 node1 ResourceManager[3729]: [3789]: info: Running /etc/init.d/apache2  stop
Sep 14 10:19:18 node1 ResourceManager[3729]: [3822]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:19:18 node1 drbdlinks[3823]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:19:18 node1 drbdlinks[3823]: Exiting with no errors
Sep 14 10:19:18 node1 ResourceManager[3729]: [3838]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:19:18 node1 Filesystem[3845]: [3873]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:19:18 node1 Filesystem[3845]: [3888]: INFO: Trying to unmount /cluster
Sep 14 10:19:18 node1 Filesystem[3845]: [3896]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:18 node1 Filesystem[3845]: [3899]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:19 node1 Filesystem[3845]: [3908]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:19 node1 Filesystem[3845]: [3911]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:20 node1 Filesystem[3845]: [3920]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:20 node1 Filesystem[3845]: [3923]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:21 node1 Filesystem[3845]: [3932]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:21 node1 Filesystem[3845]: [3935]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:22 node1 Filesystem[3845]: [3944]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:22 node1 Filesystem[3845]: [3947]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:23 node1 Filesystem[3845]: [3956]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:24 node1 Filesystem[3845]: [3959]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:25 node1 Filesystem[3845]: [3968]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:25 node1 Filesystem[3845]: [3971]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:26 node1 Filesystem[3845]: [3980]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:26 node1 Filesystem[3845]: [3983]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:27 node1 Filesystem[3845]: [3992]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:27 node1 Filesystem[3845]: [3995]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:28 node1 Filesystem[3845]: [4004]: ERROR: Couldn't unmount /cluster; trying cleanup with TERM
Sep 14 10:19:28 node1 Filesystem[3845]: [4007]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:29 node1 Filesystem[3845]: [4016]: ERROR: Couldn't unmount /cluster; trying cleanup with KILL
Sep 14 10:19:29 node1 Filesystem[3845]: [4019]: INFO: Some processes on /cluster were signalled
Sep 14 10:19:30 node1 Filesystem[3845]: [4028]: INFO: unmounted /cluster successfully
Sep 14 10:19:30 node1 Filesystem[3839]: [4035]: INFO:  Success
Sep 14 10:19:30 node1 ResourceManager[3729]: [4050]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:19:30 node1 kernel: [ 3532.543969] block drbd0: role( Primary -> Secondary )
Sep 14 10:19:30 node1 kernel: [ 3532.543979] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:19:30 node1 kernel: [ 3532.545998] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:19:30 node1 ResourceManager[3729]: [4075]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:19:30 node1 IPaddr[4100]: [4111]: INFO: ifconfig eth0:0 down
Sep 14 10:19:30 node1 IPaddr[4076]: [4115]: INFO:  Success
Sep 14 10:19:30 node1 heartbeat: [3715]: info: All HA resources relinquished.
Sep 14 10:19:30 node1 kernel: [ 3532.792118] block drbd0: peer( Secondary -> Primary )
Sep 14 10:19:30 node1 heartbeat: [3276]: WARN: 1 lost packet(s) for [node2] [134:136]
Sep 14 10:19:30 node1 heartbeat: [3276]: info: No pkts missing from node2!
Sep 14 10:19:32 node1 heartbeat: [3276]: info: killing HBREAD process 3283 with signal 15
Sep 14 10:19:32 node1 heartbeat: [3276]: info: killing HBFIFO process 3281 with signal 15
Sep 14 10:19:32 node1 heartbeat: [3276]: info: killing HBWRITE process 3282 with signal 15
Sep 14 10:19:32 node1 heartbeat: [3276]: info: Core process 3282 exited. 3 remaining
Sep 14 10:19:32 node1 heartbeat: [3276]: info: Core process 3281 exited. 2 remaining
Sep 14 10:19:32 node1 heartbeat: [3276]: info: Core process 3283 exited. 1 remaining
Sep 14 10:19:32 node1 heartbeat: [3276]: info: node1 Heartbeat shutdown complete.
Sep 14 10:19:46 node1 kernel: [ 3548.450222] block drbd0: peer( Primary -> Secondary )
Sep 14 10:21:06 node1 heartbeat: [4267]: info: No log entry found in ha.cf -- use logd
Sep 14 10:21:06 node1 heartbeat: [4267]: info: Enabling logging daemon
Sep 14 10:21:06 node1 heartbeat: [4267]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 10:21:06 node1 heartbeat: [4267]: info: **************************
Sep 14 10:21:06 node1 heartbeat: [4267]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 10:21:06 node1 heartbeat: [4268]: info: heartbeat: version 3.0.5
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Heartbeat generation: 1442218516
Sep 14 10:21:06 node1 heartbeat: [4268]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Local status now set to: 'up'
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Link node2:eth0 up.
Sep 14 10:21:06 node1 heartbeat: [4268]: info: Status update for node node2: status active
Sep 14 10:21:06 node1 heartbeat: [4276]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:21:06 node1 harc[4276]: [4283]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Comm_now_up(): updating status to active
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Local status now set to: 'active'
Sep 14 10:21:07 node1 heartbeat: [4268]: info: remote resource transition completed.
Sep 14 10:21:07 node1 heartbeat: [4268]: info: remote resource transition completed.
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Local Resource acquisition completed. (none)
Sep 14 10:21:07 node1 heartbeat: [4268]: info: Initial resource acquisition complete (T_RESOURCES(them))
Sep 14 10:21:34 node1 heartbeat: [4268]: info: Received shutdown notice from 'node2'.
Sep 14 10:21:34 node1 heartbeat: [4268]: info: Resources being acquired from node2.
Sep 14 10:21:34 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:21:34 node1 heartbeat: [4292]: info: acquire all HA resources (standby).
Sep 14 10:21:34 node1 ResourceManager[4321]: [4340]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:34 node1 IPaddr[4365]: [4428]: INFO:  Resource is stopped
Sep 14 10:21:34 node1 IPaddr[4364]: [4429]: INFO:  Resource is stopped
Sep 14 10:21:34 node1 heartbeat: [4293]: info: Local Resource acquisition completed.
Sep 14 10:21:34 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 2
Sep 14 10:21:34 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:21:34 node1 ResourceManager[4321]: [4447]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:21:34 node1 IPaddr[4472]: [4500]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:21:34 node1 IPaddr[4472]: [4522]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:21:34 node1 IPaddr[4448]: [4541]: INFO:  Success
Sep 14 10:21:34 node1 ResourceManager[4321]: [4571]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 10:21:34 node1 kernel: [ 3657.051757] block drbd0: role( Secondary -> Primary )
Sep 14 10:21:34 node1 Filesystem[4588]: [4632]: INFO:  Resource is stopped
Sep 14 10:21:34 node1 ResourceManager[4321]: [4647]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 10:21:34 node1 Filesystem[4654]: [4682]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 10:21:34 node1 Filesystem[4648]: [4702]: INFO:  Success
Sep 14 10:21:34 node1 kernel: [ 3657.120578] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 10:21:34 node1 drbdlinks[4709]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:34 node1 drbdlinks[4709]: Status mode returning stopped
Sep 14 10:21:34 node1 ResourceManager[4321]: [4719]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 10:21:34 node1 drbdlinks[4720]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:34 node1 drbdlinks[4720]: Exiting with no errors
Sep 14 10:21:34 node1 ResourceManager[4321]: [4741]: info: Running /etc/init.d/apache2  start
Sep 14 10:21:34 node1 ResourceManager[4321]: [4792]: info: Running /etc/init.d/mysql  start
Sep 14 10:21:35 node1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0+deb7u1) starting as process 5149 ...
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [Note] Plugin 'FEDERATED' is disabled.
Sep 14 10:21:35 node1 mysqld: #007/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: The InnoDB memory heap is disabled
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Compressed tables use zlib 1.2.7
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Using Linux native AIO
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Initializing buffer pool, size = 128.0M
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35 InnoDB: Completed initialization of buffer pool
Sep 14 10:21:35 node1 mysqld: 150914 10:21:35  InnoDB: Operating system error number 13 in a file operation.
Sep 14 10:21:35 node1 mysqld: InnoDB: The error means mysqld does not have the access rights to
Sep 14 10:21:35 node1 mysqld: InnoDB: the directory.
Sep 14 10:21:35 node1 mysqld: InnoDB: File name ./ibdata1
Sep 14 10:21:35 node1 mysqld: InnoDB: File operation call: 'create'.
Sep 14 10:21:35 node1 mysqld: InnoDB: Cannot continue operation.
Sep 14 10:21:35 node1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Sep 14 10:21:40 node1 heartbeat: [4268]: WARN: node node2: is dead
Sep 14 10:21:40 node1 heartbeat: [4268]: info: Cancelling pending standby operation
Sep 14 10:21:40 node1 heartbeat: [4268]: info: Dead node node2 gave up resources.
Sep 14 10:21:41 node1 heartbeat: [4268]: info: Link node2:eth0 dead.
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]:
Sep 14 10:21:49 node1 ResourceManager[4321]: [5346]: ERROR: Return code 1 from /etc/init.d/mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5348]: CRIT: Giving up resources due to failure of mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5350]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5361]: info: Running /etc/init.d/mysql  stop
Sep 14 10:21:49 node1 ResourceManager[4321]: [5399]: info: Running /etc/init.d/apache2  stop
Sep 14 10:21:50 node1 ResourceManager[4321]: [5432]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:21:50 node1 drbdlinks[5433]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:50 node1 drbdlinks[5433]: Exiting with no errors
Sep 14 10:21:50 node1 ResourceManager[4321]: [5448]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:21:50 node1 Filesystem[5455]: [5483]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:21:50 node1 Filesystem[5455]: [5498]: INFO: Trying to unmount /cluster
Sep 14 10:21:50 node1 Filesystem[5455]: [5506]: INFO: unmounted /cluster successfully
Sep 14 10:21:50 node1 Filesystem[5449]: [5513]: INFO:  Success
Sep 14 10:21:50 node1 ResourceManager[4321]: [5528]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:21:50 node1 kernel: [ 3672.661265] block drbd0: role( Primary -> Secondary )
Sep 14 10:21:50 node1 kernel: [ 3672.661273] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:21:50 node1 kernel: [ 3672.661831] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:21:50 node1 ResourceManager[4321]: [5553]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:21:50 node1 IPaddr[5578]: [5589]: INFO: ifconfig eth0:0 down
Sep 14 10:21:50 node1 IPaddr[5554]: [5593]: INFO:  Success
Sep 14 10:21:50 node1 heartbeat: [4292]: info: all HA resource acquisition completed (standby).
Sep 14 10:21:50 node1 heartbeat: [4268]: ERROR: Ignored standby message 'done' from node1 in state 0
Sep 14 10:21:50 node1 heartbeat: [5595]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:21:50 node1 harc[5595]: [5602]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:21:50 node1 mach_down[5607]: [5627]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 10:21:50 node1 mach_down[5607]: [5632]: info: mach_down takeover complete for node node2.
Sep 14 10:21:50 node1 heartbeat: [4268]: info: mach_down takeover complete.
Sep 14 10:21:50 node1 heartbeat: [5633]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:21:50 node1 harc[5633]: [5639]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 10:21:50 node1 ip-request-resp[5633]: [5645]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 10:21:50 node1 ResourceManager[5646]: [5657]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:50 node1 IPaddr[5669]: [5701]: INFO:  Resource is stopped
Sep 14 10:21:50 node1 ResourceManager[5646]: [5716]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:21:50 node1 IPaddr[5741]: [5769]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:21:50 node1 IPaddr[5741]: [5791]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:21:50 node1 IPaddr[5717]: [5810]: INFO:  Success
Sep 14 10:21:50 node1 ResourceManager[5646]: [5840]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 10:21:50 node1 kernel: [ 3672.795117] block drbd0: role( Secondary -> Primary )
Sep 14 10:21:50 node1 Filesystem[5857]: [5901]: INFO:  Resource is stopped
Sep 14 10:21:50 node1 ResourceManager[5646]: [5916]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 10:21:50 node1 Filesystem[5923]: [5951]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 10:21:50 node1 kernel: [ 3672.882601] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 10:21:50 node1 Filesystem[5917]: [5971]: INFO:  Success
Sep 14 10:21:50 node1 drbdlinks[5978]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:50 node1 drbdlinks[5978]: Status mode returning stopped
Sep 14 10:21:50 node1 ResourceManager[5646]: [5988]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 10:21:50 node1 drbdlinks[5989]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:21:50 node1 drbdlinks[5989]: Exiting with no errors
Sep 14 10:21:50 node1 ResourceManager[5646]: [6010]: info: Running /etc/init.d/apache2  start
Sep 14 10:21:50 node1 ResourceManager[5646]: [6061]: info: Running /etc/init.d/mysql  start
Sep 14 10:21:50 node1 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0+deb7u1) starting as process 6417 ...
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [Note] Plugin 'FEDERATED' is disabled.
Sep 14 10:21:50 node1 mysqld: #007/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: The InnoDB memory heap is disabled
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Compressed tables use zlib 1.2.7
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Using Linux native AIO
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Initializing buffer pool, size = 128.0M
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50 InnoDB: Completed initialization of buffer pool
Sep 14 10:21:50 node1 mysqld: 150914 10:21:50  InnoDB: Operating system error number 13 in a file operation.
Sep 14 10:21:50 node1 mysqld: InnoDB: The error means mysqld does not have the access rights to
Sep 14 10:21:50 node1 mysqld: InnoDB: the directory.
Sep 14 10:21:50 node1 mysqld: InnoDB: File name ./ibdata1
Sep 14 10:21:50 node1 mysqld: InnoDB: File operation call: 'create'.
Sep 14 10:21:50 node1 mysqld: InnoDB: Cannot continue operation.
Sep 14 10:21:50 node1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Sep 14 10:22:04 node1 /etc/init.d/mysql[6635]:
Sep 14 10:22:04 node1 ResourceManager[5646]: [6638]: ERROR: Return code 1 from /etc/init.d/mysql
Sep 14 10:22:04 node1 ResourceManager[5646]: [6640]: CRIT: Giving up resources due to failure of mysql
Sep 14 10:22:04 node1 ResourceManager[5646]: [6642]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:22:04 node1 ResourceManager[5646]: [6653]: info: Running /etc/init.d/mysql  stop
Sep 14 10:22:04 node1 ResourceManager[5646]: [6691]: info: Running /etc/init.d/apache2  stop
Sep 14 10:22:06 node1 ResourceManager[5646]: [6724]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:22:06 node1 drbdlinks[6725]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:22:06 node1 drbdlinks[6725]: Exiting with no errors
Sep 14 10:22:06 node1 ResourceManager[5646]: [6740]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:22:06 node1 Filesystem[6747]: [6775]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:22:06 node1 Filesystem[6747]: [6790]: INFO: Trying to unmount /cluster
Sep 14 10:22:06 node1 Filesystem[6747]: [6798]: INFO: unmounted /cluster successfully
Sep 14 10:22:06 node1 Filesystem[6741]: [6805]: INFO:  Success
Sep 14 10:22:06 node1 ResourceManager[5646]: [6820]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:22:06 node1 kernel: [ 3688.435641] block drbd0: role( Primary -> Secondary )
Sep 14 10:22:06 node1 kernel: [ 3688.435651] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:22:06 node1 kernel: [ 3688.436451] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:22:06 node1 ResourceManager[5646]: [6845]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:22:06 node1 IPaddr[6870]: [6881]: INFO: ifconfig eth0:0 down
Sep 14 10:22:06 node1 IPaddr[6846]: [6885]: INFO:  Success
Sep 14 10:22:20 node1 hb_standby[5594]: [6903]: Going standby [foreign].
Sep 14 10:22:20 node1 heartbeat: [4268]: info: node1 wants to go standby [foreign]
Sep 14 10:22:32 node1 heartbeat: [4268]: WARN: No reply to standby request.  Standby request cancelled.
Sep 14 10:22:36 node1 hb_standby[6886]: [6921]: Going standby [foreign].
Sep 14 10:22:36 node1 heartbeat: [4268]: info: node1 wants to go standby [foreign]
Sep 14 10:22:48 node1 heartbeat: [4268]: WARN: No reply to standby request.  Standby request cancelled.
Sep 14 10:24:23 node1 kernel: [ 3826.174202] block drbd0: role( Secondary -> Primary )
Sep 14 10:24:38 node1 kernel: [ 3840.546456] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Heartbeat restart on node node2
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Link node2:eth0 up.
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Status update for node node2: status init
Sep 14 10:24:51 node1 heartbeat: [4268]: info: Status update for node node2: status up
Sep 14 10:24:51 node1 heartbeat: [4268]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:24:51 node1 heartbeat: [6988]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:24:51 node1 harc[6988]: [6994]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:24:51 node1 heartbeat: [6999]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:24:51 node1 harc[6999]: [7005]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:24:52 node1 heartbeat: [4268]: debug: get_delnodelist: delnodelist=
Sep 14 10:24:53 node1 heartbeat: [4268]: info: Status update for node node2: status active
Sep 14 10:24:53 node1 heartbeat: [7010]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:24:53 node1 harc[7010]: [7016]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:24:53 node1 heartbeat: [4268]: info: remote resource transition completed.
Sep 14 10:25:06 node1 heartbeat: [4268]: info: Heartbeat shutdown in progress. (4268)
Sep 14 10:25:06 node1 heartbeat: [7036]: info: Giving up all HA resources.
Sep 14 10:25:06 node1 ResourceManager[7050]: [7061]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 10:25:06 node1 ResourceManager[7050]: [7072]: info: Running /etc/init.d/apache2  stop
Sep 14 10:25:06 node1 ResourceManager[7050]: [7100]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:25:06 node1 drbdlinks[7101]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 10:25:06 node1 drbdlinks[7101]: Exiting with no errors
Sep 14 10:25:06 node1 ResourceManager[7050]: [7116]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 10:25:06 node1 Filesystem[7123]: [7151]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 10:25:06 node1 Filesystem[7123]: [7166]: INFO: Trying to unmount /cluster
Sep 14 10:25:06 node1 Filesystem[7123]: [7174]: INFO: unmounted /cluster successfully
Sep 14 10:25:06 node1 Filesystem[7117]: [7181]: INFO:  Success
Sep 14 10:25:06 node1 ResourceManager[7050]: [7196]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 10:25:06 node1 kernel: [ 3868.984325] block drbd0: role( Primary -> Secondary )
Sep 14 10:25:06 node1 kernel: [ 3868.984335] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 10:25:06 node1 kernel: [ 3868.984975] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 10:25:06 node1 ResourceManager[7050]: [7221]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 10:25:06 node1 IPaddr[7222]: [7254]: INFO:  Success
Sep 14 10:25:06 node1 heartbeat: [7036]: info: All HA resources relinquished.
Sep 14 10:25:07 node1 kernel: [ 3869.439181] block drbd0: peer( Secondary -> Primary )
Sep 14 10:25:07 node1 heartbeat: [4268]: WARN: 1 lost packet(s) for [node2] [18:20]
Sep 14 10:25:07 node1 heartbeat: [4268]: info: No pkts missing from node2!
Sep 14 10:25:08 node1 heartbeat: [4268]: info: killing HBFIFO process 4272 with signal 15
Sep 14 10:25:08 node1 heartbeat: [4268]: info: killing HBWRITE process 4273 with signal 15
Sep 14 10:25:08 node1 heartbeat: [4268]: info: killing HBREAD process 4274 with signal 15
Sep 14 10:25:08 node1 heartbeat: [4268]: info: Core process 4273 exited. 3 remaining
Sep 14 10:25:08 node1 heartbeat: [4268]: info: Core process 4272 exited. 2 remaining
Sep 14 10:25:08 node1 heartbeat: [4268]: info: Core process 4274 exited. 1 remaining
Sep 14 10:25:08 node1 heartbeat: [4268]: info: node1 Heartbeat shutdown complete.
Sep 14 10:25:48 node1 heartbeat: [7356]: info: No log entry found in ha.cf -- use logd
Sep 14 10:25:48 node1 heartbeat: [7356]: info: Enabling logging daemon
Sep 14 10:25:48 node1 heartbeat: [7356]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 10:25:48 node1 heartbeat: [7356]: info: **************************
Sep 14 10:25:48 node1 heartbeat: [7356]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 10:25:48 node1 heartbeat: [7357]: info: heartbeat: version 3.0.5
Sep 14 10:25:48 node1 heartbeat: [7357]: info: Heartbeat generation: 1442218517
Sep 14 10:25:48 node1 heartbeat: [7357]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 10:25:48 node1 heartbeat: [7357]: info: Local status now set to: 'up'
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Link node2:eth0 up.
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Status update for node node2: status active
Sep 14 10:25:49 node1 heartbeat: [7365]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:25:49 node1 harc[7365]: [7372]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Comm_now_up(): updating status to active
Sep 14 10:25:49 node1 heartbeat: [7357]: info: Local status now set to: 'active'
Sep 14 10:25:50 node1 heartbeat: [7357]: info: remote resource transition completed.
Sep 14 10:25:50 node1 heartbeat: [7357]: info: remote resource transition completed.
Sep 14 10:25:50 node1 heartbeat: [7357]: info: Local Resource acquisition completed. (none)
Sep 14 10:25:50 node1 heartbeat: [7357]: info: Initial resource acquisition complete (T_RESOURCES(them))
Sep 14 10:36:22 node1 kernel: [ 4544.524360] e1000: eth0 NIC Link is Down
Sep 14 10:36:26 node1 heartbeat: [7357]: WARN: node node2: is dead
Sep 14 10:36:26 node1 heartbeat: [7357]: WARN: No STONITH device configured.
Sep 14 10:36:26 node1 heartbeat: [7357]: WARN: Shared disks are not protected.
Sep 14 10:36:26 node1 heartbeat: [7357]: info: Resources being acquired from node2.
Sep 14 10:36:26 node1 heartbeat: [7357]: info: Link node2:eth0 dead.
Sep 14 10:36:26 node1 heartbeat: [7379]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:36:26 node1 harc[7379]: [7392]: info: Running /etc/ha.d//rc.d/status status
Sep 14 10:36:26 node1 mach_down[7405]: [7442]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 10:36:26 node1 mach_down[7405]: [7452]: info: mach_down takeover complete for node node2.
Sep 14 10:36:26 node1 heartbeat: [7357]: info: mach_down takeover complete.
Sep 14 10:36:26 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:36:26 node1 IPaddr[7448]: [7483]: INFO:  Resource is stopped
Sep 14 10:36:26 node1 heartbeat: [7380]: info: Local Resource acquisition completed.
Sep 14 10:36:26 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 10:36:26 node1 heartbeat: [7487]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 10:36:26 node1 harc[7487]: [7493]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 10:36:26 node1 ip-request-resp[7487]: [7499]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 10:36:26 node1 ResourceManager[7500]: [7511]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 10:36:26 node1 IPaddr[7523]: [7555]: INFO:  Resource is stopped
Sep 14 10:36:26 node1 ResourceManager[7500]: [7570]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 10:36:26 node1 IPaddr[7595]: [7623]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 10:36:26 node1 IPaddr[7595]: [7645]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 10:36:26 node1 IPaddr[7571]: [7664]: INFO:  Success
Sep 14 10:36:26 node1 ResourceManager[7500]: [7694]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 10:36:28 node1 kernel: [ 4550.533774] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Sep 14 10:36:29 node1 heartbeat: [7357]: CRIT: Cluster node node2 returning after partition.
Sep 14 10:36:29 node1 heartbeat: [7357]: info: For information on cluster partitions, See URL: http://linux-ha.org/wiki/Split_Brain
Sep 14 10:36:29 node1 heartbeat: [7357]: WARN: Deadtime value may be too small.
Sep 14 10:36:29 node1 heartbeat: [7357]: info: See FAQ for information on tuning deadtime.
Sep 14 10:36:29 node1 heartbeat: [7357]: info: URL: http://linux-ha.org/wiki/FAQ#Heavy_Load
Sep 14 10:36:29 node1 heartbeat: [7357]: info: Link node2:eth0 up.
Sep 14 10:36:29 node1 heartbeat: [7357]: WARN: Late heartbeat: Node node2: interval 8010 ms
Sep 14 10:36:29 node1 heartbeat: [7357]: info: Status update for node node2: status active
Sep 14 10:36:29 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:12:57 node1 heartbeat: [7357]: WARN: Shutdown delayed until current resource activity finishes.
Sep 14 11:12:57 node1 kernel: [ 4554.433090] block drbd0: peer( Primary -> Secondary )
Sep 14 11:12:58 node1 heartbeat: [7357]: info: Received shutdown notice from 'node2'.
Sep 14 11:12:58 node1 heartbeat: [7357]: info: Resources being acquired from node2.
Sep 14 11:12:58 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:12:58 node1 IPaddr[7744]: [7782]: INFO:  Running OK
Sep 14 11:12:58 node1 heartbeat: [7709]: info: Local Resource acquisition completed.
Sep 14 11:12:58 node1 heartbeat: [7357]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:12:58 node1 kernel: [ 4555.399627] block drbd0: role( Secondary -> Primary )
Sep 14 11:12:58 node1 Filesystem[7798]: [7842]: INFO:  Resource is stopped
Sep 14 11:12:58 node1 ResourceManager[7500]: [7857]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 11:12:58 node1 Filesystem[7864]: [7892]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 11:12:58 node1 Filesystem[7858]: [7912]: INFO:  Success
Sep 14 11:12:58 node1 kernel: [ 4555.472030] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 11:12:58 node1 drbdlinks[7919]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:12:58 node1 drbdlinks[7919]: Status mode returning stopped
Sep 14 11:12:58 node1 ResourceManager[7500]: [7929]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 11:12:58 node1 drbdlinks[7930]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:12:58 node1 drbdlinks[7930]: Exiting with no errors
Sep 14 11:12:58 node1 ResourceManager[7500]: [7951]: info: Running /etc/init.d/apache2  start
Sep 14 11:12:58 node1 heartbeat: [7965]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:12:58 node1 harc[7965]: [7971]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:12:58 node1 heartbeat: [7980]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:12:58 node1 harc[7980]: [7987]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:12:58 node1 mach_down[7992]: [8012]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 11:12:58 node1 mach_down[7992]: [8017]: info: mach_down takeover complete for node node2.
Sep 14 11:12:58 node1 heartbeat: [7357]: info: mach_down takeover complete.
Sep 14 11:12:58 node1 heartbeat: [7357]: info: Heartbeat shutdown in progress. (7357)
Sep 14 11:12:58 node1 heartbeat: [8018]: info: Giving up all HA resources.
Sep 14 11:12:58 node1 ResourceManager[8032]: [8043]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:12:58 node1 ResourceManager[8032]: [8054]: info: Running /etc/init.d/apache2  stop
Sep 14 11:13:00 node1 ResourceManager[8032]: [8087]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 11:13:00 node1 drbdlinks[8088]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:13:00 node1 drbdlinks[8088]: Exiting with no errors
Sep 14 11:13:00 node1 ResourceManager[8032]: [8103]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 11:13:00 node1 Filesystem[8110]: [8138]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 11:13:00 node1 Filesystem[8110]: [8153]: INFO: Trying to unmount /cluster
Sep 14 11:13:00 node1 Filesystem[8110]: [8162]: INFO: unmounted /cluster successfully
Sep 14 11:13:00 node1 Filesystem[8104]: [8169]: INFO:  Success
Sep 14 11:13:00 node1 ResourceManager[8032]: [8184]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 11:13:00 node1 kernel: [ 4556.758351] block drbd0: role( Primary -> Secondary )
Sep 14 11:13:00 node1 kernel: [ 4556.758362] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 11:13:00 node1 kernel: [ 4556.758997] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 11:13:00 node1 ResourceManager[8032]: [8209]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 11:13:00 node1 IPaddr[8234]: [8245]: INFO: ifconfig eth0:0 down
Sep 14 11:13:00 node1 IPaddr[8210]: [8249]: INFO:  Success
Sep 14 11:13:00 node1 heartbeat: [8018]: info: All HA resources relinquished.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: killing HBFIFO process 7361 with signal 15
Sep 14 11:13:02 node1 heartbeat: [7357]: info: killing HBWRITE process 7362 with signal 15
Sep 14 11:13:02 node1 heartbeat: [7357]: info: killing HBREAD process 7363 with signal 15
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Core process 7362 exited. 3 remaining
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Core process 7361 exited. 2 remaining
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Core process 7363 exited. 1 remaining
Sep 14 11:13:02 node1 heartbeat: [7357]: info: node1 Heartbeat shutdown complete.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Heartbeat restart triggered.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Restarting heartbeat.
Sep 14 11:13:02 node1 heartbeat: [7357]: info: Performing heartbeat restart exec.
Sep 14 11:13:08 node1 heartbeat: [7357]: info: No log entry found in ha.cf -- use logd
Sep 14 11:13:08 node1 heartbeat: [7357]: info: Enabling logging daemon
Sep 14 11:13:08 node1 heartbeat: [7357]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 11:13:08 node1 heartbeat: [7357]: info: **************************
Sep 14 11:13:08 node1 heartbeat: [7357]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 11:13:08 node1 heartbeat: [8250]: info: heartbeat: version 3.0.5
Sep 14 11:13:08 node1 heartbeat: [8250]: info: Heartbeat generation: 1442218518
Sep 14 11:13:08 node1 heartbeat: [8250]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 11:13:08 node1 heartbeat: [8250]: info: Local status now set to: 'up'
Sep 14 11:13:08 node1 heartbeat: [8250]: info: Link node2:eth0 up.
Sep 14 11:13:08 node1 heartbeat: [8250]: debug: get_delnodelist: delnodelist=
Sep 14 11:13:09 node1 heartbeat: [8250]: info: Status update for node node2: status active
Sep 14 11:13:09 node1 heartbeat: [8250]: info: Comm_now_up(): updating status to active
Sep 14 11:13:09 node1 heartbeat: [8250]: info: Local status now set to: 'active'
Sep 14 11:13:09 node1 heartbeat: [8258]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:13:09 node1 harc[8258]: [8266]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:13:19 node1 heartbeat: [8250]: info: local resource transition completed.
Sep 14 11:13:19 node1 heartbeat: [8250]: info: Initial resource acquisition complete (T_RESOURCES(us))
Sep 14 11:13:19 node1 IPaddr[8306]: [8338]: INFO:  Resource is stopped
Sep 14 11:13:19 node1 heartbeat: [8271]: info: Local Resource acquisition completed.
Sep 14 11:13:19 node1 heartbeat: [8250]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:13:19 node1 heartbeat: [8342]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:13:19 node1 harc[8342]: [8348]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 11:13:19 node1 ip-request-resp[8342]: [8354]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 11:13:19 node1 ResourceManager[8355]: [8366]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:13:19 node1 IPaddr[8378]: [8410]: INFO:  Resource is stopped
Sep 14 11:13:19 node1 ResourceManager[8355]: [8425]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 11:13:19 node1 IPaddr[8450]: [8478]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 11:13:19 node1 IPaddr[8450]: [8500]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 11:13:19 node1 IPaddr[8426]: [8519]: INFO:  Success
Sep 14 11:13:19 node1 ResourceManager[8355]: [8549]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 11:13:19 node1 kernel: [ 4576.447471] block drbd0: role( Secondary -> Primary )
Sep 14 11:13:19 node1 Filesystem[8566]: [8610]: INFO:  Resource is stopped
Sep 14 11:13:19 node1 ResourceManager[8355]: [8625]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 11:13:19 node1 Filesystem[8632]: [8660]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 11:13:19 node1 kernel: [ 4576.519118] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 11:13:19 node1 Filesystem[8626]: [8680]: INFO:  Success
Sep 14 11:13:19 node1 drbdlinks[8687]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:13:19 node1 drbdlinks[8687]: Status mode returning stopped
Sep 14 11:13:19 node1 ResourceManager[8355]: [8697]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 11:13:19 node1 drbdlinks[8698]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:13:19 node1 drbdlinks[8698]: Exiting with no errors
Sep 14 11:13:19 node1 ResourceManager[8355]: [8719]: info: Running /etc/init.d/apache2  start
Sep 14 11:13:19 node1 heartbeat: [8250]: info: remote resource transition completed.
Sep 14 11:17:01 node1 /USR/SBIN/CRON[8741]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 14 11:32:05 node1 heartbeat: [8250]: info: Heartbeat shutdown in progress. (8250)
Sep 14 11:32:05 node1 heartbeat: [8776]: info: Giving up all HA resources.
Sep 14 11:32:05 node1 ResourceManager[8790]: [8801]: info: Releasing resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 11:32:05 node1 ResourceManager[8790]: [8812]: info: Running /etc/init.d/mysql  stop
Sep 14 11:32:05 node1 ResourceManager[8790]: [8850]: info: Running /etc/init.d/apache2  stop
Sep 14 11:32:06 node1 ResourceManager[8790]: [8883]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 11:32:06 node1 drbdlinks[8884]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:32:06 node1 drbdlinks[8884]: Exiting with no errors
Sep 14 11:32:06 node1 ResourceManager[8790]: [8899]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 stop
Sep 14 11:32:06 node1 Filesystem[8906]: [8934]: INFO: Running stop for /dev/drbd0 on /cluster
Sep 14 11:32:06 node1 Filesystem[8906]: [8949]: INFO: Trying to unmount /cluster
Sep 14 11:32:06 node1 Filesystem[8906]: [8958]: INFO: unmounted /cluster successfully
Sep 14 11:32:06 node1 Filesystem[8900]: [8965]: INFO:  Success
Sep 14 11:32:06 node1 ResourceManager[8790]: [8980]: info: Running /etc/ha.d/resource.d/drbddisk lamp stop
Sep 14 11:32:06 node1 kernel: [ 5703.480968] block drbd0: role( Primary -> Secondary )
Sep 14 11:32:06 node1 kernel: [ 5703.480978] block drbd0: bitmap WRITE of 0 pages took 0 jiffies
Sep 14 11:32:06 node1 kernel: [ 5703.481631] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
Sep 14 11:32:06 node1 ResourceManager[8790]: [9005]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 stop
Sep 14 11:32:06 node1 IPaddr[9030]: [9041]: INFO: ifconfig eth0:0 down
Sep 14 11:32:06 node1 IPaddr[9006]: [9045]: INFO:  Success
Sep 14 11:32:06 node1 heartbeat: [8776]: info: All HA resources relinquished.
Sep 14 11:32:07 node1 kernel: [ 5703.819546] block drbd0: peer( Secondary -> Primary )
Sep 14 11:32:07 node1 heartbeat: [8250]: WARN: 1 lost packet(s) for [node2] [583:585]
Sep 14 11:32:07 node1 heartbeat: [8250]: info: No pkts missing from node2!
Sep 14 11:32:08 node1 heartbeat: [8250]: info: killing HBFIFO process 8253 with signal 15
Sep 14 11:32:08 node1 heartbeat: [8250]: info: killing HBWRITE process 8254 with signal 15
Sep 14 11:32:08 node1 heartbeat: [8250]: info: killing HBREAD process 8255 with signal 15
Sep 14 11:32:08 node1 heartbeat: [8250]: info: Core process 8254 exited. 3 remaining
Sep 14 11:32:08 node1 heartbeat: [8250]: info: Core process 8253 exited. 2 remaining
Sep 14 11:32:08 node1 heartbeat: [8250]: info: Core process 8255 exited. 1 remaining
Sep 14 11:32:08 node1 heartbeat: [8250]: info: node1 Heartbeat shutdown complete.
Sep 14 11:32:22 node1 kernel: [ 5719.439494] block drbd0: peer( Primary -> Secondary )
Sep 14 11:34:02 node1 heartbeat: [9150]: info: No log entry found in ha.cf -- use logd
Sep 14 11:34:02 node1 heartbeat: [9150]: info: Enabling logging daemon
Sep 14 11:34:02 node1 heartbeat: [9150]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf)
Sep 14 11:34:02 node1 heartbeat: [9150]: info: **************************
Sep 14 11:34:02 node1 heartbeat: [9150]: info: Configuration validated. Starting heartbeat 3.0.5
Sep 14 11:34:02 node1 heartbeat: [9151]: info: heartbeat: version 3.0.5
Sep 14 11:34:02 node1 heartbeat: [9151]: info: Heartbeat generation: 1442218519
Sep 14 11:34:02 node1 heartbeat: [9151]: info: glib: UDP multicast heartbeat started for group 239.0.0.10 port 694 interface eth0 (ttl=1 loop=0)
Sep 14 11:34:02 node1 heartbeat: [9151]: info: Local status now set to: 'up'
Sep 14 11:34:03 node1 heartbeat: [9151]: info: Link node2:eth0 up.
Sep 14 11:34:03 node1 heartbeat: [9151]: info: Comm_now_up(): updating status to active
Sep 14 11:34:03 node1 heartbeat: [9151]: info: Local status now set to: 'active'
Sep 14 11:34:04 node1 heartbeat: [9151]: info: remote resource transition completed.
Sep 14 11:34:04 node1 heartbeat: [9151]: info: remote resource transition completed.
Sep 14 11:34:04 node1 heartbeat: [9151]: info: Local Resource acquisition completed. (none)
Sep 14 11:34:04 node1 heartbeat: [9151]: info: Initial resource acquisition complete (T_RESOURCES(them))
Sep 14 11:34:04 node1 heartbeat: [9151]: info: Status update for node node2: status active
Sep 14 11:34:04 node1 heartbeat: [9162]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:04 node1 harc[9162]: [9168]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:13 node1 heartbeat: [9151]: info: Received shutdown notice from 'node2'.
Sep 14 11:34:13 node1 heartbeat: [9151]: info: Resources being acquired from node2.
Sep 14 11:34:13 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:34:13 node1 heartbeat: [9173]: info: acquire all HA resources (standby).
Sep 14 11:34:13 node1 ResourceManager[9202]: [9221]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:34:13 node1 IPaddr[9246]: [9309]: INFO:  Resource is stopped
Sep 14 11:34:13 node1 IPaddr[9245]: [9310]: INFO:  Resource is stopped
Sep 14 11:34:13 node1 heartbeat: [9174]: info: Local Resource acquisition completed.
Sep 14 11:34:13 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 2
Sep 14 11:34:13 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:34:13 node1 ResourceManager[9202]: [9328]: info: Running /etc/ha.d/resource.d/IPaddr 152.168.1.220/24/eth0 start
Sep 14 11:34:13 node1 IPaddr[9353]: [9381]: INFO: Using calculated netmask for 152.168.1.220: 255.255.255.0
Sep 14 11:34:13 node1 IPaddr[9353]: [9403]: INFO: eval ifconfig eth0:0 152.168.1.220 netmask 255.255.255.0 broadcast 157.164.144.255
Sep 14 11:34:13 node1 IPaddr[9329]: [9422]: INFO:  Success
Sep 14 11:34:13 node1 ResourceManager[9202]: [9452]: info: Running /etc/ha.d/resource.d/drbddisk lamp start
Sep 14 11:34:13 node1 kernel: [ 5830.121541] block drbd0: role( Secondary -> Primary )
Sep 14 11:34:13 node1 Filesystem[9469]: [9513]: INFO:  Resource is stopped
Sep 14 11:34:13 node1 ResourceManager[9202]: [9528]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /cluster ext4 start
Sep 14 11:34:13 node1 Filesystem[9535]: [9563]: INFO: Running start for /dev/drbd0 on /cluster
Sep 14 11:34:13 node1 Filesystem[9529]: [9583]: INFO:  Success
Sep 14 11:34:13 node1 kernel: [ 5830.189046] EXT4-fs (drbd0): mounted filesystem with ordered data mode. Opts: (null)
Sep 14 11:34:13 node1 drbdlinks[9590]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:34:13 node1 drbdlinks[9590]: Status mode returning stopped
Sep 14 11:34:13 node1 ResourceManager[9202]: [9600]: info: Running /etc/ha.d/resource.d/drbdlinks  start
Sep 14 11:34:13 node1 drbdlinks[9601]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'start']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:34:13 node1 drbdlinks[9601]: Exiting with no errors
Sep 14 11:34:13 node1 ResourceManager[9202]: [9622]: info: Running /etc/init.d/apache2  start
Sep 14 11:34:13 node1 heartbeat: [9173]: info: all HA resource acquisition completed (standby).
Sep 14 11:34:13 node1 heartbeat: [9151]: info: Standby resource acquisition done [all].
Sep 14 11:34:13 node1 heartbeat: [9636]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:13 node1 harc[9636]: [9642]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:13 node1 mach_down[9652]: [9672]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Sep 14 11:34:13 node1 mach_down[9652]: [9677]: info: mach_down takeover complete for node node2.
Sep 14 11:34:13 node1 heartbeat: [9151]: info: mach_down takeover complete.
Sep 14 11:34:13 node1 heartbeat: [9678]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:13 node1 harc[9678]: [9684]: info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
Sep 14 11:34:13 node1 ip-request-resp[9678]: [9690]: received ip-request-resp IPaddr::152.168.1.220/24/eth0 OK yes
Sep 14 11:34:13 node1 ResourceManager[9691]: [9702]: info: Acquiring resource group: node1 IPaddr::152.168.1.220/24/eth0 drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2
Sep 14 11:34:13 node1 IPaddr[9714]: [9752]: INFO:  Running OK
Sep 14 11:34:13 node1 Filesystem[9779]: [9818]: INFO:  Running OK
Sep 14 11:34:13 node1 drbdlinks[9825]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'status']", configfile: "/etc/drbdlinks.conf"
Sep 14 11:34:13 node1 drbdlinks[9825]: Status mode returning ok
Sep 14 11:34:19 node1 heartbeat: [9151]: WARN: node node2: is dead
Sep 14 11:34:19 node1 heartbeat: [9151]: info: Dead node node2 gave up resources.
Sep 14 11:34:19 node1 heartbeat: [9151]: info: Link node2:eth0 dead.
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Heartbeat restart on node node2
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Link node2:eth0 up.
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Status update for node node2: status init
Sep 14 11:34:30 node1 heartbeat: [9151]: info: Status update for node node2: status up
Sep 14 11:34:30 node1 heartbeat: [9151]: debug: StartNextRemoteRscReq(): child count 1
Sep 14 11:34:30 node1 heartbeat: [9841]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:30 node1 harc[9841]: [9847]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:30 node1 heartbeat: [9852]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:30 node1 harc[9852]: [9858]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:31 node1 heartbeat: [9151]: debug: get_delnodelist: delnodelist=
Sep 14 11:34:31 node1 heartbeat: [9151]: info: Status update for node node2: status active
Sep 14 11:34:31 node1 heartbeat: [9863]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 14 11:34:31 node1 harc[9863]: [9869]: info: Running /etc/ha.d//rc.d/status status
Sep 14 11:34:32 node1 heartbeat: [9151]: info: remote resource transition completed.

Dernière modification par thomas (14-09-2015 13:02:46)

Hors ligne

#7 14-09-2015 13:22:27

unaM
Membre
Distrib. : Debian Stretch
Noyau : Linux 4.5.0-2-amd64
(G)UI : Gnome 3.20.2
Inscription : 06-01-2012

Re : cluster 2 noeuds actif/passif la ressource mysql plante


Sep 14 10:21:35 node1 mysqld: InnoDB: The error means mysqld does not have the access rights to
Sep 14 10:21:35 node1 mysqld: InnoDB: the directory.
 



As tu vérifier les droits sur les répertoires copiés sur le partage drbd ?


Sep 14 10:21:35 node1 mysqld: InnoDB: File name ./ibdata1
Sep 14 10:21:35 node1 mysqld: InnoDB: File operation call: 'create'.
Sep 14 10:21:35 node1 mysqld: InnoDB: Cannot continue operation.
Sep 14 10:21:35 node1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Sep 14 10:21:40 node1 heartbeat: [4268]: WARN: node node2: is dead
Sep 14 10:21:40 node1 heartbeat: [4268]: info: Cancelling pending standby operation
Sep 14 10:21:40 node1 heartbeat: [4268]: info: Dead node node2 gave up resources.
Sep 14 10:21:41 node1 heartbeat: [4268]: info: Link node2:eth0 dead.
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Sep 14 10:21:49 node1 /etc/init.d/mysql[5343]:
Sep 14 10:21:49 node1 ResourceManager[4321]: [5346]: ERROR: Return code 1 from /etc/init.d/mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5348]: CRIT: Giving up resources due to failure of mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5350]: info: Releasing resource group: node1 IPaddr::[b]152.168.1.220/24/eth0[/b] drbddisk::lamp Filesystem::/dev/drbd0::/cluster::ext4 drbdlinks apache2 mysql
Sep 14 10:21:49 node1 ResourceManager[4321]: [5361]: info: Running /etc/init.d/mysql  stop
Sep 14 10:21:49 node1 ResourceManager[4321]: [5399]: info: Running /etc/init.d/apache2  stop
Sep 14 10:21:50 node1 ResourceManager[4321]: [5432]: info: Running /etc/ha.d/resource.d/drbdlinks  stop
Sep 14 10:21:50 node1 drbdlinks[5433]: drbdlinks starting: args: "['/etc/ha.d/resource.d/drbdlinks', 'stop']", configfile: "/etc/drbdlinks.conf"
 



Quelle est cette adresse IP mentionnée en gras ? Il semblerait qu'elle ne concorde pas avec ta convention d'adressage ? Ca peut être secondaire et non bloquant toutefois.

Le processus mysql tourne il avant que tu ajoute la ressource ou tu laisse le gestionnaire ha le démarrer ?

Je ne connais pas bien heartbeat, j'ai eut quelques expériences avec pacemaker/corosync, ça doit s'en rapprocher smile

Hors ligne

#8 14-09-2015 13:30:51

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

pacemaker est un gestionnaire de ressource
le messager peut être soit corosync, openstack ou bien encore heartbeat


insserv -r apache2
insserv -r mysql
 



c'est bien heartbeat qui gère le démarrage

je regarde pour les droits

Hors ligne

#9 14-09-2015 14:17:32

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

bon j'ai rajouté sur le share drbd /usr/sbin/mysqld et /var/run/mysqld
plus de plantage du cluster

mais l'erreur suivante quand je demande mysql-u root -p status

Can't connect to local MySQL server through socket '/var/mysql/mysql.sock' (2)

Hors ligne

#10 14-09-2015 14:43:03

unaM
Membre
Distrib. : Debian Stretch
Noyau : Linux 4.5.0-2-amd64
(G)UI : Gnome 3.20.2
Inscription : 06-01-2012

Re : cluster 2 noeuds actif/passif la ressource mysql plante

Le fichier /var/mysql/mysql.sock existe bien ?

Hors ligne

#11 14-09-2015 15:11:18

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

c'est fichier qui est généré par le serveur mysql au démarage
donc non il n'existe pas

Hors ligne

#12 14-09-2015 15:36:36

unaM
Membre
Distrib. : Debian Stretch
Noyau : Linux 4.5.0-2-amd64
(G)UI : Gnome 3.20.2
Inscription : 06-01-2012

Re : cluster 2 noeuds actif/passif la ressource mysql plante

L'emplacement du fichier socket indiqué dans la configuration de mysql est peut être mauvais.

Lorsque tu lance mysql, que renvoie la commande

find / -name "*.sock"

Hors ligne

#13 14-09-2015 15:41:42

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

seul fichier .sock trouvé

/run/rpcbind.sock

Hors ligne

#14 14-09-2015 15:44:07

unaM
Membre
Distrib. : Debian Stretch
Noyau : Linux 4.5.0-2-amd64
(G)UI : Gnome 3.20.2
Inscription : 06-01-2012

Re : cluster 2 noeuds actif/passif la ressource mysql plante

Que renvoie

cat /etc/mysql/my.cnf | grep socket



Chez moi cette commande renvoie

socket    = /var/run/mysqld/mysqld.sock
socket    = /var/run/mysqld/mysqld.sock
socket    = /var/run/mysqld/mysqld.sock
 

Hors ligne

#15 15-09-2015 12:41:25

thomas
Membre
Distrib. : Debian wheezy
Noyau : Linux 3.2.0-4-amd64
Inscription : 14-09-2015

Re : cluster 2 noeuds actif/passif la ressource mysql plante

je ne pense pas que le problème soir le fichier /var/run/mysqld/mysqld.sock étant donné que celui-ci est généré par le serveur mysql lui même au démarrage de mysql

tous les tutos que j'ai lu disent que seul le répertoire des bases de données /var/lib/mysql et le fichier /etc/mysql/debian.cnf doivent être mis sur le share drbd

soit en déplaçant les fichiers et en créant des liens symboliques
soit en utilisant drbdlinks qui va les gérer lui-même

hors cela ne fonctionne  pas

Hors ligne

#16 15-09-2015 13:54:23

unaM
Membre
Distrib. : Debian Stretch
Noyau : Linux 4.5.0-2-amd64
(G)UI : Gnome 3.20.2
Inscription : 06-01-2012

Re : cluster 2 noeuds actif/passif la ressource mysql plante

Que dit le log de mysql au démarrage maintenant que le cluster n'hurle plus ?

Hors ligne

#17 15-09-2015 15:29:52

smolski
administrateur quasi...modo
Lieu : AIN
Distrib. : 8 (jessie) 64 bits + backports
Noyau : 4.6.0-0.bpo.1-amd64
(G)UI : gnome 3.14.1
Inscription : 21-10-2008

Re : cluster 2 noeuds actif/passif la ressource mysql plante

qu'il va retirer ses boules quies...

Ok je sors ... vite ! [ ]

"Définition d'eric besson : S'il fallait en chier des tonnes pour devenir ministre, il aurait 2 trous du cul." - JP Douillon
"L'utopie ne signifie pas l'irréalisable, mais l'irréalisée." - T Monod (source :  La zone de Siné)
"Je peux rire de tout mais pas avec n'importe qui." - P Desproges
"saque eud dun" (patois chtimi : fonce dedans)

En ligne

Pied de page des forums