MySQLTalk.org

Sebastien LELIEVRE · Guest

Hi everyone,

First of all, I would like to thank the owner of this Mailing-List and
MySQL for their support (and products, of course ;)

I am Sebastien, a french student who is now on a training period with
TBS-Internet. TBS-Internet is a certification authority.

I've found this project because my goal is to re-design the
TBS-datacenter architecture and give it fault-tolerance and disaster
recovery.

TBS-Internet is currently using MySQL, and I'm about to deploy a MySQL
Cluster architecture. But first, I'm running some tests, and I need your
help to fully understand the MySQL Cluster principles.

Here is my test configuration :

Server1
/ \
Manager --- |
\ /
Server2

All machines are PIV HT 3GHz 1 Gb RAM, with Ubuntu Server (edgy)
I'm running the latest MySQL source provided, 5.1.17.

I've read quite all I could about MySQL Cluster on Internet in order to
understand how it worked

I've followed this tutorial which was pretty clear :
http://dev.mysql.com/tech-resources/articles/mysql-cluster-for-two-servers.html

Now that I'm running some tests, I'm experiencing some behaviour I
wouldn't expect.

Here I go :

- I've killed (9) the Manager (ndb_mgmd)
- I've killed (9) ndbd on both servers

One I got here, both nodes wasn't responding (quite normal, regards to
the doc)

- I've restarted the Manager (ndb_mgmd)
- I've restarted ndbd on server 2 only

There, I've got an error on server 2, I couldn't access the tables.

- I've restared mysqld (through the mysl.server init script), but the
same error came up

- Strange thing (to me) was that Server 1 could access the Database,
even without ndbd running (I didn't restarted it on server 1)

- I've re-restarted MySQL Server 2, he came up good, and was elected
Master Node (still no ndbd on Server 1)

I don't understand why server 2 couldn't access the database on the
"first" restart, but I do understand why he was elected Master on the
second.

Could you help me with it ? Do you need any more infos (logs or so) to
get a clearly understanding of what happened ?

I let you there another scenario, that I haven't tested yet. I f you
have the answer, please tell me :

Again with the same config, is it possible for Server1 and Server2 to
access and work with the database IF server1 can't establish any
communication of any kind with server2 (and vice-versa)

We suppose here that ndbd are up and running, but that the Manager is
shut down (or unavailable, cable unplugged or so..)

If yes, what would happen when the whole scheme would come back to
normal , (What would happen if the same data was accessed by both
without the other knowing it ?)

More to come, I'm afraid

Sebastien LELIEVRE
slelievre@tbs-internet.com Services to ISP
TBS-internet http://www.TBS-internet.com/

Geert Vanderkelen · Guest

On Apr 27, 2007, at 11:50, Sebastien LELIEVRE wrote:
...

Sebastien LELIEVRE · Guest

Hi, and thanks for answering so quickly !

I'm going to try to develop this scenario :

Geert Vanderkelen · Guest

Hi Sebastien,

On Apr 27, 2007, at 15:11, Sebastien LELIEVRE wrote:
...

Sebastien LELIEVRE · Guest

Geert Vanderkelen a écrit :

Jon Stephens · Guest

Geert Vanderkelen wrote:

Sebastien LELIEVRE · Guest

Hi,

As a fact that I've still found nothing about arbitrator less
configurations for MySQL Cluster, nor that the HA-solution proposed by
both website tutorial & the forum links is able to suite me, I'm
re-posting here my disastrous scenario and questions I've risen and
which didn't find answers yet.

I'am so sorry for the inconvenience it may cause.

So, as I understand it, every "new" data node would become available
only after being synced with the others? Is there any problem with this
scenario and the case where an application constantly access/modify
data? How is this handled?

Does the MGM Node have something to do with database backup? I got the
answer that online backups are made using the ndb_mgm tool and sending
the BACKUP command. The backups are then made on the Data nodes. So that
would mean that ndb_mgm is not linked what so ever with the ndb_mgmd
process and so that the MGM Node(s) could be down while doing BACKUP ?

I've tried that kind of backup, and only Data Node 1 (above 2) was
backed up. Is this normal?

Here comes now the real aim of my post :

Let's take the configuration described before :

SQL API Node 1 +
Data Node 1
/ \
MGM node --- |
\ /
Data Node 2
+ SQL API Node 2

ndb_mgm> show ;
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @192.168.28.5 (Version: 5.1.17, Nodegroup: 0, Master)
id=3 @192.168.28.6 (Version: 5.1.17, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @192.168.28.7 (Version: 5.1.17)

[mysqld(API)] 2 node(s)
id=4 @192.168.28.5 (Version: 5.1.17)
id=5 @192.168.28.6 (Version: 5.1.17)

MGM Node is in room A, with the Data Node 1 and SQL/API Node 1. there is
a fiber link to room B where the Data Node 2 and SQL/API Node 2.

Let's now say that the fiber link is severed (let's say, cut by accident)

(I understood it was more realistic that way Wink

)

The description below might be like an echo for some of you. You can
jump 3 paragraphs ;)

As I understood the Cluster principle for MySQL Cluster, Database is
still accessible in each room while communication between nodes is
unavailable.

Let's now say that the same data is altered in each room, but
differently (someone puts a "0" in a line, the other a "1", It doesn't
really matter...)

My question is : When the fiber link will be restored, how MySQL Cluster
will handle the modifications that was made ?

The link about "network partitioning" assumed that all nodes are always
in contact with the arbitrator. This scenario doesn't take that in count.

Another question would be how much time will it take to get the Cluster
(re)synchronized? (not only with this scenario)
Is there any way to know if data on a node is synchronized or not?

Finally, what configuration would you suggest for a High-Availability
solution? Mr. Geert Vanderkelen was talking about MySQL Cluster with
MySQL Replication between the Datacenters.

He has also pointed me a new "arbitrator less" feature that would solve
my silly scenario above, but told me it would only be useful to 1% of
MySQL Cluster users.

I didn't find any documentation about it. Would you please enlighten me
on this special feature? How does it work?

I was also wondering about any loncking mechanism. Indeed, what would
happen if 2 applications would try to alter the same data at the same
time, but on different nodes? He told me about the Single User Mode
(using the ndb_mgm CLI) which will basically halt the application as it
can't access data anymore. He said this was something they were working
on. Do you know what features are implemented yet? Is there any
documentation about it?

Heartbeat solutions are not possible in our case since we do not really
have the hand on our IPs nor machines.

Regards,

Sebastien LELIEVRE
slelievre@tbs-internet.com Services to ISP
TBS-internet http://www.TBS-internet.com/

Geert Vanderkelen · Guest

Hi Sebastien,

On May 2, 2007, at 14:27, Sebastien LELIEVRE wrote:
...

Sebastien LELIEVRE · Guest

Hi Geert,

Once again, thanks for elightening me. I'm almost prepare to preach the
MySQL Cluster way across my IT and bring them synchronous data
consistency! ;)

(I've already begun : I left for lunch yelling "never put the --initial
parameteeeeeeeeeer!!!" Wink

They all thought I was mad Razz

)

I said "almost" because I still have ONE little dark spot with all this.
Only one, and I will have a clear view of MySQL Cluster.

This dark spot concerns the arbitrator (him again!)

Geert Vanderkelen a écrit :

Geert Vanderkelen · Guest

Hi Sebastien,

On May 4, 2007, at 15:44, Sebastien LELIEVRE wrote:
...

Sebastien LELIEVRE · Guest

Good Afternoon everyone

It's nearly the end of the tunnel for my understanding of MySQL Cluster
basics, and it's all thanks to you and your patience. Cheers!

Geert Vanderkelen a écrit :

Geert Vanderkelen · Guest

On May 10, 2007, at 17:09, Sebastien LELIEVRE wrote:
...