MySQLTalk.org MYSQL discussions groups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  ndbd process causing high mem usage and cpu load          MySQLTalk.org Forum Index -> MySQL Cluster Discussion View previous topic :: View next topic   Author Message Aziz Sasmaz Guest Posted: Thu May 17, 2007 10:10 am    Post subject: ndbd process causing high mem usage and cpu load Hi All, On my production environment I am using Mysql 5.1.17 (MYISAM tables ) on a 4U machine with 4 Intel Xeon MP and 6 GB of ram. I have also 3 web servers and 3 of them are machines with 2 Intel Xeon processor and 6 Gb of ram each. For the SQL Node i am planning to use the 4U machine with 4 CPU. For NDB nodes i am planning to use 2 of my web servers. Also i want to run extra SQL nodes on these machines. For the management node I am planning to use an extra machine. Before migration: I created my testing environment at home with 4 computers. But each of them are P4 cpu and 512 Mb of ram. I successfully install and configure mysql cluster and for newly created databases it works great. But, When i try to convert my existing database tables with ALTER TABLE, ndbd nodes use lots of memory and swap space also CPU load is very high. While trying to convert huge tables, ndbd node dies. when i look ps -auxf i see ndbd process uses %60 of memory and increases every minute. But SQL processes seem ok. No load no swap space. My existing database size is 1 GB. Some tables has 2.000.000 rows. I also tried with a dump file. Memory usage and swap usage seems the same on ndb hosts. I am not sure if i expreince the same problem with my production server. Why NDB nodes use so much memory and load but SQL node doesn't use so much while converting tables or inserting rows? Is it a good idea to use web servers at the same machine with ndbd nodes. Or is it good to use on the same machine with sql nodes? my config.ini [NDBD DEFAULT] NoOfReplicas=2 DataMemory=512M IndexMemory=64M MaxNoOfAttributes=3000 MaxNoOfTables = 1000 MaxNoOfOrderedIndexes = 1024 [TCP DEFAULT] portnumber=2202 [NDB_MGMD] hostname=192.168.1.1 datadir=/var/lib/mysql-cluster [NDBD] hostname=192.168.1.2 datadir=/usr/local/mysql/data [NDBD] hostname=192.168.1.3 datadir=/usr/local/mysql/data [MYSQLD] [MYSQLD] [MYSQLD] ----------------------------------------- Thank you.. Aziz Sasmaz Back to top Geert Vanderkelen Guest Posted: Thu May 17, 2007 10:10 am    Post subject: Re: ndbd process causing high mem usage and cpu load Hi Aziz, On May 17, 2007, at 11:36, Aziz Sasmaz wrote: Quote: Hi All, On my production environment I am using Mysql 5.1.17 (MYISAM tables ) on a 4U machine with 4 Intel Xeon MP and 6 GB of ram. I have also 3 web servers and 3 of them are machines with 2 Intel Xeon processor and 6 Gb of ram each. For the SQL Node i am planning to use the 4U machine with 4 CPU. For NDB nodes i am planning to use 2 of my web servers. Also i want to run extra SQL nodes on these machines. No, you don't want to run mysqld processes together on same machine where ndbd is running. ndbd processes should be (best) 1 on each machine and nothing else. Quote: For the management node I am planning to use an extra machine. Before migration: I created my testing environment at home with 4 computers. But each of them are P4 cpu and 512 Mb of ram. ... i see ndbd process uses %60 of memory and increases every minute. But SQL processes seem ok. No load no swap space. My existing database size is 1 GB. Some tables has 2.000.000 rows. You're trying to put an elephant in a match box. You can't just put 1Gb of data into a system with only 512Mb of RAM/Datamemory Quote: I also tried with a dump file. Memory usage and swap usage seems the same on ndb hosts. I am not sure if i expreince the same problem with my production server. Why NDB nodes use so much memory and load but SQL node doesn't use so much while converting tables or inserting rows? Because NDB is an in-memory database. With 5.1 there is some disk-based support too. Quote: Is it a good idea to use web servers at the same machine with ndbd nodes. Or is it good to use on the same machine with sql nodes? No, like I mentioned above: ndbd should be standalone on the machine so it can use system resources exclusively. This is important for RAM as MySQL Cluster is mainly in-memory. Quote: [NDBD DEFAULT] NoOfReplicas=2 DataMemory=512M IndexMemory=64M Try to add here: LockPagesInMainMemory=2 It will prevent swapping, which is good. BR, Geert -- Geert Vanderkelen, Support Engineer MySQL GmbH, Germany, www.mysql.com Back to top Aziz Sasmaz Guest Posted: Thu May 17, 2007 8:00 pm    Post subject: Re: ndbd process causing high mem usage and cpu load Thank you Geert, You are exactly right, i tried to put an elephant into a match box. I have to add more ram for my testing environment cause It is making 'on memory query execution' Also i will buy extra servers for my production environment to run stand-alone ndbd processes. thanks, AS On 5/17/07, Geert Vanderkelen <geert (AT) mysql (DOT) com> wrote: Quote: Hi Aziz, On May 17, 2007, at 11:36, Aziz Sasmaz wrote: Hi All, On my production environment I am using Mysql 5.1.17 (MYISAM tables ) on a 4U machine with 4 Intel Xeon MP and 6 GB of ram. I have also 3 web servers and 3 of them are machines with 2 Intel Xeon processor and 6 Gb of ram each. For the SQL Node i am planning to use the 4U machine with 4 CPU. For NDB nodes i am planning to use 2 of my web servers. Also i want to run extra SQL nodes on these machines. No, you don't want to run mysqld processes together on same machine where ndbd is running. ndbd processes should be (best) 1 on each machine and nothing else. For the management node I am planning to use an extra machine. Before migration: I created my testing environment at home with 4 computers. But each of them are P4 cpu and 512 Mb of ram. .. i see ndbd process uses %60 of memory and increases every minute. But SQL processes seem ok. No load no swap space. My existing database size is 1 GB. Some tables has 2.000.000 rows. You're trying to put an elephant in a match box. You can't just put 1Gb of data into a system with only 512Mb of RAM/Datamemory I also tried with a dump file. Memory usage and swap usage seems the same on ndb hosts. I am not sure if i expreince the same problem with my production server. Why NDB nodes use so much memory and load but SQL node doesn't use so much while converting tables or inserting rows? Because NDB is an in-memory database. With 5.1 there is some disk-based support too. Is it a good idea to use web servers at the same machine with ndbd nodes. Or is it good to use on the same machine with sql nodes? No, like I mentioned above: ndbd should be standalone on the machine so it can use system resources exclusively. This is important for RAM as MySQL Cluster is mainly in-memory. [NDBD DEFAULT] NoOfReplicas=2 DataMemory=512M IndexMemory=64M Try to add here: LockPagesInMainMemory=2 It will prevent swapping, which is good. BR, Geert -- Geert Vanderkelen, Support Engineer MySQL GmbH, Germany, www.mysql.com Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         MySQLTalk.org Forum Index -> MySQL Cluster Discussion All times are GMT Page 1 of 1   Jump to: Select a forum MySQL----------------MySQL and JavaMySQL AnnouncementsMySQL++MySQL and .NETMySQL ODBCMySQL and PerlDatabase BenchmarksMySQL BugsMySQL Cluster DiscussionMySQL CommunityMySQL DocumentationEventum DevelopmentEventum General DiscussionMySQL General DiscussionMySQL GUI ToolsMySQL InternalsMySQL CommitsMaxDBMySQL PackagersMySQL Replication DiscussionBrisbane MySQL Users GroupMySQL on Win32  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.