how to increase the number of file descriptors

(from : http://b.oldhu.com/2012/07/19/increase-tcp-max-connections-on-mac-os-x/)

Increase TCP max connections on Mac OS X
19 July 2012

Reason

I was trying to benchmark a TCP based file server on Mac OS X. The server is running latest Node.js (0.8.2), while the client to push the server is written in go. ‘Go’ can start thousands of concurrent goroutines easily. I got error when there are hundreds of concurrent clients.

Steps

1. increate max open files

   $ sysctl -a | grep files
   kern.maxfiles = 12288
   kern.maxfilesperproc = 10240
   

   kern.maxfiles and kern.maxfilesperproc were small numbers, they need to be increased:

   $ sudo sysctl -w kern.maxfiles=12288
   $ sudo sysctl -w kern.maxfilesperproc=10240
   

   after this, you can increase your account’s limit by ulimit -n:

   $ ulimit -n 10240
   

2. increate max sockets

   $ sysctl -a | grep somax
   kern.ipc.somaxconn: 2048
   

   It was a small number and need to be increased:

   $ sudo sysctl -w kern.ipc.somaxconn=2048
   

from : http://docs.basho.com/riak/latest/cookbooks/Open-Files-Limit/

Open Files Limit

1. 1.0.0
2. 1.1.0
3. 1.2.1
4. 1.3.0rc2

Contents

1. Changing the limit
2. Linux
3. Solaris
4. Mac OS X

Riak can consume a large number of open file handles during normal operation. In particular, the Bitcask backend may accumulate a number of data files before it has a chance to run a merge process. You can count the number of data files in the bitcask directory with following command:

ls data/bitcask/*/* | wc -l

Please note that the creation of numerous data files is normal. Each time Riak is started Bitcask creates a new data file per partition; every so often Bitcask will merge a collection of data files into a single file to avoid accumulating file handles. It’s possible to artificially inflate the number of file handles Bitcask uses by repeatedly writing data and restarting Riak. The shell command below illustrates this issue:

for i in {1..100}
  do
    riak stop
    riak start
    sleep 3
    curl http://localhost:8098/riak/test -X POST -d "x" \
      -H "Content-Type: text/plain"
    ls data/bitcask/*/* | wc -l
done

Changing the limit

Most operating systems can change the open-files limit using the ulimit -n command. Example:

ulimit -n 1024

However, this only changes the limit for the current shell session. Changing the limit on a system-wide, permanent basis varies more between systems.

Linux

On most Linux distributions, the total limit for open files is controlled by sysctl.

sysctl fs.file-max
fs.file-max = 50384

As seen above, it is generally set high enough for Riak. If you have other things running on the system, you might want to consult the sysctl manpage for how to change that setting. However, what most needs to be changed is the per-user open files limit. This requires editing /etc/security/limits.conf, which you’ll need superuser access to change. If you installed Riak or Riak Search from a binary package, add lines for the riak user like so, substituting your desired hard and soft limits:
On Ubuntu, if you’re always relying on the init scripts to start Riak, you can create the file /etc/default/riak and specify a manual limit like so:

ulimit -n 4096

This file is automatically sourced from the init script, and the Riak process started by it will properly inherit this setting. As init scripts are always run as the root user, there’s no need to specifically set limits in /etc/security/limits.conf if you’re solely relying on init scripts.
On CentOS/RedHat systems make sure to set a proper limit for the user you’re usually logging in with to do any kind of work on the machine, including managing Riak. On CentOS, sudo properly inherits the values from the executing user.
Reference: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Enabling PAM Based User Limits

It can be helpful to enable PAM user limits so that non-root users, such as the riak user may specify a higher value for maximum open files. For example, to enable PAM user limits and set the soft and hard values for all non-root users to 8192 open files, follow these steps:

1. Edit /etc/pam.d/su and remove the comment on this line:
   

    #session    required   pam_limits.so
   

   so that it resembles this line:
   

    session    required   pam_limits.so
   

2. Save and close the file
3. Edit /etc/security/limits.conf, and append the following lines to the file:
   

    riak               soft     nofile          8192
    riak               hard     nofile          8192
   

4. Save and close the file
5. Restart the machine so that the limits to take effect, and verify with the command: ulimit -a.

Solaris

In Solaris 8, there is a default limit of 1024 file descriptors per process. In Solaris 9, the default limit was raised to 65536. To increase the per-process limit on Solaris, add the following line to /etc/system:

set rlim_fd_max=65536

Reference: http://blogs.oracle.com/elving/entry/too_many_open_files

Mac OS X

To check the current limits on your Mac OS X system, run (the final two columns are the soft and hard limits, respectively):

launchctl limit maxfiles 2048 unlimited

This will set the limit until the next time you reboot. To make the change permanent add the line below to /etc/launchd.conf (superuser access required):

limit maxfiles 2048 unlimited

Note: Snow Leopard (10.6) may not allow “unlimited” for the maxfiles setting, in which case, just omit it.
Reference: http://artur.hefczyc.net/node/27

OS X 10.7 (Lion)

To adjust the maximum open file limits in OS X 10.7 (Lion), edit /etc/launchd.conf, and increase the limits for both values as appropriate.
For example, to set the soft limit to 16384 files, and the hard limit to 32768 files, perform the following steps:
Verify current limits:

launchctl limit

    cpu         unlimited      unlimited
    filesize    unlimited      unlimited
    data        unlimited      unlimited
    stack       8388608        67104768
    core        0              unlimited
    rss         unlimited      unlimited
    memlock     unlimited      unlimited
    maxproc     709            1064
    maxfiles    10240          10240

Edit /etc/launchd.conf (if the file does not exist, create it), and increase the limits, so that it resembles the following values, being sure to use values specific to your environment or as directed:

limit maxfiles 16384 32768

Save the file, and restart the system for the new limits to take effect. After restarting, verify the new limits with the launchctl limit command:

launchctl limit

    cpu         unlimited      unlimited
    filesize    unlimited      unlimited
    data        unlimited      unlimited
    stack       8388608        67104768
    core        0              unlimited
    rss         unlimited      unlimited
    memlock     unlimited      unlimited
    maxproc     709            1064
    maxfiles    16384          32768