Index of Questions

Backbone of a large file-streaming system
Requirements to submit jobs with TORQUE (resource manager)
How do I build a Beowulf cluster?
Can DansGuardian be set up on a cluster?
Communication problems with SGE6
Advice on the best cluster OS and cluster management software

New Questions -(If you have knowledge on these solutions, please submit an answer to the questions)

1. How many clusters have been implemented at US research institutions? In the US as a whole? Worldwide?

2. I would like to learn more about general basics of scheduling, i.e. how to design a scheduler, what are the common do's and don'ts, what are common implementation architectures for schedulers, etc.

Could you kindly point me to some starting points (books or web links) where I might find such basic knowledge?

Ongoing Questions

1. I am looking for a variety of solutions to be a backbone of a large-file streaming system providing thousands of concurrent download streams. Preferably commodity hardware and Linux, though I'm open to commercial solutions.

• David from the NFS user list submitted the following:
  This is the type of thing where I would expect you would want a san with something like gfs so that all servers can directly access all disks whenever they need to. Depending on the speed needed and the amount of money you have to spend, you could go fiber, or iscsi (that could be 1Gb or 10Gb). It all comes down to money.

• Eric Anderson wrote:
  I suppose the answer depends on the protocol used for the streaming downloads. Assuming HTTP, apache on a FreeBSD server should handle this nicely.

• A. Aggelis wrote:
  It all depends on job. This is the moto of clustering. From what I have read the solution is load balancing/redundancy cluster. Probably something like what is described in the article http://www.countersiege.com/doc/pfsync-carp/ at the section "Something Bigger. " This larger configuration is in place at a large educational institution, providing load balancing and redundancy for a cluster of web servers:"

• The key point here is on load balancing and redundancy.
  
  client
  |
  |
  internet
  |
  Load Balancer/Firewall
  | | . .......... |
  | | |
  | | |
  server1 server2 serverN

• As far as the hardware , that depends on the kind and magnitude of load that you expect.
  To read about one specific device and it's offerings, please visit:   http://www.coyotepoint.com/pdfs/extreme_05.pdf

• NFS user list response: If it’s streaming software needed, try Darwin from Apple

2. What do I need besides a resource manager (TORQUE) to submit jobs on a Red Hat Linux? Do I need to install an MPI implementation?
(see Resource Manager, Communications).

• To do basic submissions in a cluster environment, just having TORQUE is sufficient. It does not have any prerequisites for job submissions. However, the type of jobs submitted may require certain components to be installed like an MPI library. It comes down to site specific configurations (See

• On complex cluster environments, a resource manager like TORQUE can increase the utilization of a system from 20% to 70%. By adding a workload manager to the environment you can increase system utilization to consistently reach 90 to 99% (See Workload Manager).

3. How do I build a Beowulf cluster from used PCs? How do I get it to work with Oracle9i or Oracle10g?

• Recommended at Beowulf.org, Robert G. Brown, a professor in the Duke University Physics Department, gives detailed instructions on how to build a Beowulf cluster. Click here.

• Oracle's documentation Web site has many valuable instructions. Click here for those guides and installation instructions on Oracle 10g and RAC (Real Application Clusters). A valuable Oracle.com guide for installation and configuration on the most used operating systems can be found here. These guides give requirements and instructions on how to set up the system. To view the documentation, you will need to create an account at Oracle.com.

4. I want to know if DansGuardian could be set up on any of the forms of clusters available today. Has anyone ever done this?

• According to the conversations on the DansGuardian user group, people have ran DansGuardian on a cluster. Some people suggested using OpenMosix as the solution to get it running. Others, however, suggest that OpenMosix is not the best solution and that programs such as Linux Virtual Server Project, KeepAliveD and Mandrake work best.

• The differences between OpenMoxix and LVS are detailed in this response at http://groups.yahoo.com/group/dansguardian/message/7006SGE6
  
  "[OpenMosix] works well when it doesn't need a local, unique storage. In that case, forking the process to a different computer is bad as you take the process away from the data. In case of proxying this is NOT a problem as all cache is valid, no matter where it is (be it the original or the forked CPU). So it might seem OpenMosix is OK for this. But, in OpenMosix you impose a load managing the processes. The processes move around based on which CPU has some spare power. This adds some latency (process jumping, management, algorithms) and consumes CPU by itself. Instead, with LVS you have a defined number of processes in a cluster of CPUs, and the LVS daemon just keeps sending data to each one of them, it doesnt FORK a process (that by itself consumes CPU) but sends data to a different daemon. If you consider the same number of DG processes in all CPUs in both cases, LVS is much lighter to the system as it doesn't need to fork, nor control where a process is, or if it needs to come back or jump again... LVS is less stressful for a cluster."

5. I installed the latest version of SGE6 on my cluster. The installation of server is OK. But when I install nodes, it shows that it can't communicate with the server even when I turn off the firewall. But I can ssh to the node from the server. NIS and NFS work. How do I fix this?

• The following is a response submitted from the SGE email list user. He gave two items to make sure and check.

  • Does qping work?

  • Are the ports correctly specified (/etc/services or via environment variables)?
    

6. I'm bulding a project for a cluster that consists on 64 machines running each 2x Dual Core AMD Opteron with 2GB of memory per core (8GB per machine). We already have everything else accounted for except for the software. We have serious doubts on what would be best for the OS and the cluster management software. The OS has to be open source. Please give me some advice on what you think you be the best performace solution for it. It doesn''t matter if it's not free. 
Due to the overwhelming response from the HPC community, click here for the responses.

7. I built our Beowulf cluster from 10 PC(pentium4 2.2) but I do not know whether I can build such a cluster from some old computers which have 2 or 4 processors integrated (multiprocessors). The problem is, instead of using an Ethernet card, I want to use another communication card for better performance. What is the best solution?

• Before spending serious money on some low-latency interconnect I would check if this is really what you need.Check if your bottleneck isn't elsewhere. Also if your nodes don't support pci-x or pci-e it makes no sense whatsoever to buy an infiniband or myrinet card, because your bottleneck will be the pci bus. Even for Gigabit ethernet a 32-bit pci bus will give lousy performance.

8. Can a compute node be actually used as a host for submitting jobs?

• Yes, in most cases you can use the host node as a compute node and submit jobs from it. For instance, in Torque resource manager, along with installing the server on the head node you also install and run the mom client on the head node.
  

• At this point, the host node also shows up as a compute node and you can submit jobs. However, using the host node as a compute node to submit jobs has its drawbacks. If a large job that takes a lot of processing power is submitted, then the entire cluster can be slowed down.

9. Which software is best to load the Redhat Hat Linux ES4 on 150-odd nodes from one cluster management node?

1. Just set all the hosts to netboot from their ethernet card
2. Set up a host as a kickstart server
3. Then turn on the 150 hosts and watch them install rh es4 over the network.

• SystemImager vs. Rocks Avalanche (bittorrent-enhanced kickstart), Iguess. SystemImager's more general-purpose (e.g. not limited to redhatoids, easily used for updating with up-to-the-minute patched images), but its "Flamethrower" distribution method (to make it vaguely throughput-competitive with avalanche) uses multicast and therefore depends on multicast working right and is generally a bit more brittle (it's 'casting, so the installs have to be synchronised).
• http://www.rocksclusters.org/
  http://www.systemimager.org/

10. How do you configure a two node GFS Cluster? 

• Two places have detailed documentation on GFS cluster. You can find documentation at Red Hat's site (http://www.redhat.com/docs/manuals/csgfs/browse/rh-gfs-en/ ) and Datacore - https://open.datacore.ch/DCwiki.open/Wiki.jsp?page=GFS#section-GFS-LocalDocuments

11. How to utilize MPICH2/mpiexec with Torque/qsub in order to submit jobs?

Example: I can submit parallel processes with mpiexec just by issuing command: mpiexec -n 3 ./cpi
In order for me to start it with Torque, do i have to just do it like this: qsub mpiexec.script
(mpiexec.script includes "mpiexec -n 3 ./cpi") (Submitted December 1, 2005)