The following is a printout of performance metrics gathered from Electric Cloud's emake product. It has several sections that analyze different parts of system performance, such as overall utilizations, bandwidths, etc.
Cluster summary statistics -------------------------- Total time: 1004.3s, 1949 jobs, 6282 commands 10 agents Overall time usage: Startup: 5.35s ( 0.1%) 10 intvls, avg 535.0ms Command: 3032.57s (30.2%) 6282 intvls, avg 482.7ms Emake request: 2124.16s (21.2%) 176495 intvls, avg 12.0ms A2A request: 202.80s ( 2.0%) 6478 intvls, avg 31.3ms Disk cache: 43.11s ( 0.4%) 4289 intvls, avg 10.1ms Command end: 20.45s ( 0.2%) 6282 intvls, avg 3.3ms Return: 3292.81s (32.8%) 1949 intvls, avg 1689.5ms Idle: 1040.77s (10.4%) 1939 intvls, avg 536.8ms End: 280.99s ( 2.8%) Emake request timers: A2E_GET_FILE_NAME_AND_ATTRIBUTES: 249.87s ( 2.5%) A2E_GET_FILE_DATA (<50000 bytes): 831.88s ( 8.3%) A2E_GET_FILE_DATA (50000-500000 bytes): 206.65s ( 2.1%) A2E_GET_FILE_DATA (>=500000 bytes): 956.61s ( 9.5%) A2E_GET_DIRECTORY_ENTRIES: 112.59s ( 1.1%) A2E_RUN_COMMAND: 0.00s ( 0.0%) A2E_GET_REG_KEY: 12.45s ( 0.1%) Bandwidth: Network from emake: 4905.8 MB, 2.1 MB/s active, 4.9 MB/s overall Network to emake: 8553.8 MB, 2.6 MB/s active, 8.5 MB/s overall Network from agents: 1451.7 MB, 7.2 MB/s active, 1.4 MB/s overall Network to agents: 1451.7 MB, 8.5 MB/s active, 1.4 MB/s overall From disk: 5262.2 MB, 122.1 MB/s active, 5.2 MB/s overall Small files: 590.6 MB, 0.7 MB/s Medium files: 581.9 MB, 2.8 MB/s Large files: 3429.1 MB, 3.6 MB/s Average # active agents: Receiving from emake: 2.4 Sending to emake: 3.3 Receiving from agents: 0.2 Sending to agents: 0.2 Reading from disk: 0.0 Receiving small files: 0.8 Receiving medium files: 0.2 Receiving large files: 1.0 Caching: Lookups: 0.1% (86031937 to EFS, 81749 to emake) Read traffic: 8.3% (72954.9 MB from EFS, 6053.4 MB from emake) Write traffic: 100.9% (8400.7 MB to EFS, 8475.8 MB to emake) Directory scans: 0.5% (4039608 to EFS, 19636 to emake) Cache group 0: 39260 file loads ( 33.6 MB), 1949 flushes (100.0% of jobs) Cache group 1: 1420200 file loads ( 753.5 MB), 255 flushes ( 13.1% of jobs) Cache group 2: 228016 file loads ( 91.5 MB), 923 flushes ( 47.4% of jobs) Buffer usage: Average buffer usage: 17.1 Maximum buffer usage: 308 Per-job statistics: File opens: 3172.8 per job ( 6183854 total) Failed lookups: 5017.2 per job ( 9778497 total) Emake requests: 90.6 per job ( 176495 total) File bytes read remotely (KB): 3105.9 per job ( 6053383 total) Agent-to-agent file transfers: Incoming: 2249 requests, 96 rejections ( 4.3%) Outgoing: 2249 requests, 96 rejections ( 4.3%) Disk cache: Write traffic: 2208 files, 3723.0 MB ( 1.7 MB avg) Read traffic: 4289 files, 5262.2 MB ( 1.2 MB avg) Other timers: Emake latency: 864.69s ( 8.6%) 176495 intvls, avg 4.9ms A2A startup: 50.16s ( 0.5%) 2249 intvls, avg 22.3ms A2A transmit: 170.19s ( 1.7%) 2153 intvls, avg 79.0ms Clear usage: 37.16s ( 0.4%) 1949 intvls, avg 19.1ms Usage records: Create 8052 ( 0.2%), 4.1 per job Link 2 ( 0.0%), 0.0 per job Lookup 1702017 ( 51.6%), 873.3 per job Modify 1267 ( 0.0%), 0.7 per job Modify atts 118 ( 0.0%), 0.1 per job New name 1494624 ( 45.3%), 766.9 per job Read 94718 ( 2.9%), 48.6 per job Rename 0 ( 0.0%), 0.0 per job Truncate 145 ( 0.0%), 0.1 per job Unlink 1 ( 0.0%), 0.0 per job Unlink(last) 9 ( 0.0%), 0.0 per job Total 3300953 (100.0%), 1693.7 per job