claude-mcp-demo/claude-nfs-benchmark.md

# NFS Performance Benchmark - Claude Analysis

**Date:** 2026-01-19  
**Storage Class:** nfs-csi  
**NFS Server:** 192.168.0.105:/nfs/NFS/ocp  
**Test Environment:** OpenShift Container Platform (OCP)  
**Tool:** fio (Flexible I/O Tester)

## Executive Summary

Performance testing of the NAS storage via nfs-csi storage class reveals actual throughput of **65-80 MiB/s** for sequential operations. This represents typical performance for 1 Gbps Ethernet NFS configurations.

## Test Configuration

### NFS Mount Options
- **rsize/wsize:** 1048576 (1MB) - optimal for large sequential transfers
- **Protocol options:** hard, noresvport
- **Timeout:** 600 seconds
- **Retrans:** 2

### Test Constraints
- CPU: 500m
- Memory: 512Mi
- Namespace: nfs-benchmark (ephemeral)
- PVC Size: 5Gi

## Benchmark Results

### Sequential I/O (1M block size)

#### Sequential Write
- **Throughput:** 70.2 MiB/s (73.6 MB/s)
- **IOPS:** 70
- **Test Duration:** 31 seconds
- **Data Written:** 2176 MiB

**Latency Distribution:**
- Median: 49 µs
- 95th percentile: 75 µs
- 99th percentile: 212 ms (indicating occasional network delays)

#### Sequential Read
- **Throughput:** 80.7 MiB/s (84.6 MB/s)
- **IOPS:** 80
- **Test Duration:** 20 seconds
- **Data Read:** 1615 MiB

**Latency Distribution:**
- Median: 9 ms
- 95th percentile: 15 ms
- 99th percentile: 150 ms

### Synchronized Write Test

**Purpose:** Measure actual NAS performance without local caching

- **Throughput:** 65.9 MiB/s (69.1 MB/s)
- **IOPS:** 65
- **fsync latency:** 13-15ms average

This test provides the most realistic view of actual NAS write performance, as each write operation is synchronized to disk before returning.

### Random I/O (4K block size, cached)

**Note:** These results heavily leverage local page cache and do not represent actual NAS performance.

#### Random Write
- **Throughput:** 1205 MiB/s (cached)
- **IOPS:** 308k (cached)

#### Random Read
- **Throughput:** 1116 MiB/s (cached)
- **IOPS:** 286k (cached)

### Mixed Workload (70% read / 30% write, 4 concurrent jobs)

- **Read Throughput:** 426 MiB/s
- **Read IOPS:** 109k
- **Write Throughput:** 183 MiB/s
- **Write IOPS:** 46.8k

**Note:** High IOPS values indicate substantial local caching effects.

## Analysis

### Performance Characteristics

1. **Actual NAS Bandwidth:** ~65-80 MiB/s
   - Consistent across sequential read/write tests
   - Synchronized writes confirm this range

2. **Network Bottleneck Indicators:**
   - Performance aligns with 1 Gbps Ethernet (theoretical max ~125 MiB/s)
   - Protocol overhead and network latency account for 40-50% overhead
   - fsync operations show 13-15ms latency, indicating network RTT

3. **Caching Effects:**
   - Random I/O tests show 10-15x higher throughput due to local page cache
   - Not representative of actual NAS capabilities
   - Useful for understanding application behavior with cached data

### Bottleneck Analysis

The ~70 MiB/s throughput is likely limited by:

1. **Network Bandwidth** (Primary)
   - 1 Gbps link = ~125 MiB/s theoretical maximum
   - NFS protocol overhead reduces effective throughput to 55-60%
   - Observed performance matches expected 1 Gbps NFS behavior

2. **Network Latency**
   - fsync showing 13-15ms indicates network + storage latency
   - Each synchronous operation requires full round-trip

3. **NAS Backend Storage** (Unknown)
   - Current tests cannot isolate NAS disk performance
   - Backend may be faster than network allows

## Recommendations

### Immediate Improvements

1. **Upgrade to 10 Gbps Networking**
   - Most cost-effective improvement
   - Could provide 8-10x throughput increase
   - Requires network infrastructure upgrade

2. **Enable NFS Multichannel** (if supported)
   - Use multiple network paths simultaneously
   - Requires NFS 4.1+ with pNFS support

### Workload Optimization

1. **For Write-Heavy Workloads:**
   - Consider async writes (with data safety trade-offs)
   - Batch operations where possible
   - Use larger block sizes (already optimized at 1MB)

2. **For Read-Heavy Workloads:**
   - Current performance is acceptable
   - Application-level caching will help significantly
   - Consider ReadOnlyMany volumes for shared data

### Alternative Solutions

1. **Local NVMe Storage** (for performance-critical workloads)
   - Use local-nvme-retain storage class for high-IOPS workloads
   - Reserve NFS for persistent data and backups

2. **Tiered Storage Strategy**
   - Hot data: Local NVMe
   - Warm data: NFS
   - Cold data: Object storage (e.g., MinIO)

## Conclusion

The NAS is performing as expected for a 1 Gbps NFS configuration, delivering consistent 65-80 MiB/s throughput. The primary limitation is network bandwidth, not NAS capability. Applications with streaming I/O patterns will benefit from the current configuration, while IOPS-intensive workloads should consider local storage options.

For significant performance improvements, upgrading to 10 Gbps networking is the most practical path forward.

---

## Test Methodology

All tests were conducted using:
- Ephemeral namespace with automatic cleanup
- Constrained resources (500m CPU, 512Mi memory)
- fio version 3.6
- Direct I/O where applicable to minimize caching effects

Benchmark pod and resources were automatically cleaned up after testing, following ephemeral testing protocols.