tutus-chain/docs/storage.md

# Tutus Storage Interface

## Overview

The Tutus Storage Interface provides a pluggable architecture for sovereign data storage in government blockchain deployments. Different nations have different data residency requirements, and this interface allows each deployment to choose storage backends that comply with their legal framework.

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                   Tutus Blockchain                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │ Block Store │  │ State Store │  │ Oracle Data │     │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘     │
│         │                │                │             │
│  ┌──────┴────────────────┴────────────────┴──────┐     │
│  │            Storage Interface (pkg/storage)     │     │
│  └──────┬────────────────┬────────────────┬──────┘     │
└─────────┼────────────────┼────────────────┼─────────────┘
          │                │                │
   ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐
   │   NeoFS     │  │    Local    │  │     S3      │
   │   Adapter   │  │   Adapter   │  │   Adapter   │
   └─────────────┘  └─────────────┘  └─────────────┘
          │                │                │
   ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐
   │  Private    │  │ Government  │  │  Sovereign  │
   │  NeoFS Net  │  │    NAS      │  │    Cloud    │
   └─────────────┘  └─────────────┘  └─────────────┘
```

## Data Sovereignty Compliance

### Why This Matters

Governments deploying Tutus must comply with national data residency laws:

| Jurisdiction | Law | Requirement |
|-------------|-----|-------------|
| European Union | GDPR | EU citizen data must remain in EU |
| China | Cybersecurity Law | Critical data must be stored domestically |
| Russia | Federal Law 242-FZ | Personal data of citizens stored in Russia |
| India | DPDP Act 2023 | Certain data categories require local storage |
| Brazil | LGPD | Similar to GDPR for Brazilian citizens |

### Compliance Strategy

1. **Choose Compliant Storage**: Select storage providers that operate within your jurisdiction
2. **Private Networks**: For NeoFS, deploy a private network within national borders
3. **Sovereign Clouds**: Use government-approved cloud providers
4. **Network Isolation**: Configure firewalls to prevent cross-border data transfer
5. **Encryption**: Enable encryption at rest and in transit

## Supported Providers

### NeoFS (Decentralized Storage)

Best for:
- Distributed government networks
- Multi-region deployments within a country
- High availability requirements

```yaml
storage:
  provider: neofs
  neofs:
    endpoints:
      - "grpc://neofs-node1.gov.example:8080"
      - "grpc://neofs-node2.gov.example:8080"
    container: "7s23kG4xLmNpQ..."
    timeout: 30s
    key: "/etc/tutus/neofs.key"
```

**Sovereign Deployment**: Run your own NeoFS network within national infrastructure. Do not use the public NeoFS network for government data.

### Local Filesystem

Best for:
- Single-node deployments
- Development and testing
- Air-gapped networks
- NAS/SAN-backed storage

```yaml
storage:
  provider: local
  local:
    path: /var/lib/tutus/storage
    max_size: 1099511627776  # 1TB
    compression: true
```

### S3-Compatible Storage

Best for:
- Sovereign cloud providers
- Existing government cloud infrastructure
- AWS GovCloud, Azure Government, etc.

```yaml
storage:
  provider: s3
  s3:
    endpoint: "s3.sovereign-cloud.gov.example"
    region: "national-1"
    bucket: "tutus-blocks"
    access_key: "${S3_ACCESS_KEY}"
    secret_key: "${S3_SECRET_KEY}"
    use_path_style: true
```

## Interface Reference

### Provider Interface

The core interface that all storage backends must implement:

```go
type Provider interface {
    // Name returns the provider identifier
    Name() string

    // Put stores data and returns a unique identifier
    Put(ctx context.Context, data io.Reader, opts PutOptions) (ObjectID, error)

    // Get retrieves data by its identifier
    Get(ctx context.Context, id ObjectID) (io.ReadCloser, error)

    // Delete removes an object from storage
    Delete(ctx context.Context, id ObjectID) error

    // Exists checks if an object exists
    Exists(ctx context.Context, id ObjectID) (bool, error)

    // List returns objects matching the given prefix
    List(ctx context.Context, prefix string, opts ListOptions) ([]ObjectInfo, error)

    // Head retrieves object metadata without content
    Head(ctx context.Context, id ObjectID) (ObjectInfo, error)

    // Close releases resources
    Close() error
}
```

### BlockStorage Interface

Extended interface for blockchain-specific operations:

```go
type BlockStorage interface {
    Provider

    // PutBlock stores a block with its index
    PutBlock(ctx context.Context, index uint32, data []byte) (ObjectID, error)

    // GetBlock retrieves a block by its index
    GetBlock(ctx context.Context, index uint32) ([]byte, error)

    // GetBlockRange retrieves multiple blocks efficiently
    GetBlockRange(ctx context.Context, start, end uint32) ([][]byte, error)

    // GetLatestBlockIndex returns the highest block index
    GetLatestBlockIndex(ctx context.Context) (uint32, error)
}
```

### StateStorage Interface

Extended interface for state snapshot operations:

```go
type StateStorage interface {
    Provider

    // PutState stores a state snapshot at a specific height
    PutState(ctx context.Context, height uint32, data io.Reader) (ObjectID, error)

    // GetState retrieves the state snapshot for a given height
    GetState(ctx context.Context, height uint32) (io.ReadCloser, error)

    // GetLatestState returns the most recent state snapshot
    GetLatestState(ctx context.Context) (height uint32, data io.ReadCloser, err error)
}
```

## Usage Examples

### Basic Usage

```go
import (
    "github.com/tutus-one/tutus-chain/pkg/storage"
    "github.com/tutus-one/tutus-chain/pkg/storage/local"
)

// Create provider
provider, err := local.New(local.Config{
    Path: "/var/lib/tutus/storage",
})
if err != nil {
    log.Fatal(err)
}
defer provider.Close()

// Store data
id, err := provider.Put(ctx, strings.NewReader("block data"), storage.PutOptions{
    ContentType: "application/octet-stream",
    Attributes: map[string]string{
        "block_index": "12345",
    },
})

// Retrieve data
reader, err := provider.Get(ctx, id)
if err != nil {
    log.Fatal(err)
}
defer reader.Close()
data, _ := io.ReadAll(reader)
```

### Registry Pattern

```go
// Create registry with multiple providers
registry := storage.NewRegistry()
registry.Register(neofsAdapter)
registry.Register(localAdapter)

// Get provider by name
provider, ok := registry.Get("neofs")
if !ok {
    // Fallback to local
    provider, _ = registry.Get("local")
}
```

## Implementing Custom Providers

To add support for a new storage backend:

1. Create a new package under `pkg/storage/`
2. Implement the `storage.Provider` interface
3. Optionally implement `BlockStorage` or `StateStorage`
4. Add configuration to `storage.Config`

Example structure:

```
pkg/storage/
├── storage.go       # Core interfaces
├── config.go        # Configuration types
├── errors.go        # Error definitions
├── neofs/           # NeoFS adapter
│   └── adapter.go
├── local/           # Local filesystem adapter
│   └── adapter.go
└── mycloud/         # Custom provider
    └── adapter.go
```

## Security Considerations

1. **Encryption at Rest**: Enable storage-level encryption for sensitive data
2. **Access Control**: Use provider-specific ACLs to restrict access
3. **Audit Logging**: Enable logging for all storage operations
4. **Key Management**: Use HSMs or secure key storage for encryption keys
5. **Network Security**: Encrypt all traffic between nodes and storage

## Performance Tuning

### NeoFS
- Increase `timeout` for large objects
- Use multiple endpoints for load balancing
- Configure appropriate `session_expiration_duration`

### Local Storage
- Use SSDs for best performance
- Enable compression for storage efficiency
- Set appropriate `max_size` to prevent disk exhaustion

### S3
- Enable transfer acceleration where available
- Use regional endpoints for lower latency
- Configure appropriate retry policies

## Monitoring

Key metrics to monitor:

| Metric | Description |
|--------|-------------|
| `storage_put_duration_seconds` | Time to store objects |
| `storage_get_duration_seconds` | Time to retrieve objects |
| `storage_errors_total` | Total storage errors by type |
| `storage_objects_total` | Total objects stored |
| `storage_bytes_total` | Total bytes stored |

## Troubleshooting

### Common Issues

**ErrNotFound**
- Object was deleted or never stored
- Check object ID format

**ErrConnectionFailed**
- Storage backend is unreachable
- Check network connectivity and firewall rules

**ErrQuotaExceeded**
- Storage limits reached
- Increase quota or clean up old data

**ErrTimeout**
- Operation took too long
- Increase timeout or check network latency