tutus-chain/docs/storage.md

Tutus Storage Interface

Overview

The Tutus Storage Interface provides a pluggable architecture for sovereign data storage in government blockchain deployments. Different nations have different data residency requirements, and this interface allows each deployment to choose storage backends that comply with their legal framework.

Architecture

┌─────────────────────────────────────────────────────────┐
│                   Tutus Blockchain                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │ Block Store │  │ State Store │  │ Oracle Data │     │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘     │
│         │                │                │             │
│  ┌──────┴────────────────┴────────────────┴──────┐     │
│  │            Storage Interface (pkg/storage)     │     │
│  └──────┬────────────────┬────────────────┬──────┘     │
└─────────┼────────────────┼────────────────┼─────────────┘
          │                │                │
   ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐
   │   NeoFS     │  │    Local    │  │     S3      │
   │   Adapter   │  │   Adapter   │  │   Adapter   │
   └─────────────┘  └─────────────┘  └─────────────┘
          │                │                │
   ┌──────┴──────┐  ┌──────┴──────┐  ┌──────┴──────┐
   │  Private    │  │ Government  │  │  Sovereign  │
   │  NeoFS Net  │  │    NAS      │  │    Cloud    │
   └─────────────┘  └─────────────┘  └─────────────┘

Data Sovereignty Compliance

Why This Matters

Governments deploying Tutus must comply with national data residency laws:

Jurisdiction	Law	Requirement
European Union	GDPR	EU citizen data must remain in EU
China	Cybersecurity Law	Critical data must be stored domestically
Russia	Federal Law 242-FZ	Personal data of citizens stored in Russia
India	DPDP Act 2023	Certain data categories require local storage
Brazil	LGPD	Similar to GDPR for Brazilian citizens

Compliance Strategy

1. Choose Compliant Storage: Select storage providers that operate within your jurisdiction
2. Private Networks: For NeoFS, deploy a private network within national borders
3. Sovereign Clouds: Use government-approved cloud providers
4. Network Isolation: Configure firewalls to prevent cross-border data transfer
5. Encryption: Enable encryption at rest and in transit

Supported Providers

NeoFS (Decentralized Storage)

Best for:

• Distributed government networks
• Multi-region deployments within a country
• High availability requirements

storage:
  provider: neofs
  neofs:
    endpoints:
      - "grpc://neofs-node1.gov.example:8080"
      - "grpc://neofs-node2.gov.example:8080"
    container: "7s23kG4xLmNpQ..."
    timeout: 30s
    key: "/etc/tutus/neofs.key"

Sovereign Deployment: Run your own NeoFS network within national infrastructure. Do not use the public NeoFS network for government data.

Local Filesystem

Best for:

• Single-node deployments
• Development and testing
• Air-gapped networks
• NAS/SAN-backed storage

storage:
  provider: local
  local:
    path: /var/lib/tutus/storage
    max_size: 1099511627776  # 1TB
    compression: true

S3-Compatible Storage

Best for:

• Sovereign cloud providers
• Existing government cloud infrastructure
• AWS GovCloud, Azure Government, etc.

storage:
  provider: s3
  s3:
    endpoint: "s3.sovereign-cloud.gov.example"
    region: "national-1"
    bucket: "tutus-blocks"
    access_key: "${S3_ACCESS_KEY}"
    secret_key: "${S3_SECRET_KEY}"
    use_path_style: true

Interface Reference

Provider Interface

The core interface that all storage backends must implement:

type Provider interface {
    // Name returns the provider identifier
    Name() string

    // Put stores data and returns a unique identifier
    Put(ctx context.Context, data io.Reader, opts PutOptions) (ObjectID, error)

    // Get retrieves data by its identifier
    Get(ctx context.Context, id ObjectID) (io.ReadCloser, error)

    // Delete removes an object from storage
    Delete(ctx context.Context, id ObjectID) error

    // Exists checks if an object exists
    Exists(ctx context.Context, id ObjectID) (bool, error)

    // List returns objects matching the given prefix
    List(ctx context.Context, prefix string, opts ListOptions) ([]ObjectInfo, error)

    // Head retrieves object metadata without content
    Head(ctx context.Context, id ObjectID) (ObjectInfo, error)

    // Close releases resources
    Close() error
}

BlockStorage Interface

Extended interface for blockchain-specific operations:

type BlockStorage interface {
    Provider

    // PutBlock stores a block with its index
    PutBlock(ctx context.Context, index uint32, data []byte) (ObjectID, error)

    // GetBlock retrieves a block by its index
    GetBlock(ctx context.Context, index uint32) ([]byte, error)

    // GetBlockRange retrieves multiple blocks efficiently
    GetBlockRange(ctx context.Context, start, end uint32) ([][]byte, error)

    // GetLatestBlockIndex returns the highest block index
    GetLatestBlockIndex(ctx context.Context) (uint32, error)
}

StateStorage Interface

Extended interface for state snapshot operations:

type StateStorage interface {
    Provider

    // PutState stores a state snapshot at a specific height
    PutState(ctx context.Context, height uint32, data io.Reader) (ObjectID, error)

    // GetState retrieves the state snapshot for a given height
    GetState(ctx context.Context, height uint32) (io.ReadCloser, error)

    // GetLatestState returns the most recent state snapshot
    GetLatestState(ctx context.Context) (height uint32, data io.ReadCloser, err error)
}

Usage Examples

Basic Usage

import (
    "github.com/tutus-one/tutus-chain/pkg/storage"
    "github.com/tutus-one/tutus-chain/pkg/storage/local"
)

// Create provider
provider, err := local.New(local.Config{
    Path: "/var/lib/tutus/storage",
})
if err != nil {
    log.Fatal(err)
}
defer provider.Close()

// Store data
id, err := provider.Put(ctx, strings.NewReader("block data"), storage.PutOptions{
    ContentType: "application/octet-stream",
    Attributes: map[string]string{
        "block_index": "12345",
    },
})

// Retrieve data
reader, err := provider.Get(ctx, id)
if err != nil {
    log.Fatal(err)
}
defer reader.Close()
data, _ := io.ReadAll(reader)

Registry Pattern

// Create registry with multiple providers
registry := storage.NewRegistry()
registry.Register(neofsAdapter)
registry.Register(localAdapter)

// Get provider by name
provider, ok := registry.Get("neofs")
if !ok {
    // Fallback to local
    provider, _ = registry.Get("local")
}

Implementing Custom Providers

To add support for a new storage backend:

1. Create a new package under pkg/storage/
2. Implement the storage.Provider interface
3. Optionally implement BlockStorage or StateStorage
4. Add configuration to storage.Config

Example structure:

pkg/storage/
├── storage.go       # Core interfaces
├── config.go        # Configuration types
├── errors.go        # Error definitions
├── neofs/           # NeoFS adapter
│   └── adapter.go
├── local/           # Local filesystem adapter
│   └── adapter.go
└── mycloud/         # Custom provider
    └── adapter.go

Security Considerations

1. Encryption at Rest: Enable storage-level encryption for sensitive data
2. Access Control: Use provider-specific ACLs to restrict access
3. Audit Logging: Enable logging for all storage operations
4. Key Management: Use HSMs or secure key storage for encryption keys
5. Network Security: Encrypt all traffic between nodes and storage

Performance Tuning

NeoFS

• Increase timeout for large objects
• Use multiple endpoints for load balancing
• Configure appropriate session_expiration_duration

Local Storage

• Use SSDs for best performance
• Enable compression for storage efficiency
• Set appropriate max_size to prevent disk exhaustion

S3

• Enable transfer acceleration where available
• Use regional endpoints for lower latency
• Configure appropriate retry policies

Monitoring

Key metrics to monitor:

Metric	Description
storage_put_duration_seconds	Time to store objects
storage_get_duration_seconds	Time to retrieve objects
storage_errors_total	Total storage errors by type
storage_objects_total	Total objects stored
storage_bytes_total	Total bytes stored

Troubleshooting

Common Issues

ErrNotFound

• Object was deleted or never stored
• Check object ID format

ErrConnectionFailed

• Storage backend is unreachable
• Check network connectivity and firewall rules

ErrQuotaExceeded

• Storage limits reached
• Increase quota or clean up old data

ErrTimeout

• Operation took too long
• Increase timeout or check network latency