RALE: RAJ-Authorizing Logical Endpoint

Sequel to The Boy Who Ate The Menu (RAJ/TAJ as Software-Defined Addressing for Universal Storage Locators)

The Problem

Organizations need to share datasets without exposing infrastructure:

  • Clients shouldn’t know bucket names, regions, or AWS accounts
  • Access should be fine-grained (specific packages, not entire buckets)
  • Authorization should be efficient (not re-evaluated per file)
  • Data locations should change without breaking client code

Traditional S3 approaches fail:

  • Bucket policies: Too coarse, expose bucket names
  • IAM roles: Grant bucket-level access, complex cross-account setup
  • Presigned URLs: Individual objects only, no collections
  • API gateways: Policy evaluation on every request adds latency and cost

The Solution

RALE (RAJ-Authorizing Logical Endpoint) virtualizes S3 access using logical addresses instead of physical coordinates.

Clients use USLs like registry/my/package@hash/data.csv without knowing the physical bucket, region, or key structure. RALE translates logical requests to physical S3 locations using Quilt package manifests.

Authorization happens once via RAJ tokens – cryptographically signed tokens containing compiled authorization. No policy lookups on subsequent requests. RALE uses TAJs (Translated Access JWTs), which are RAJs granting authority over specific package manifests.

RALE extends RAJEE (RAJ Envoy Endpoint) by adding Authorizer and Router Lambda functions for manifest-based translation.

Current scope: Read-only (s3:GetObject)

Request Flow

Processing

  1. RALE receives request
    • Intercepts S3 API call via Envoy filter
    • Extracts TAJ from Authorization: Bearer header
    • Routes to Authorizer Lambda (no/expired TAJ) or Router Lambda (valid TAJ)
  2. Authorizer Lambda
    • Validates user identity
    • Pins package to specific manifest hash
    • Calls AVP for authorization decision
  3. Amazon Verified Permissions (AVP)
    • Maintains access grants using Cedar policy language
    • Returns yes/no decision
  4. Authorizer mints TAJ (on approval)
    • Creates TAJ with encrypted RAJ, manifest hash, and logical scope
    • Caches by manifest_hash:user_id
    • Returns to Router Lambda
  5. Router Lambda
    • Validates TAJ signature and scope
    • Retrieves manifest by hash (cached indefinitely – manifests are immutable)
    • Proves membership: does logical key exist in manifest?
    • Resolves destination: extracts physical S3 URI with versionId
    • Rewrites request with physical coordinates
    • Signs with AWS SigV4 and forwards to S3
  6. S3 returns object

Key Concepts

Manifest-Scoped Authorization: Policy evaluation happens once at TAJ issuance. Router Lambda only validates membership—no policy decisions at request time.

  • Traditional: 1000 files = 1000 authorization calls
  • RALE: 1000 files = 1 authorization + 1000 validations

Manifest Pinning: Every TAJ references a specific immutable manifest hash. No floating authority as packages evolve. Old TAJs use old manifests.

Two-Lambda Architecture: Authorizer (security) and Router (infrastructure) separate concerns. Router makes no policy decisions, enabling aggressive caching and horizontal scaling.

Trade-offs

Advantages

  • Location transparency: clients never see physical infrastructure
  • Efficient authorization: single policy check per package
  • S3 API compatible: drop-in replacement
  • Cross-infrastructure: collections span buckets, regions, accounts
  • Immutable semantics: manifest pinning ensures consistency
  • Clean separation: security (RAJA), enforcement (RALE), routing (manifests)

Disadvantages

  • Latency: two Lambda invocations per request
  • Complexity: multiple coordinated components
  • Cost: Lambda invocations and data transfer
  • Availability: single point of failure

Comparison to Alternatives

S3 Bucket Policy

  • Authorization: Coarse (bucket-level)
  • Location Transparency: ❌ Exposes buckets
  • Multi-Bucket: ❌ No
  • Performance: Fast

IAM Roles

  • Authorization: Coarse (bucket-level)
  • Location Transparency: ❌ Exposes buckets
  • Multi-Bucket: ❌ No
  • Performance: Fast

Presigned URLs

  • Authorization: Fine (per-object)
  • Location Transparency: ✅ Opaque
  • Multi-Bucket: ✅ Yes
  • Performance: Fast

API Gateway + Lambda

  • Authorization: Fine (custom)
  • Location Transparency: ✅ Hides backend
  • Multi-Bucket: ✅ Yes
  • Performance: Slow (PDP per request)

RALE (TAJ)

  • Authorization: Fine (manifest-scoped)
  • Location Transparency: ✅ Fully logical
  • Multi-Bucket: ✅ Yes
  • Performance: Medium (compiled auth)

Glossary

1. Amazon Verified Permissions (AVP)

AWS service for policy management using Cedar language. Maintains access grants and returns yes/no authorization decisions. RAJA calls AVP to evaluate package-level access.

2. Authorizer Lambda

RALE component that mints TAJs. Calls RAJA/AVP for policy decisions and caches TAJs by manifest_hash:user_id.

3. Manifest

Immutable Quilt package manifest mapping logical keys to physical S3 URIs:

  • Logical keys: data/customers.csv (user-facing names)
  • Physical URIs: s3://bucket/hash/customers.csv?versionId=abc123 (actual locations)

Content-addressed by hash. Both manifests and S3 objects use versioning for referential integrity.

4. Manifest Pinning

Every TAJ references a specific manifest hash:

  • No floating semantics as packages evolve
  • Deterministic, cacheable resolution
  • Old TAJs use old manifests (intentional version skew)

5. RAJ (Resource Access JWT)

Compiled authorization token containing cryptographic proof of access. Authorization evaluated once at issuance, not per request.

6. RAJA (RAJ Authority)

Control plane service that mints TAJs:

  • Receives authorization requests with USLs
  • Evaluates policy using AVP
  • Pins package names to manifest hashes
  • Returns signed TAJs

7. RAJEE (RAJ Envoy Endpoint)

Foundational Envoy filter that enforces RAJs. RALE extends RAJEE by adding Authorizer and Router Lambda architecture for manifest-based translation.

RAJEE provides:

  • S3 API interception with TAJ authentication
  • TAJ signature validation
  • Request routing to Lambda functions

RALE adds:

  • Authorizer Lambda for TAJ minting
  • Router Lambda for manifest translation
  • AVP integration
  • Manifest pinning and logical key resolution

See RAJEE Against the Machine and TAJ Mahalo.

8. Router Lambda

RALE component that validates and routes requests:

  • Validates TAJ signature and scope
  • Fetches manifest by hash (cached – manifests are immutable)
  • Proves set membership
  • Resolves physical destination
  • Rewrites and signs S3 request

Performs deterministic proof, not policy decision. Authorization already happened at TAJ issuance.

9. TAJ (Translated Access JWT)

RAJ granting authority over a specific manifest revision:

  • Contains encrypted RAJ with manifest hash (not physical mappings)
  • Specifies logical scope within manifest
  • Carries authority over the routing table, not the mappings

10. USL (Uniform Storage Locator)

Logical identifier hiding infrastructure topology:

registry/package_name[@hash]/logical_key

Examples:

  • registry/genomics/rnaseq-001/results.csv (latest)
  • registry/genomics/rnaseq-001@abc123/results.csv (specific version)

registry refers to a Quilt registry – an S3 bucket or local directory storing package manifests and data.

References

Leave a comment

Blog at WordPress.com.

Up ↑