LoRA multi-tenant inference serving

Design reference · technical review pending. This UAT expansion describes a proposed serving pattern. It is not a tested deployment or a performance benchmark. Validate model compatibility, isolation, and load behavior before adopting it.

Problem

Multi-tenant GenAI systems often need customization without duplicating the full model per tenant. A shared base-model pool with approved LoRA adapters is one possible design, but adapter loading, tenant authorization, and noisy neighbors need explicit operational controls.

When to use this

• Tenants can use compatible adapters on the same base model.
• The service has a trusted operator who controls model and adapter releases.
• Sharing capacity is acceptable within the required isolation boundaries.

When not to use this

Use a different design when tenants require incompatible base models, strict dedicated hardware, or arbitrary untrusted adapter uploads. A shared model process is not a security boundary between tenants. For a small workload, a simpler dedicated deployment may be easier to operate.

Architecture

Only the gateway is client-facing. A trusted release process validates and stores versioned adapter artifacts, then makes approved versions available to the private serving pool. Metrics and audit events cover the complete path. Streaming responses return through the gateway.

Components

• Gateway: authenticates callers and enforces request-size, concurrency, and rate limits.
• Tenant policy registry: maps an authenticated tenant to approved adapter versions. Client input cannot override this mapping.
• Serving workers: run the selected base-model revision and compatible adapters. vLLM is one possible runtime; verify its adapter support for the chosen model and release.
• Artifact store and release process: retain approved artifacts and the identifiers needed for rollback.
• Telemetry: correlates gateway decisions, worker behavior, and tenant-visible outcomes without storing prompt content by default.

Request flow

1. Authenticate the caller and resolve the tenant from trusted identity data.
2. Apply the tenant policy, request limits, and an end-to-end deadline.
3. Resolve the approved base model and adapter version server-side.
4. Select a ready worker able to serve that combination. Queue within a bound or return an explicit capacity error.
5. Forward the request, stream output, and propagate cancellation when the client disconnects.
6. Record the selected version, outcome, latency, and usage. Release concurrency capacity on success, failure, or cancellation.

Deployment assumptions

The team operates a private serving network and an authenticated gateway. Adapter artifacts come from a trusted release pipeline. Hardware capacity, supported model architecture, adapter rank, context limits, and request distribution must be specified for the chosen deployment. No GPU sizing or tested runtime version is claimed here.

Operational characteristics

Publish immutable model and adapter versions. Keep new workers out of routing until loading and a representative smoke request succeed. Drain in-flight requests before replacing workers. Roll back the policy mapping and compatible worker release together; retain the previous artifacts until the change is accepted.

Scaling considerations

Scale against a measured bottleneck: queue delay, sustained request concurrency, memory pressure, or missed latency objectives. Extra replicas need time to load their artifacts. Decide whether frequently used adapters stay warm and how less-used adapters enter service. Test a traffic burst from one tenant while observing the others. Kubernetes autoscaling can use custom metrics, but the metric pipeline and readiness behavior must be configured.

Observability

Track client TTFT, completion latency, errors, queue time, request cancellation, token counts, and adapter load failures. Correlate traces with a bounded tenant identifier and model/adapter version. Keep high-cardinality tenant detail in controlled logs or a dedicated analytics system rather than attaching every tenant to every metric. Define separate views for warm requests, cold loads, and rejected traffic.

Failure modes

• Adapter unavailable: fail clearly or route only to an explicitly approved compatible fallback. Do not silently substitute another tenant’s adapter.
• Memory exhaustion: reduce admitted work, drain or restart unhealthy workers, and review context and adapter residency limits.
• Noisy neighbor: enforce admission limits before the shared queue and consider dedicated capacity for demanding tenants.
• Partial streamed response: report the interrupted result; a blind retry can duplicate work or change the answer.
• Registry or artifact-store outage: define whether already approved warm versions can continue serving and when new admissions must stop.

Security considerations

Keep runtime management endpoints private. Dynamic LoRA loading is a privileged operation; vLLM’s documentation warns against exposing it to untrusted clients. Let a trusted release process control loading. Enforce tenant-to-adapter authorization at the gateway, restrict artifact access, and redact sensitive payloads from logs. See the vLLM security guidance.

Cost considerations

Measure GPU time, idle warm capacity, artifact storage, transfer, gateway services, and observability overhead. Separate the cost of successful tenant requests from failed work and cold loads. Compare with dedicated serving under the same traffic and quality requirements. Sharing capacity may reduce duplication but does not establish a numerical saving without measurements.

Tradeoffs

A shared pool can simplify model management and reduce duplicated capacity. It also couples tenants through shared resources and makes admission control, routing, and release coordination more important. Warm adapters consume capacity; aggressive eviction can increase loading delays. Choose the balance using the actual tenant mix.

Alternatives

• Dedicated tenant deployments: clearer resource boundaries and independent changes, with more capacity and fleet-management overhead.
• Shared base model without adapters: a simpler serving path when prompting or retrieval is sufficient.
• Managed inference service: less infrastructure ownership, subject to its isolation, adapter, privacy, and cost constraints.

Example implementation

This pseudocode illustrates the gateway contract, not a deployable runtime configuration. Implement authentication, distributed admission control, cancellation, and streaming with your chosen infrastructure.

tenant = authenticate(request.credentials)
policy = registry.approved_policy(tenant)
validate_input(request, policy.limits)
lease = admission.acquire(tenant, deadline=request.deadline)
try:
    worker = pool.ready_worker(policy.base_model, policy.adapter_version)
    stream(worker.generate(
        model=policy.adapter_alias,  # selected by policy, not caller input
        prompt=request.prompt,
        limits=policy.limits,
        deadline=request.deadline
    ), cancel_on_disconnect=True)
finally:
    lease.release()

Before implementation, verify the runtime interface using the official LoRA adapter examples. Test unauthorized adapter selection, unavailable adapters, saturation, client cancellation, and release rollback. Record the deployment-specific versions and outcomes alongside the Blueprint.

References

• vLLM: LoRA adapters
• vLLM: security considerations
• vLLM: production metrics
• Kubernetes: horizontal pod autoscaling

Contributors and reviewers

The original author’s identity is withheld in this anonymized UAT copy. The expanded design guidance has not received a recorded practitioner review. Add verified contributor names, affiliations, reviewers, review date, and validation evidence before presenting it as a reviewed deployment reference.