Features¶

Compatibility Matrix¶

The tables below show mutually exclusive features and the support on some hardware.

The symbols used have the following meanings:

✅ = Full compatibility
🟠 = Partial compatibility
❌ = No compatibility
❔ = Unknown or TBD

Note

Check the ❌ or 🟠 with links to see tracking issue for unsupported feature/hardware combination.

Feature x Feature¶

Feature	CP	APC	LoRA	SD	CUDA graph	pooling	enc-dec	logP	prmpt logP	async output	multi-step	mm	best-of	beam-search
CP	✅
APC	✅	✅
LoRA	✅	✅	✅
SD	✅	✅	❌	✅
CUDA graph	✅	✅	✅	✅	✅
pooling	🟠*	🟠*	✅	❌	✅	✅
enc-dec	❌	❌	❌	❌	✅	✅	✅
logP	✅	✅	✅	✅	✅	❌	✅	✅
prmpt logP	✅	✅	✅	✅	✅	❌	✅	✅	✅
async output	✅	✅	✅	❌	✅	❌	❌	✅	✅	✅
multi-step	❌	✅	❌	❌	✅	❌	❌	✅	✅	✅	✅
mm	✅	✅	🟠^{^}	❔	✅	✅	✅	✅	✅	✅	❔	✅
best-of	✅	✅	✅	❌	✅	❌	✅	✅	✅	❔	❌	✅	✅
beam-search	✅	✅	✅	❌	✅	❌	✅	✅	✅	❔	❌	❔	✅	✅

* Chunked prefill and prefix caching are only applicable to last-token pooling.
^{^} LoRA is only applicable to the language backbone of multimodal models.

Feature x Hardware¶

Feature	Volta	Turing	Ampere	Ada	Hopper	CPU	AMD	TPU
CP	❌	✅	✅	✅	✅	✅	✅	✅
APC	❌	✅	✅	✅	✅	✅	✅	✅
LoRA	✅	✅	✅	✅	✅	✅	✅	✅
SD	✅	✅	✅	✅	✅	✅	✅	❌
CUDA graph	✅	✅	✅	✅	✅	❌	✅	❌
pooling	✅	✅	✅	✅	✅	✅	✅	❌
enc-dec	✅	✅	✅	✅	✅	✅	❌	❌
mm	✅	✅	✅	✅	✅	✅	✅	❌
logP	✅	✅	✅	✅	✅	✅	✅	❌
prmpt logP	✅	✅	✅	✅	✅	✅	✅	❌
async output	✅	✅	✅	✅	✅	❌	❌	❌
multi-step	✅	✅	✅	✅	✅	❌	✅	❌
best-of	✅	✅	✅	✅	✅	✅	✅	❌
beam-search	✅	✅	✅	✅	✅	✅	✅	❌

Note

Please refer to Feature support through NxD Inference backend for features supported on AWS Neuron hardware