towhee.models.perceiver.create_self_attention.create_self_attention

towhee.models.perceiver.create_self_attention.create_self_attention(num_channels: int, num_heads: int, dropout: float, activation_checkpoint: bool = False)[source]

Self attention block for Perceiver https://arxiv.org/pdf/2103.03206.pdf.

Parameters:

  • num_channels (int) – Number of q channels.

  • num_heads (int) – Number of parallel attention heads.

  • dropout (nn.Module) – Dropout probability.

  • activation_checkpoint (bool) – Use activation checkpointing.

Return (nn.Module):

Configured self attention layer.