toy transformer

@vinhowe @grantpitt github

train loss: 0.0000

val accuracy: 0.0000

Dataset

Task

Learn to sort a sequence of characters

Sequence Length

2

Number of items to sort

Max Number

26

Maximum value in the sequence

Mask Out Prefix

Include Commas

Include Colon

Model Architecture

Number of Transformer Layers

1

Number of Attention Heads

6

Embedding Size

16 per head

Activation Function

Positional Encoding

LayerNorm Position

Attention-only (no MLP layers)

Optimizer

Optimizer Type

Learning Rate

1.00e-5

10^-5

10^-4

10^-3

10^-2

10^-1

1

Learning Rate Scheduler

Training Options

Batch Size

2

Label Smoothing

1.00e-5

10^-5

10^-4

10^-3

10^-2

10^-1

1

Random Seed (optional)

Caching Enabled