Finetuning tinyBERT with 4.4M parameters

Copilot and I successfully did our first finetuning exercise on tinyBERT, an old transformer model with only 4.4M parameters.

We are training it to recognize data structures and to convert one structure, say S-expression format, to another, say JSON. It doesn't need to even speak English. All it needs to do is to see the field name/key and values of two different data structures. We can write a backend server and wire its output to a backend for further processing. The next finetuning attempt will be to let tinyBERT extract some key data from emails.

(person (name "Alice") (age "42") (city "Tokyo") (country "Japan")) || { "name": [MASK], "age": "42", "city": "Tokyo", "country": "Japan" } (person (name "Alice") (age "42") (city "Tokyo") (country "Japan")) || { "name": "Alice", "age": [MASK], "city": "Tokyo", "country": "Japan" } (person (name "Alice") (age "42") (city "Tokyo") (country "Japan")) || { "name": "Alice", "age": "42", "city": [MASK], "country": "Japan" } ...

In the future, humans no longer need SQL queries to retrieve any information. You only need a small LLM worker that can recognize patterns.