ctrl by Salesforce

Salesforce has developed a large-scale language model called CTRL, which has the capability to generate text but with the added ability to control the output by applying domain, subdomain, entities, relationships between entities, dates and task-specific behaviour. The model has been trained using structure that appears naturally in raw texts, allowing users to take advantage of unsupervised learning, while having more control over the generated text. CTRL is a 1.6 billion-parameter conditional transformer language model.

The code currently supports two functions:

• Generating from a trained model, two models are available for download - one with a sequence length of 256 and another with a sequence length of 512 -- they are trained with word-level vocabularies and through a sliding window approach can generate well beyond their trained sequence lengths.
• Source attribution - given a prompt, prints the perplexity of the prompt conditional on each domain control code (see Section 5 of the paper).