LangKit

LangKit is an open-source text metrics toolkit developed by WhyLabs for monitoring language models. It includes a variety of methods for obtaining important signals from input and output text, which can be used along with the open-source data logging library whylogs.

When deploying language models, such as LLMs, there is a potential for various risks due to the infinite amount of possible input combinations and the resulting infinite number of outputs. The unstructured nature of text presents a challenge in terms of ML observability. LangKit is designed to tackle this challenge, as not having visibility into the model's behavior can lead to serious consequences.

Source: https://github.com/whylabs/langkit