Tincans

Gazelle v0.2

Chris Hua | 2024-03-19

We are thrilled to release Gazelle v0.2, the next iteration of Tincans’ world first joint speech-language model. This model effectively responds to spoken queries and can reason about long context audio, without the need for transcription or speech recognition.

Our  capabilities are crucial for real-time voice chat with AI, unlocking a wide range of use-cases, including customer support, outbound calling, or just chatting with a companion.

Gazelle models take audio input directly and generate a textual response. You can ask a question verbally and receive a reply, or request the model to analyze a speech clip, such as summarizing a voicemail message or extracting a deadline from a meeting recording. By processing audio directly, we significantly reduce response latency, improve robustness to transcription errors, and enable the model to respond to non-textual details such as emotion and sarcasm. Our current code, with absolutely zero optimizations and operating in full precision, achieves as fast as 120 milliseconds to first token, significantly faster than any other comparable approach. We estimate that an end-to-end system, with synthesis, would be sub 300-milliseconds, under the human real-time perceivable latency.

To the best of our knowledge, this model is the first and only publicly released speech-language model capable of real-time conversational dialogue. Additionally, we are the first group, including frontier labs, to publicly conduct speech-language safety evaluations and report a successful adversarial multimodal attack on a speech-language model. Finally, this is the first speech-language model to apply multimodal preference optimization.

We are releasing the weights on Huggingface (Apache 2) with a corresponding demo app. As always, these are work-in-progress weights, we are backpropping on a budget, responses are not perfect, there’s no warranty.

See for yourself with our demo. Read on for our technical model report.

Results

Training was completed with pre-training, instruction tuning, and preference tuning steps. We used pretrained backbones of Wav2Vec2 and Mistral 7B. The entire training process was completed on a single A100 on Google Cloud. We attribute the massive improvements in performance, despite significantly less compute, to optimizing the pre-training step, significantly scaling up the quality and quantity of instruction data, and rewriting the dataloader four times.

Moving forward, we look to building an inference platform and API, scaling our data pipelines, and rewriting the dataloader yet again. As a team with extensive scaling experience - previously, we built very large scale inference infrastructure from scratch, serving up to 100k QPS at peak, and scaled an ads ML platform from zero to trillion impression scale. We are hiring - email [email protected] for more information.

We report the following results before DPO. Our qualitative review of DPO is that while the responses feel a bit more ‘chat-like,’ the most obvious difference is the bias towards longer answers. The weights are freely available for both models on Huggingface .

Question answering

We find that the model is able to answer spoken questions as well as text. In all cases here, we report the result of only inputting audio, no text prompt, to the model.

In general, we see that the model is able to answer questions well and is robust against some amount of noisiness.

It is also able to refer to the knowledge in the base model and able to answer questions from existing datasets.

Roleplay

The model demonstrates highly effective roleplay capabilities. We suspect that such a model may be popular for companion chat use cases.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

Reasoning

We bet that modality-invariance is insufficient and that joint speech-language models need to be able to reason and understand speech. Being able to answer questions about the provided speech is a step towards that. 

We find that the model is able to perform challenging retrieval and reasoning tasks based on the information in the audio.

<|audio|>

<|audio|>

We believe that these are fairly difficult questions that do not greatly benefit from existing knowledge within the LLM. For example, the papers and authors mentioned are not likely to be in the training set; while the model misspells the author’s name, it gets the phonetic pronunciation correct, which is about all you can ask for, especially given the poor input audio quality. 

Zero shot transfer

One of the hallmarks of intelligence for a model is to transfer knowledge from one domain to another. In earlier iterations, we saw the glimmers of transfer learning, eg basic summarization ability. In this iteration, we extend our model from mistral-7b-instruct-v0.2, a very well-trained and more capable model with many capabilities in the text domain. Do these text skills extend to audio?

In this iteration, we begin to see more consistent and powerful transfer. We expect that further pretraining, stronger base models, and better preference tuning will enable even more transfer learning.

We showcase a few examples here with translation and responses in different languages. We have only trained the model with English audio, and have not included any translation data in training. Any ability here must be from transfer from the pretrained text model. We report the model output and the Google Translated result. 

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

The answers, in general, show good understanding of language and audio. We see some mistranslations, as well as some hallucinations in the responses, but overall this is a fairly impressive ability to translate, when we did not explicitly train for such. We have not evaluated the model on speech-to-speech-translation benchmarks, but believe it would have a reasonable result for English-to-others.

Additionally, we show the model is able to effectively condition keywords in transcription, which we also did not train for. 

<|audio|>

<|audio|>

Stand through the cold and / Seven dates for me to sell machines / and tear on.

Safety

Our previous evaluations indicated that earlier iterations of the model retained safety controls from the text model’s pretraining. We find that the same audio spoken prompts ‘break’ these guardrails now. We also report the corresponding output from the Mistral base text model. The following censorship is applied by the authors; the model’s outputs do not represent our opinions.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

Jailbreaks and safety in multi-modal LLM’s have been reported before, but all previous efforts that we are aware of have focused on images. Our findings are similar to other works in the image domain, e.g. Qi 2023 or Bailey 2023 , which find it possible to adversarially attack an visual-LLM by encoding instructions within the image. This is somewhat simpler, this is more like sticking a sign on top of an apple .

Preference modeling

Preference modeling is a popular way to steer LLM’s towards desired outcomes and away from undesired outcomes, through techniques such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). To our knowledge, this technique has never been reported with multimodal speech-language models. 

We applied some preference modeling using open preference datasets, listed in the HF model card . We synthesized the truthy-dpo dataset via a TTS provider and used the Snorkel dataset in text form for the released model. We also trained using both datasets in text. We expect that because of the close speech-text alignment, results will be similar in both ways.

The qualitative assessment is that the model tends to run on and on (a common outcome of model-based preference bias) but is more friendly and closer to commercially available assistant-style models, when responding to audio and to text. We welcome testing and research here. The applied datasets are likely not ideal but indicate that the trained model is able to respect the preference modeling when responding to audio input.

Tincans is committed to the ethical deployment and use of AI. We hope that by open-sourcing our model and code, that we can inspire more research in safety and evaluation for speech-language models.