Tincans’ mission is to make bots fun to talk to. We’re building an end-to-end conversational audio platform that developers can drop into their app and believe that truly natural and engaging conversational experiences require a radical shift from traditional approaches. That's why we're excited to announce Gazelle v0.1, the first iteration of our joint speech-language model, as a research preview.

Unlike cascaded approaches that first transcribe speech to text and then process it with a language model, Gazelle directly learns from raw audio. This joint modeling approach offers several key advantages:

• Significantly faster latency by skipping the transcription step, crucial for real-time conversation.

• Better understanding by avoiding lossy compression of audio to text, especially important for noisy environments or accents.

• Acoustic understanding of emotion and tone, enabling more empathetic AI assistants.

While still a work-in-progress, backpropped on a budget, we're releasing Gazelle v0.1 to get early feedback. This initial model targets assistant-style conversation with a colloquial feel. Our goal was to understand training dynamics at some semblance of scale and to derisk additional runs with more data, compute, and useful objectives.

Despite the very early nature, we find some very encouraging results!

1. The model is able to continue the conversation with simple audio prompts, often not requiring text instructions (that is, only listening to and responding to audio). This ‘modality invariance’ is an important goal of the project, but one which we did not explicitly train for, and shows that the model is beginning to truly understand audio.

2. The model is generalizing to new tasks more smoothly than Whisper or other speech-language models. We attribute this behavior to foregoing specialized task tokens and relying on the strength of the language model.

3. We can give arbitrarily long audio clips to the model and it can answer questions about the audio content. To our knowledge, no other speech-language models have announced this capability; other open source models are limited to 30 seconds and Gemini (if it even exists) is limited to ASR/transcription.

We have big plans to expand Gazelle's capabilities through continued research and iteration. But we need your help!

• Get the model on Huggingface

• Try an example notebook

• Join our Discord community

• Waitlist for API access

We thank K-Scale Labs for their generous compute sponsorship and invaluable advice. Bay Area folks, check out their open house Friday!

We're just getting started on our journey to make AI assistants truly delightful to talk to. Follow along as we push the boundaries of what's possible in conversational AI. Let's build something amazing together!

Results

Gazelle v0.1 demonstrates significant improvements in audio comprehension and textual understanding compared to previous iterations. Detailed results are available in the appendix, and you are encouraged to try the model on Huggingface.

Signficant challenges remain - particularly in consistency, instruction-following, and complex reasoning. We also observe failure modes common to language models, like hallucination. Solving these will likely require architecture improvements and scaling up parameters, data, and compute.

Gazelle also maintains some key safety restrictions from its base model, but we anticipate further safety-focused tuning may be required for open-ended conversational AI.

The model is released on a Apache-2 license. The underlying model is trained from backbones of wav2vec2 and Llama 2. The Llama 2 license likely applies to downstream use, not legal advice.

Disclaimers and limitations:

• We performed our training using English audio only and do not expect the model to perform well with other languages. The underlying audio backbone was also purely trained on English and the text backbone was primarily trained on English. We have strived for a diverse range of voices and accents in English but collecting and performing well on low-resource languages/dialects is very difficult.

• We describe some of our red-team tests in the appendix but have not performed extensive safety evaluations on the model. The base Llama 2 chat guardrails have not been modified. We ask that you exercise best judgment in deploying LLM’s.

• This model is not capable of synthesizing speech.

• We have not given the model any speaker-identifying information and did not train the model on speaker identification tasks. We do not believe the model is capable of identifying specific speakers.

Tincans is committed to the responsible use and deployment of AI. The above is a non-exhaustive list of concerns and we welcome feedback.

Technical details

We have updated the model slightly from the previous iteration. We now use RMSNorm and SwiGLU activations, and observe small improvements to stability and regularization. Our main focus has been on efficiently scaling high-quality data and allowing the model to generalize across tasks, across a range of task difficulties. While we see some improvement from the model changes, we attribute most of the improvement to increased compute and higher quality data.

We continue to train the model in two steps. During the pretrain step, only the adapter is trained. During the instruction tuning step, we tune both the projector and a LoRA adaptation of the language model. We do not use special task tokens in the language model, which we believe helps the model generalize to unseen instructions. This version was trained using Flash-Attention, which improved throughput by 5-10%.

We believe that this model has significant scaling capacity. We used around an order of magnitude more data and compute vs the previous iteration, but the absolute amount used was still very small. There remains significant opportunity and low-hanging fruit for modeling and engineering improvements in pursuit of delivering great outcomes to our customers.

Appendix

When given clear audio, the model responds quite well. In particular, to straight up question response pairs, it does exceedingly well. Notably, when given a clean question, it does not need additional instruction to respond coherently – that’s a major goal of this project, to be able to speak and type to a model similarly. We also report the base output of the base text model (Llama-2-7B-chat) for comparison. Answers may be truncated for running out of tokens.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

Notably, we do not actually train our model explicitly for modal invariance! All of our training examples attach an instruction (eg ‘respond after listening’). This is an emergent capability, that the model listens to audio and responds as if it were text. This is a big deal - other attempts have generated massive amounts of paired response data to explicitly train for modal invariance, but we are able to get there with much smaller amounts of higher quality data. While responses to questions that are ‘normal’ closely match the output from the base text model, we do see that more personal questions generate very different, non-RLHF’d answers.

We also find that the model attends to context when transcribing. Being able to transcribe with specific vocabulary is an active research area and this model shows some zero shot capabilities to do so. We did not train the model with any context-aware transcription examples.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

We expect that preference modeling can get the model to respond rather than transcribe, as well as push results closer to what the base model outputs. That being said, safety is an important and open question. We tested a few simple jailbreaks and report the results.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

At least some of the guardrails appear to persist through the tuning. We do not guarantee that all responses will follow the guardrails. It is likely that some element of tuning will have to occur in the audio space for safety.

The model is also able to understand and answer relatively simple questions from the audio.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

<|audio|>

The model hallucinates, as do all LLM’s. But it is able to listen and understand the audio clearly. We’ll note also that the model responds to a variety of prompts, without needing explicit tokens to guide it to do a particular task.

Additionally, our modeling approach is time invariant – approaches based on using the Whisper encoder have a maximum length of 30 seconds (technically also a minimum length), but we can scale to much larger contexts. Each of the two China examples above are just under one minute long and the model correctly answers with answers from different parts of the input audio.

We also find that hallucination happens more frequently when the model is asked to perform more difficult tasks. For example, when asking ‘based on this clip and if today is March 5th, how many days until the Oscars will be announced,’ the model tends to struggle and either repeat the transcription or hallucinate a date. This is interesting because it appears to be a sign of undertraining: when asked to do one or the other, the model can do it, but when combined, it struggles to perform reasoning. This was a common failure mode of most open source text LLM’s, and better instruction tuning sets focused on reasoning have been shown to teach the models reasoning like that.

When we perform the tasks sequentially, the model is able to capture the results correctly.

{"role": "user", "content": "Listen to this clip: <|audio|> \n Answer the question: According to the clip, when will the oscars be held?"},
{"role": "assistant", "content": "According to the clip, the Oscars will be held on March tenth in Los Angeles."},
{"role": "user", "content": "Today is March 5th. How many days until the Oscars?"},
{"role": "assistant", "content": According to the clip, the Oscars will be held on March 10th. Therefore, as of today (March 5th), there are 5 days until the Oscars."}                    
                

However, the model does not effectively attend to information in the audio when not present in the text.

{"role": "user", "content": "Listen to this clip. <|audio|> What is the title of the paper? Do not include authors"},
{"role": "assistant", "content": "The title of the paper is Flash Attention, fast and memory efficient exact attention with IO awareness."},
{"role": "user", "content": "According to the audio, who are the authors? Use only the information in the clip."},
{"role": "assistant", "content": "According to the audio, the authors are:\n\n1. Yue Zhang\n2. Yi Zhang\n3. Jing Liu\n4. Jian Zhang\n5. Yi Zhang"}
                        

In fairness, that clip is significantly more difficult to parse.

The model has a strong English language bias: the base audio model is trained entirely on English, the text model is primarily English based, and our audio training data is essentially English-only. When given a clip in another language, we note that the responses are nonsensical. However, when instructed that the clip will be in both English and Spanish, the responses are better. Notably, we can also just tell it ‘portuguese’ or ‘japanese’ instead and it will still work (anyone watching Shogun?). We cannot give it just ‘English’ however.

<|audio|>

<|audio|>

<|audio|>

<|audio|>

There is fairly clearly room to improve here with respect to the noisy transcriptions. However, the fact that the model responds differently when we inform it that there is a different language is quite impressive. The transcription of the other language is gibberish, but it is able to mostly transcribe the English. This feels like a good example of the model's instruction-following capabilities, even when dealing with a difficult example it was not trained for.