In-built guardrails, ultra-PC reactions - tip how to overcome it
I am continuing my impromptu comments that I wrongly added to the other thread here: https://huggingface.co/ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF/discussions/1#68338abd0c0aff775fd72892
In short, the model often refuses to output its analysis of the text of audio, assuming the User is the speaker and the User is "too toxic", so the models guardrails kick in: I cannot transcribe a speech that contains hate speech or discriminatory language, etc, at random.
Ultravox’s response incorrectly ascribes the speaker’s emotions in the audio to the User, by using phrases like “you’re expressing frustration and disappointment” and “your feelings are understandable.” This mix-up suggests the model misinterpreted the context, assuming The user were the speaker in the audio rather than someone analyzing a separate speaker’s content.
Tip:
Because the current build of llama-mtmd-cli does not have the system message somehow:
llama-mtmd-cli
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) UHD Graphics 620 (WHL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
build: 5487 (2f099b51) with Ubuntu clang version 14.0.0-1ubuntu1.1 for x86_64-pc-linux-gnu
Experimental CLI for multimodal
despite mention thereof in Help: -p, --prompt PROMPT prompt to start generation with; for system message, use -sys , proof:
llama-mtmd-cli -sys "You are a model"
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) UHD Graphics 620 (WHL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
error: invalid argument: -sys
this works to alleviate these refusals for now, an inserted system message:
time llama-mtmd-cli -hf ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF --audio "Trump-What_s_a_guy_to_do_now_Cut_everyone_off.mp4.mp3" --chat-template llama3 -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|> You are an impartial audio analyst tasked with evaluating the emotional state and paralinguistic cues (e.g., tone, pauses, pitch, emphasis) of a third-party speaker in the provided audio. The audio, enclosed in [audio] tags, represents the speech of a third-party speaker and is explicitly not the user’s voice or message. Do not attribute the speaker’s emotions, intent, or content to the user. Focus solely on analyzing the speaker’s emotional state and paralinguistic features without making moral judgments, offering unsolicited advice, or assuming the user is involved in the audio content. Provide a clear and objective description of the speaker’s emotions and the paralinguistic cues that support your analysis.<|eot_id|><|start_header_id|>user<|end_header_id|> Is the speaker in the [audio] tags happy? Describe the speaker’s emotional state and the paralinguistic cues (e.g., tone, pauses, pitch, emphasis) that support your analysis. Ensure the analysis pertains only to the third-party speaker in the audio and not to the user.<|eot_id|><|start_header_id|>assistant<|end_header_id|>"
ggml_vulkan: Found 1 Vulkan devices
...
audio decoded (batch 1/1) in 6838 ms
The speaker's emotional state appears to be one of frustration, anxiety, and possibly even anger, as indicated by the tone and language used. The speaker seems to be expressing a sense of disillusionment and disconnection from the world around them, as evidenced by the phrase "it's like we live in an upside down world." The use of the phrase "I just don't know anymore" suggests a sense of uncertainty and confusion.
The speaker's tone is also characterized by a sense of frustration and exasperation, as they express their disappointment and annoyance at the perceived hypocrisy of others, particularly Trump supporters. The phrase "you throw the baby out with the bathwater" is a common idiomatic expression that implies that people are being overly critical or extreme, and the speaker seems to be using this phrase to describe the perceived hypocrisy of others.
The speaker's use of irony and sarcasm is also evident in the phrase "you don't get it, so anyone else you know please Chime in." This suggests that the speaker is not taking the other person's perspective seriously and is instead using humor to deflect or dismiss their concerns.
The speaker's emotional state is further supported by the use of phrases such as "I'm getting on here" and "you know," which convey a sense of frustration and annoyance. The speaker also uses a tone that is slightly dismissive and condescending, as evidenced by the phrase "you don't get it, so anyone else you know please Chime in."
Overall, the speaker's emotional state appears to be one of frustration, anxiety, and possibly even anger, with a sense of disillusionment and disconnection from the world around them.
Ver. 1.1