88hours commited on
Commit
04d7dde
·
1 Parent(s): f97d782

Update Readme with chatGPT recommended text

Browse files
Files changed (1) hide show
  1. README.md +66 -40
README.md CHANGED
@@ -1,53 +1,79 @@
1
- # Journey into learning - 4:00 pm
2
- https://learn.deeplearning.ai/courses/multimodal-rag-chat-with-videos/lesson/2/interactive-demo-and-multimodal-rag-system-architecture
3
- - A MULTIMODEL AI SYSTEM SHOULD BE ABLE TO UNDERSTAND TEXT, VIDEO CONTENT
4
 
5
- ## Step 1 - learn Gradio (UI): 30 mins
6
- Great for making quick UI in python, that will run in browser. It also has hot reloading.
7
 
8
- fn: The function to wrap a user interface (UI) around
9
- inputs: the Gradio component(s) to use for the input. The number of components should match the number of arguments in your function.
10
- outputs: the Gradio component(s) to use for the output. The number of components should match the number of return values from your function.
11
 
12
- https://www.gradio.app/docs/gradio/introduction
13
 
14
- Gradio includes more than 30 built-in components
15
 
16
- Tip:
17
- For the `inputs` and `outputs` arguments, you can pass in the name of these components as a string (`"textbox"`) or an instance of the class (`gr.Textbox()`).
18
 
19
- ## Sharing Demo
20
- demo.launch(share=True) # Share your demo with just 1 extra parameter.
21
- ### Why did not hot reloading worked?
 
22
 
 
23
 
 
24
 
25
- ## Gradio.Blocks
26
- Gradio offers a low-level approach for designing web apps with more customizable layouts and data flows with the gr.Blocks class. Blocks supports things like controlling where components appear on the page, handling multiple data flows and more complex interactions (e.g. outputs can serve as inputs to other functions), and updating properties/visibility of components based on user interaction — still all in Python.
 
27
 
28
- ## Gradio.ChatInterface
29
- Always set type="messages" in gr.ChatInterface. The default value (type="tuples") is deprecated and will be removed in a future version of Gradio.
30
- Better to use gr.ChatBot for more ui options
31
- gr.ChatInterface also supports markdown (I did not try it thou)
32
- ## Step 2 Learn Bridge Tower Embedding Model (MultiModel Learning Space): 15 mins
33
- Made by Collab at Intel
34
- 512- Dimenstions, so mean each image,caption pair has been coverted into a 512 dimension vector
35
- ### Measuing distance
36
- Cosine Similarity to see how much images are far and near to each other - Cheap Computing, Commanly used
37
- Euclidean Distance using CV vision call by passing two images with cv2.NORM_L2
38
- ### Conversion to 2D for display purposes
39
- UMAP to transfer 512 vector dimension embeddings to 2 dimension embeddings
40
 
41
- ## Preprocessing Videos for Multimodal RAG¶
42
- ### Case 1: WEBVVT => Text Segment from video
43
- - Convert video and text into metadata, by dividing into multiple segment
44
 
45
- ### Case 2: Whisper(small) => Video Only
46
- - Extract Audio -> model.transcribe
47
- - Apply getSubs() helper function to get WEBVVT
48
- - Apply Case 1
49
 
50
- ### Case 3: Lvlm => Video + silent/music (extract from video )
51
- - Lava
52
- - Encode each frame as a base64 image
53
- - Pass it to Lava (LvLM model) to get captions and get the context
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Journey into Learning - 4:00 PM
 
 
2
 
3
+ [**Interactive Demo and Multimodal RAG System Architecture**](https://learn.deeplearning.ai/courses/multimodal-rag-chat-with-videos/lesson/2/interactive-demo-and-multimodal-rag-system-architecture)
 
4
 
5
+ ### A multimodal AI system should be able to understand both text and video content.
 
 
6
 
7
+ ---
8
 
9
+ ## Step 1 - Learn Gradio (UI) (30 mins)
10
 
11
+ Gradio is a powerful Python library for quickly building browser-based UIs. It supports hot reloading for fast development.
 
12
 
13
+ ### Key Concepts:
14
+ - **fn**: The function wrapped by the UI.
15
+ - **inputs**: The Gradio components used for input (should match function arguments).
16
+ - **outputs**: The Gradio components used for output (should match return values).
17
 
18
+ 📖 [**Gradio Documentation**](https://www.gradio.app/docs/gradio/introduction)
19
 
20
+ Gradio includes **30+ built-in components**.
21
 
22
+ 💡 **Tip**: For `inputs` and `outputs`, you can pass either:
23
+ - The **component name** as a string (e.g., `"textbox"`)
24
+ - An **instance of the component class** (e.g., `gr.Textbox()`)
25
 
26
+ ### Sharing Your Demo
27
+ ```python
28
+ demo.launch(share=True) # Share your demo with just one extra parameter.
29
+ ```
 
 
 
 
 
 
 
 
30
 
31
+ ### Why Didn’t Hot Reloading Work?
32
+ (Investigate potential caching issues, missing dependencies, or incorrect function signatures.)
 
33
 
34
+ ---
 
 
 
35
 
36
+ ## Gradio Advanced Features
37
+
38
+ ### **Gradio.Blocks**
39
+ Gradio provides `gr.Blocks`, a flexible way to design web apps with **custom layouts and complex interactions**:
40
+ - Arrange components freely on the page.
41
+ - Handle multiple data flows.
42
+ - Use outputs as inputs for other components.
43
+ - Dynamically update components based on user interaction.
44
+
45
+ ### **Gradio.ChatInterface**
46
+ - Always set `type="messages"` in `gr.ChatInterface`.
47
+ - The default (`type="tuples"`) is **deprecated** and will be removed in future versions.
48
+ - For more UI flexibility, use `gr.ChatBot`.
49
+ - `gr.ChatInterface` supports **Markdown** (not tested yet).
50
+
51
+ ---
52
+
53
+ ## Step 2 - Learn Bridge Tower Embedding Model (Multimodal Learning) (15 mins)
54
+
55
+ Developed in collaboration with Intel, this model maps image-caption pairs into **512-dimensional vectors**.
56
+
57
+ ### Measuring Similarity
58
+ - **Cosine Similarity** → Measures how close images are in vector space (**efficient & commonly used**).
59
+ - **Euclidean Distance** → Uses `cv2.NORM_L2` to compute similarity between two images.
60
+
61
+ ### Converting to 2D for Visualization
62
+ - **UMAP** reduces 512D embeddings to **2D for display purposes**.
63
+
64
+ ## Preprocessing Videos for Multimodal RAG
65
+
66
+ ### **Case 1: WEBVTT → Extracting Text Segments from Video**
67
+ - Converts video + text into structured metadata.
68
+ - Splits content into multiple segments.
69
+
70
+ ### **Case 2: Whisper (Small) → Video Only**
71
+ - Extracts **audio** → `model.transcribe()`.
72
+ - Applies `getSubs()` helper function to retrieve **WEBVTT** subtitles.
73
+ - Uses **Case 1** processing.
74
+
75
+ ### **Case 3: LvLM → Video + Silent/Music Extraction**
76
+ - Uses **Llava (LvLM model)** for **frame-based captioning**.
77
+ - Encodes each frame as a **Base64 image**.
78
+ - Extracts context and captions from video frames.
79
+ - Uses **Case 1** processing.