hybrid format + llm + keywoard, + 4000, 30

old version with fixed, 3000, 30

added the new llm query expanding + 4000,30 + latin to cyrilic

added the new llm query expanding

from cry to latin + 2000,30

new steel latin to crylic

max chars = 4500 + max rows =50, topk 60+ 60 = 120, cut off = 0.45

max chars = 3000 + max rows =10, topk 60+ 60 = 120, cut off = 0.45

max chars = 3000 + max rows =20, topk 60+ 60 = 120, cut off = 0.55

max chars = 3000 + max rows =25, topk 60+ 60 = 120, cut off = 0.55

max chars = 3000 + max rows =20, topk 60+ 60 = 120, cut off = 0.65

max chars = 4000 + max rows =20, topk 60+ 40 = 100, cut off = 0.7

max chars = 2000 + max rows =20, topk 50+ 50 = 100, cut off = 0.75

max chars = 2000 + max rows =20, topk 50+ 50 = 100, cut off = 0.75

max chars = 5000 + max rows =20, topk 50+ 50 = 100, cut off = 0.65

max chars = 1500 + max rows =15, topk 50+ 50 = 100, cut off = 0.55

max chars = 3000 + max rows =15, topk 80+ 80 = 120, cut off = 0.55

max chars = 2500 + max rows =15

max chars = 2500 + max rows =25

max chars = 5000 + max rows =25

new normalizer C to Latin C + max table = 20, max chunk = 4000

new normalizer C to Latin C + max table = 20, max chunk = 3000

old utils + topk 50 + 50, hybrid = 100, cut off = 0.45

max chunk size =3000+ max row =20

max chunk size =4000+ max row =20

max chunk size = 3000

added the normalization of functions + max chars = 2500+ max rows = 15

new normalizing + max chars 3000 max rows =15

chunk size = 3000, 15 + normalized query

query normalization

normalize anyways + max row = 15 + max chars = 3000

new debug functions + 2000 chunk size

top k 200, 150 + max chunk size = 10 000, max chunk row = 40

top k = 150

top k = 100 + max chunk size is 10000 + max rows =40 + sim cut off = 0.55

top k = 100 + topk rerank = 20 + max chunk size is 2000 + max rows =10 + sim cut off = 0.55

top k = 70 + topk rerank = 20 + max chunk size is 4000 + max rows =2 + sim cut off = 0.55

top k = 50 + topk rerank = 20 + max chunk size is 4000 + max rows =30 + sim cut off = 0.25

top k = 100 + topk rerank = 30 + max chunk size is 1024 + max rows =5 + sim cut off = 0.25

top k = 120 + max chunk size is 4000 + max rows =5 + sim cut off = 0.35

top k = 100 + max chunk size is 6000 + max rows =30 + sim cut off = 0.45

top k = 100 + max chunk size is 512 + max rows =3

top k = 100 + max chunk size is 5000 + max rows =30

top k = 100 + max chunk size is 1024 + max rows =10

top k = 100 + max chunk size is 3000 + max rows =15

top k = 80 + max chunk size is 3000

new api = retrieve chunks + some more text fixing

new documents_prep

new way of chunking

a new chunking with 3 000 size for all types of data