could you release the lora for this model ?
i would like to apply this to my model , to apply the new layers you created they would me in the unmerged verison ?
or a jupyter notebook for creating the model from an exisiting mistral ? .... to train it we need a collection of problems that are solved with these thoughts ... ie this model would have to generate the examples during conversations .. so generate a set of conversations with the model to make some data to program the model .... eg.... maybe!.. but nice work still
liminerity/Mistral-quiet-star-demo-lora
i dont wanna release the notebook yet but i will. It'll be like a "lazymergekit" sorta thing where you will be able to apply this to any model. Im working on a lot of things as far as this goes but it is hard work lol.
i know thanks it was very confusing , but one project leads to another .
what you have done is add a mini layer ... its hard to weave the layer in without the lora...or trainer as it also adds some tokens to the tokenizer. this is quit an important project bro ! as this process is internal mind process, ie it could also be a self rag process. augmenting its thinking with the task performing the logic in its Mindspace (ie prompt: generate expert agents to perform a task discuss the task requirements between agents and produce a solution between the team of agents performing any required testing and reformulations before presenting the output" - This can be trained by using open-interpreter for a while and capturing all the logs, and using it as finetuning data... (ie how agents communicate and solve a task together) that would be the context and the output be the result....
also the technique of adding the head(we can use unsloth to merge the lora and mistral no probs) ... is also important technique as i have the inkling to add the whisper tokenizer replacing the normal text tokenizer in my models but how to do this .... it would also have to be created in the lora (replacing the lmhead? with the new adapter (openwhisper)... creating the audio to audio llm (lol)... as well as these models being able to exchange this tokenizer for a image head or document parser head... as it is only this input process ... slightly customized tokenizer , and im sure we can create a multi input head!.... so essentially the tokenizer would choose its own tokenization logic based on the tags found in the input! telling it to recieve an image or sound... (essentially all is sent to a text(even the image is some binary encoded text)
i recently fine tuned a model which not only codes but will play any role sexy or not, angry or happy or sad..as well as still perform tasks and write summaries and essays ... as well as answer cyber hacking questions and medical questions and diagnosis etc... even after discussing history etc .... wow totally blown away!...... (LeroyDyer/Mixtral_AI_Cyber_3.0) 7b -i would love to see its thinking and calculations as well as its inner monologue... (my new base model totally blew my favorite mistralorca out the window!)
great stuff bro ! ....we are in the front of the race !(i'm not making youtubers either -im out here banging out experiments too bro!)