chore: remove unused files

Browse files

Signed-off-by: jupyterjazz <saba.sturua@elastic.co>

Files changed (13) hide show

LICENSE +0 -202
adapters/classification/README.md +0 -202
adapters/clustering/README.md +0 -202
adapters/clustering/checkpoint.ckpt/checkpoint/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt +0 -3
adapters/clustering/checkpoint.ckpt/checkpoint/mp_rank_00_model_states.pt +0 -3
adapters/clustering/checkpoint.ckpt/latest +0 -1
adapters/clustering/checkpoint.ckpt/zero_to_fp32.py +0 -760
adapters/clustering/extra_layers.bin +0 -3
adapters/clustering/worker0-dataset.json +0 -1
adapters/retrieval/README.md +0 -202
adapters/retrieval/extra_layers.bin +0 -3
adapters/text-matching/README.md +0 -202
adapters/text-matching/extra_layers.bin +0 -3

LICENSE DELETED Viewed

@@ -1,202 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-   1. Definitions.
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-   END OF TERMS AND CONDITIONS
-   APPENDIX: How to apply the Apache License to your work.
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-   Copyright 2024 Alibaba Cloud
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-       http://www.apache.org/licenses/LICENSE-2.0
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

adapters/classification/README.md DELETED Viewed

@@ -1,202 +0,0 @@
----
-base_model: jinaai/jina-embeddings-v5-text-0.6B-pair-ckpt
-library_name: peft
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.15.2

adapters/clustering/README.md DELETED Viewed

@@ -1,202 +0,0 @@
----
-base_model: jinaai/jina-embeddings-v5-text-0.6B-pair-ckpt
-library_name: peft
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.15.2

adapters/clustering/checkpoint.ckpt/checkpoint/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c72f1f514e18ff8af2ba0f98f9ba28c8bd405dd10a9e642be32013063551c30a
-size 273756613

adapters/clustering/checkpoint.ckpt/checkpoint/mp_rank_00_model_states.pt DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:bc414faaaff68aed6b4caa076a95e03287fece14379d3aea88acf5f7925bbc6f
-size 1420663051

adapters/clustering/checkpoint.ckpt/latest DELETED Viewed

	@@ -1 +0,0 @@
1	- checkpoint

adapters/clustering/checkpoint.ckpt/zero_to_fp32.py DELETED Viewed

@@ -1,760 +0,0 @@
-#!/usr/bin/env python
-# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: Apache-2.0
-# DeepSpeed Team
-# This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
-# copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
-# the future. Once extracted, the weights don't require DeepSpeed and can be used in any
-# application.
-#
-# example:
-#   python zero_to_fp32.py . output_dir/
-#   or
-#   python zero_to_fp32.py . output_dir/ --safe_serialization
-import argparse
-import torch
-import glob
-import math
-import os
-import re
-import gc
-import json
-import numpy as np
-from tqdm import tqdm
-from collections import OrderedDict
-from dataclasses import dataclass
-# while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
-# DeepSpeed data structures it has to be available in the current python environment.
-from deepspeed.utils import logger
-from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
-                                            FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
-                                            FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
-@dataclass
-class zero_model_state:
-    buffers: dict()
-    param_shapes: dict()
-    shared_params: list
-    ds_version: int
-    frozen_param_shapes: dict()
-    frozen_param_fragments: dict()
-debug = 0
-# load to cpu
-device = torch.device('cpu')
-def atoi(text):
-    return int(text) if text.isdigit() else text
-def natural_keys(text):
-    '''
-    alist.sort(key=natural_keys) sorts in human order
-    http://nedbatchelder.com/blog/200712/human_sorting.html
-    (See Toothy's implementation in the comments)
-    '''
-    return [atoi(c) for c in re.split(r'(\d+)', text)]
-def get_model_state_file(checkpoint_dir, zero_stage):
-    if not os.path.isdir(checkpoint_dir):
-        raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
-    # there should be only one file
-    if zero_stage <= 2:
-        file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
-    elif zero_stage == 3:
-        file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
-    if not os.path.exists(file):
-        raise FileNotFoundError(f"can't find model states file at '{file}'")
-    return file
-def get_checkpoint_files(checkpoint_dir, glob_pattern):
-    # XXX: need to test that this simple glob rule works for multi-node setup too
-    ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
-    if len(ckpt_files) == 0:
-        raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
-    return ckpt_files
-def get_optim_files(checkpoint_dir):
-    return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
-def get_model_state_files(checkpoint_dir):
-    return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
-def parse_model_states(files):
-    zero_model_states = []
-    for file in files:
-        state_dict = torch.load(file, map_location=device, weights_only=False)
-        if BUFFER_NAMES not in state_dict:
-            raise ValueError(f"{file} is not a model state checkpoint")
-        buffer_names = state_dict[BUFFER_NAMES]
-        if debug:
-            print("Found buffers:", buffer_names)
-        # recover just the buffers while restoring them to fp32 if they were saved in fp16
-        buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
-        param_shapes = state_dict[PARAM_SHAPES]
-        # collect parameters that are included in param_shapes
-        param_names = []
-        for s in param_shapes:
-            for name in s.keys():
-                param_names.append(name)
-        # update with frozen parameters
-        frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
-        if frozen_param_shapes is not None:
-            if debug:
-                print(f"Found frozen_param_shapes: {frozen_param_shapes}")
-            param_names += list(frozen_param_shapes.keys())
-        # handle shared params
-        shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
-        ds_version = state_dict.get(DS_VERSION, None)
-        frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
-        z_model_state = zero_model_state(buffers=buffers,
-                                         param_shapes=param_shapes,
-                                         shared_params=shared_params,
-                                         ds_version=ds_version,
-                                         frozen_param_shapes=frozen_param_shapes,
-                                         frozen_param_fragments=frozen_param_fragments)
-        zero_model_states.append(z_model_state)
-    return zero_model_states
-def parse_optim_states(files, ds_checkpoint_dir):
-    total_files = len(files)
-    state_dicts = []
-    for f in tqdm(files, desc='Loading checkpoint shards'):
-        state_dict = torch.load(f, map_location=device, mmap=True, weights_only=False)
-        # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
-        # and also handle the case where it was already removed by another helper script
-        state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
-        state_dicts.append(state_dict)
-    if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
-        raise ValueError(f"{files[0]} is not a zero checkpoint")
-    zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
-    world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
-    # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
-    # parameters can be different from data parallelism for non-expert parameters. So we can just
-    # use the max of the partition_count to get the dp world_size.
-    if type(world_size) is list:
-        world_size = max(world_size)
-    if world_size != total_files:
-        raise ValueError(
-            f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
-            "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
-        )
-    # the groups are named differently in each stage
-    if zero_stage <= 2:
-        fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
-    elif zero_stage == 3:
-        fp32_groups_key = FP32_FLAT_GROUPS
-    else:
-        raise ValueError(f"unknown zero stage {zero_stage}")
-    fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
-    return zero_stage, world_size, fp32_flat_groups
-def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
-    """
-    Returns fp32 state_dict reconstructed from ds checkpoint
-    Args:
-        - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
-    """
-    print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
-    optim_files = get_optim_files(ds_checkpoint_dir)
-    zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
-    print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
-    model_files = get_model_state_files(ds_checkpoint_dir)
-    zero_model_states = parse_model_states(model_files)
-    print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
-    if zero_stage <= 2:
-        return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
-                                                          exclude_frozen_parameters)
-    elif zero_stage == 3:
-        return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
-                                                          exclude_frozen_parameters)
-def _zero2_merge_frozen_params(state_dict, zero_model_states):
-    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
-        return
-    frozen_param_shapes = zero_model_states[0].frozen_param_shapes
-    frozen_param_fragments = zero_model_states[0].frozen_param_fragments
-    if debug:
-        num_elem = sum(s.numel() for s in frozen_param_shapes.values())
-        print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
-        wanted_params = len(frozen_param_shapes)
-        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
-        avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
-        print(f'Frozen params: Have {avail_numel} numels to process.')
-        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
-    total_params = 0
-    total_numel = 0
-    for name, shape in frozen_param_shapes.items():
-        total_params += 1
-        unpartitioned_numel = shape.numel()
-        total_numel += unpartitioned_numel
-        state_dict[name] = frozen_param_fragments[name]
-        if debug:
-            print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
-    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
-def _has_callable(obj, fn):
-    attr = getattr(obj, fn, None)
-    return callable(attr)
-def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
-    param_shapes = zero_model_states[0].param_shapes
-    # Reconstruction protocol:
-    #
-    # XXX: document this
-    if debug:
-        for i in range(world_size):
-            for j in range(len(fp32_flat_groups[0])):
-                print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
-    # XXX: memory usage doubles here (zero2)
-    num_param_groups = len(fp32_flat_groups[0])
-    merged_single_partition_of_fp32_groups = []
-    for i in range(num_param_groups):
-        merged_partitions = [sd[i] for sd in fp32_flat_groups]
-        full_single_fp32_vector = torch.cat(merged_partitions, 0)
-        merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
-    avail_numel = sum(
-        [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
-    if debug:
-        wanted_params = sum([len(shapes) for shapes in param_shapes])
-        wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
-        # not asserting if there is a mismatch due to possible padding
-        print(f"Have {avail_numel} numels to process.")
-        print(f"Need {wanted_numel} numels in {wanted_params} params.")
-    # params
-    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
-    # out-of-core computing solution
-    total_numel = 0
-    total_params = 0
-    for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
-        offset = 0
-        avail_numel = full_single_fp32_vector.numel()
-        for name, shape in shapes.items():
-            unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
-            total_numel += unpartitioned_numel
-            total_params += 1
-            if debug:
-                print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
-            state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
-            offset += unpartitioned_numel
-        # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
-        # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
-        # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
-        # live optimizer object, so we are checking that the numbers are within the right range
-        align_to = 2 * world_size
-        def zero2_align(x):
-            return align_to * math.ceil(x / align_to)
-        if debug:
-            print(f"original offset={offset}, avail_numel={avail_numel}")
-        offset = zero2_align(offset)
-        avail_numel = zero2_align(avail_numel)
-        if debug:
-            print(f"aligned  offset={offset}, avail_numel={avail_numel}")
-        # Sanity check
-        if offset != avail_numel:
-            raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
-    print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
-def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
-                                               exclude_frozen_parameters):
-    state_dict = OrderedDict()
-    # buffers
-    buffers = zero_model_states[0].buffers
-    state_dict.update(buffers)
-    if debug:
-        print(f"added {len(buffers)} buffers")
-    if not exclude_frozen_parameters:
-        _zero2_merge_frozen_params(state_dict, zero_model_states)
-    _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
-    # recover shared parameters
-    for pair in zero_model_states[0].shared_params:
-        if pair[1] in state_dict:
-            state_dict[pair[0]] = state_dict[pair[1]]
-    return state_dict
-def zero3_partitioned_param_info(unpartitioned_numel, world_size):
-    remainder = unpartitioned_numel % world_size
-    padding_numel = (world_size - remainder) if remainder else 0
-    partitioned_numel = math.ceil(unpartitioned_numel / world_size)
-    return partitioned_numel, padding_numel
-def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
-    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
-        return
-    if debug:
-        for i in range(world_size):
-            num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
-            print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
-        frozen_param_shapes = zero_model_states[0].frozen_param_shapes
-        wanted_params = len(frozen_param_shapes)
-        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
-        avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
-        print(f'Frozen params: Have {avail_numel} numels to process.')
-        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
-    total_params = 0
-    total_numel = 0
-    for name, shape in zero_model_states[0].frozen_param_shapes.items():
-        total_params += 1
-        unpartitioned_numel = shape.numel()
-        total_numel += unpartitioned_numel
-        param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
-        state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
-        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
-        if debug:
-            print(
-                f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
-            )
-    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
-class GatheredTensor:
-    """
-    A pseudo tensor that collects partitioned weights.
-    It is more memory efficient when there are multiple groups.
-    """
-    def __init__(self, flat_groups, flat_groups_offset, offset, partitioned_numel, shape):
-        self.flat_groups = flat_groups
-        self.flat_groups_offset = flat_groups_offset
-        self.offset = offset
-        self.partitioned_numel = partitioned_numel
-        self.shape = shape
-        self.dtype = self.flat_groups[0][0].dtype
-    def contiguous(self):
-        """
-        Merge partitioned weights from flat_groups into a single tensor.
-        """
-        end_idx = self.offset + self.partitioned_numel
-        world_size = len(self.flat_groups)
-        pad_flat_param_chunks = []
-        for rank_i in range(world_size):
-            # for each rank, we need to collect weights from related group/groups
-            flat_groups_at_rank_i = self.flat_groups[rank_i]
-            start_group_id = None
-            end_group_id = None
-            for group_id in range(len(self.flat_groups_offset)):
-                if self.flat_groups_offset[group_id] <= self.offset < self.flat_groups_offset[group_id + 1]:
-                    start_group_id = group_id
-                if self.flat_groups_offset[group_id] < end_idx <= self.flat_groups_offset[group_id + 1]:
-                    end_group_id = group_id
-                    break
-            # collect weights from related group/groups
-            for group_id in range(start_group_id, end_group_id + 1):
-                flat_tensor = flat_groups_at_rank_i[group_id]
-                start_offset = self.offset - self.flat_groups_offset[group_id]
-                end_offset = min(end_idx, self.flat_groups_offset[group_id + 1]) - self.flat_groups_offset[group_id]
-                pad_flat_param_chunks.append(flat_tensor[start_offset:end_offset])
-        # collect weights from all ranks
-        pad_flat_param = torch.cat(pad_flat_param_chunks, dim=0)
-        param = pad_flat_param[:self.shape.numel()].view(self.shape).contiguous()
-        return param
-def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
-    param_shapes = zero_model_states[0].param_shapes
-    avail_numel = sum([flat_group.numel() for flat_group in fp32_flat_groups[0]]) * world_size
-    # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
-    # param, re-consolidating each param, while dealing with padding if any
-    # merge list of dicts, preserving order
-    param_shapes = {k: v for d in param_shapes for k, v in d.items()}
-    if debug:
-        for i in range(world_size):
-            print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
-        wanted_params = len(param_shapes)
-        wanted_numel = sum(shape.numel() for shape in param_shapes.values())
-        # not asserting if there is a mismatch due to possible padding
-        avail_numel = fp32_flat_groups[0].numel() * world_size
-        print(f"Trainable params: Have {avail_numel} numels to process.")
-        print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
-    # params
-    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
-    # out-of-core computing solution
-    offset = 0
-    total_numel = 0
-    total_params = 0
-    flat_groups_offset = [0] + list(np.cumsum([flat_tensor.numel() for flat_tensor in fp32_flat_groups[0]]))
-    for name, shape in tqdm(param_shapes.items(), desc='Gathering sharded weights'):
-        unpartitioned_numel = shape.numel()
-        total_numel += unpartitioned_numel
-        total_params += 1
-        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
-        if debug:
-            print(
-                f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
-            )
-        # memory efficient tensor
-        tensor = GatheredTensor(fp32_flat_groups, flat_groups_offset, offset, partitioned_numel, shape)
-        state_dict[name] = tensor
-        offset += partitioned_numel
-    offset *= world_size
-    # Sanity check
-    if offset != avail_numel:
-        raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
-    print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
-def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
-                                               exclude_frozen_parameters):
-    state_dict = OrderedDict()
-    # buffers
-    buffers = zero_model_states[0].buffers
-    state_dict.update(buffers)
-    if debug:
-        print(f"added {len(buffers)} buffers")
-    if not exclude_frozen_parameters:
-        _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
-    _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
-    # recover shared parameters
-    for pair in zero_model_states[0].shared_params:
-        if pair[1] in state_dict:
-            state_dict[pair[0]] = state_dict[pair[1]]
-    return state_dict
-def to_torch_tensor(state_dict, return_empty_tensor=False):
-    """
-    Convert state_dict of GatheredTensor to torch tensor
-    """
-    torch_state_dict = {}
-    converted_tensors = {}
-    for name, tensor in state_dict.items():
-        tensor_id = id(tensor)
-        if tensor_id in converted_tensors:  # shared tensors
-            shared_tensor = torch_state_dict[converted_tensors[tensor_id]]
-            torch_state_dict[name] = shared_tensor
-        else:
-            converted_tensors[tensor_id] = name
-            if return_empty_tensor:
-                torch_state_dict[name] = torch.empty(tensor.shape, dtype=tensor.dtype)
-            else:
-                torch_state_dict[name] = tensor.contiguous()
-    return torch_state_dict
-def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
-                                             tag=None,
-                                             exclude_frozen_parameters=False,
-                                             lazy_mode=False):
-    """
-    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
-    ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
-    via a model hub.
-    Args:
-        - ``checkpoint_dir``: path to the desired checkpoint folder
-        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
-        - ``exclude_frozen_parameters``: exclude frozen parameters
-        - ``lazy_mode``: get state_dict in lazy mode. It returns a dict of pesduo tensor instead of torch tensor, which is more memory efficient.
-          Convert the pesduo tensor to torch tensor by ``.contiguous()``
-    Returns:
-        - pytorch ``state_dict``
-    A typical usage might be ::
-        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
-        # do the training and checkpoint saving
-        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
-        model = model.cpu() # move to cpu
-        model.load_state_dict(state_dict)
-        # submit to model hub or save the model to share with others
-    In this example the ``model`` will no longer be usable in the deepspeed context of the same
-    application. i.e. you will need to re-initialize the deepspeed engine, since
-    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
-    If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
-    Note: the above usage may not work if your application doesn't have sufficient free CPU memory.
-    You may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
-    the checkpoint. Or you can load state_dict in lazy mode ::
-        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
-        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, lazy_mode=True) # not on cpu
-        for name, lazy_tensor in state_dict.item():
-            tensor = lazy_tensor.contiguous()  # to cpu
-            print(name, tensor)
-            # del tensor to release memory if it no longer in use
-    """
-    if tag is None:
-        latest_path = os.path.join(checkpoint_dir, 'latest')
-        if os.path.isfile(latest_path):
-            with open(latest_path, 'r') as fd:
-                tag = fd.read().strip()
-        else:
-            raise ValueError(f"Unable to find 'latest' file at {latest_path}")
-    ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
-    if not os.path.isdir(ds_checkpoint_dir):
-        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
-    state_dict = _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
-    if lazy_mode:
-        return state_dict
-    else:
-        return to_torch_tensor(state_dict)
-def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
-                                               output_dir,
-                                               max_shard_size="5GB",
-                                               safe_serialization=False,
-                                               tag=None,
-                                               exclude_frozen_parameters=False):
-    """
-    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
-    loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
-    Args:
-        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
-        - ``output_dir``: directory to the pytorch fp32 state_dict output files
-        - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
-        - ``safe_serialization``:  whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
-        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
-        - ``exclude_frozen_parameters``: exclude frozen parameters
-    """
-    # Dependency pre-check
-    if safe_serialization:
-        try:
-            from safetensors.torch import save_file
-        except ImportError:
-            print('If you want to use `safe_serialization`, please `pip install safetensors`')
-            raise
-    if max_shard_size is not None:
-        try:
-            from huggingface_hub import split_torch_state_dict_into_shards
-        except ImportError:
-            print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
-            raise
-    # Convert zero checkpoint to state_dict
-    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
-                                                          tag,
-                                                          exclude_frozen_parameters,
-                                                          lazy_mode=True)
-    # Shard the model if it is too big.
-    weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
-    if max_shard_size is not None:
-        filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
-        # an memory-efficient approach for sharding
-        empty_state_dict = to_torch_tensor(state_dict, return_empty_tensor=True)
-        state_dict_split = split_torch_state_dict_into_shards(empty_state_dict,
-                                                              filename_pattern=filename_pattern,
-                                                              max_shard_size=max_shard_size)
-    else:
-        from collections import namedtuple
-        StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
-        state_dict_split = StateDictSplit(is_sharded=False,
-                                          filename_to_tensors={weights_name: list(state_dict.keys())})
-    # Save the model by shard
-    os.makedirs(output_dir, exist_ok=True)
-    filename_to_tensors = state_dict_split.filename_to_tensors.items()
-    for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
-        shard_state_dict = {tensor_name: state_dict[tensor_name] for tensor_name in tensors}
-        shard_state_dict = to_torch_tensor(shard_state_dict)
-        output_path = os.path.join(output_dir, shard_file)
-        if safe_serialization:
-            save_file(shard_state_dict, output_path, metadata={"format": "pt"})
-        else:
-            torch.save(shard_state_dict, output_path)
-        # release the memory of current shard
-        for tensor_name in list(shard_state_dict.keys()):
-            del state_dict[tensor_name]
-            del shard_state_dict[tensor_name]
-        del shard_state_dict
-        gc.collect()
-    # Save index if sharded
-    if state_dict_split.is_sharded:
-        index = {
-            "metadata": state_dict_split.metadata,
-            "weight_map": state_dict_split.tensor_to_filename,
-        }
-        save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
-        save_index_file = os.path.join(output_dir, save_index_file)
-        with open(save_index_file, "w", encoding="utf-8") as f:
-            content = json.dumps(index, indent=2, sort_keys=True) + "\n"
-            f.write(content)
-def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
-    """
-    1. Put the provided model to cpu
-    2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
-    3. Load it into the provided model
-    Args:
-        - ``model``: the model object to update
-        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
-        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
-    Returns:
-        - ``model`: modified model
-    Make sure you have plenty of CPU memory available before you call this function. If you don't
-    have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
-    conveniently placed for you in the checkpoint folder.
-    A typical usage might be ::
-        from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
-        model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
-        # submit to model hub or save the model to share with others
-    Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
-    of the same application. i.e. you will need to re-initialize the deepspeed engine, since
-    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
-    """
-    logger.info(f"Extracting fp32 weights")
-    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
-    logger.info(f"Overwriting model with fp32 weights")
-    model = model.cpu()
-    model.load_state_dict(state_dict, strict=False)
-    return model
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser.add_argument("checkpoint_dir",
-                        type=str,
-                        help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
-    parser.add_argument("output_dir",
-                        type=str,
-                        help="directory to the pytorch fp32 state_dict output files"
-                        "(e.g. path/checkpoint-12-output/)")
-    parser.add_argument(
-        "--max_shard_size",
-        type=str,
-        default="5GB",
-        help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
-        "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
-        "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
-        "without CPU OOM issues.")
-    parser.add_argument(
-        "--safe_serialization",
-        default=False,
-        action='store_true',
-        help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
-    parser.add_argument("-t",
-                        "--tag",
-                        type=str,
-                        default=None,
-                        help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
-    parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
-    parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
-    args = parser.parse_args()
-    debug = args.debug
-    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
-                                               args.output_dir,
-                                               max_shard_size=args.max_shard_size,
-                                               safe_serialization=args.safe_serialization,
-                                               tag=args.tag,
-                                               exclude_frozen_parameters=args.exclude_frozen_parameters)

adapters/clustering/extra_layers.bin DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:fe90731a3f8de4abbebc11db58a60d6ef55194160f238d44663d77b6335b8ab3
-size 45619980

adapters/clustering/worker0-dataset.json DELETED Viewed

@@ -1 +0,0 @@

- {"text_batch_size": 512, "image_batch_size": null, "text_only_datasets": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "num_hard_negatives": 0, "datasets": [{"name": "fr/pairs/alloprof", "weight": 200.0, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["fr/pairs/alloprof"], "current_index": 159, "instruction_config": {}, "global_seed": 0, "local_seed": 18, "num_hard_negatives": 0, "shuffle": true}}, {"name": "en/pairs-dedup/stack_exchange_title_body", "weight": 1.0, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["en/pairs-dedup/stack_exchange_title_body"], "current_index": 881663, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "de/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["de/pairs/ccnews_title_desc"], "current_index": 852991, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "en/pairs/biorxiv", "weight": 10, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["en/pairs/biorxiv"], "current_index": 62463, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "en/pairs/medrxiv", "weight": 10, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["en/pairs/medrxiv"], "current_index": 14335, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "en/pairs-dedup/cc_news_title_description", "weight": 2, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["en/pairs-dedup/cc_news_title_description"], "current_index": 54783, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "en/pairs-dedup/s2orc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["en/pairs-dedup/s2orc"], "current_index": 875007, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "es/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["es/pairs/ccnews_title_desc"], "current_index": 873471, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "fi/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["fi/pairs/ccnews_title_desc"], "current_index": 67071, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "fr/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["fr/pairs/ccnews_title_desc"], "current_index": 865488, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "hi/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["hi/pairs/ccnews_title_desc"], "current_index": 596884, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "id/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["id/pairs/ccnews_title_desc"], "current_index": 238079, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "it/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["it/pairs/ccnews_title_desc"], "current_index": 915967, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "ja/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["ja/pairs/ccnews_title_desc"], "current_index": 284213, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "ka/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["ka/pairs/ccnews_title_desc"], "current_index": 5631, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "ko/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["ko/pairs/ccnews_title_desc"], "current_index": 296959, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "lv/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["lv/pairs/ccnews_title_desc"], "current_index": 32255, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "nl/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["nl/pairs/ccnews_title_desc"], "current_index": 258047, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "no/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["no/pairs/ccnews_title_desc"], "current_index": 49663, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "pl/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["pl/pairs/ccnews_title_desc"], "current_index": 143871, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "pt/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["pt/pairs/ccnews_title_desc"], "current_index": 246271, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "ro/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["ro/pairs/ccnews_title_desc"], "current_index": 335359, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "ru/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["ru/pairs/ccnews_title_desc"], "current_index": 875007, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "sk/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["sk/pairs/ccnews_title_desc"], "current_index": 38399, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "sv/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["sv/pairs/ccnews_title_desc"], "current_index": 120831, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "th/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["th/pairs/ccnews_title_desc"], "current_index": 5631, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "tr/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["tr/pairs/ccnews_title_desc"], "current_index": 532479, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "uk/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["uk/pairs/ccnews_title_desc"], "current_index": 144383, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "ur/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["ur/pairs/ccnews_title_desc"], "current_index": 84991, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "vi/pairs/ccnews_title_desc", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["vi/pairs/ccnews_title_desc"], "current_index": 229375, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}, {"name": "zh/pairs/ccnews_desc_maintext", "weight": 1, "state_dict": {"data_dir": "/media/gcp-ml-jina-embeddings/text-embedding-training", "dataset_names": ["zh/pairs/ccnews_desc_maintext"], "current_index": 93695, "instruction_config": {}, "global_seed": 0, "local_seed": null, "num_hard_negatives": 0, "shuffle": true}}]}, "text_image_datasets": null, "text_to_image_ratio": null, "model_instructions": {"document": "Document: ", "query": "Document: "}, "max_batches": 20001, "num_batches": 0, "seed": 0, "rng": [3, [4060041028, 2525655853, 22022613, 2432513887, 3183929879, 293358025, 3981749443, 1368298699, 854460374, 3611998497, 1054462155, 523497980, 1319500776, 902821040, 1185172547, 4100890727, 2967129712, 62566099, 3408625012, 2954898987, 3926772196, 825435138, 3736024504, 1435805512, 2826856930, 4179240298, 3318609768, 1705764287, 1961831560, 245919082, 307449648, 1944657026, 3665107540, 2306139750, 1729049830, 2384014381, 3342532918, 3788329751, 1817985790, 2026073664, 3150637164, 2702585843, 3730032421, 1657073812, 2760182770, 1677384445, 1181776129, 2883253230, 3994913799, 106295136, 2809757337, 3289150025, 1956693718, 3659681850, 1952203718, 2136928272, 3108466169, 2202939327, 719175203, 2547676577, 2692979646, 2197926762, 383988400, 4080050610, 4219011957, 2031788274, 3221308998, 773862647, 1663226155, 1440951606, 2394705698, 2558248306, 3747398659, 998016611, 4158542002, 1459349939, 504814102, 4252102367, 3847655795, 3030841223, 1470830944, 3765838791, 802201688, 454548171, 3731669268, 2828855261, 2224282747, 781206840, 131881556, 651581975, 920722140, 886336913, 1299649949, 3269365391, 979472372, 730194031, 4062221579, 2284774540, 771513254, 4274746650, 2206691719, 4204689907, 2547511977, 1173155246, 3455634572, 3491578956, 1467963486, 2451407908, 1610196051, 2278594889, 1504738410, 2222441002, 531957205, 2556040654, 3310258682, 4033328385, 698046092, 1565070384, 2336961122, 2372555765, 3613770397, 2129263814, 2582339858, 2621005712, 3661242897, 1586552847, 3828836390, 3246338552, 462276634, 705459359, 3293184017, 399670763, 1679586659, 145370467, 3097687543, 2151385444, 2264265099, 495770290, 4276417375, 631905911, 3049582588, 2508696452, 2336292094, 763896259, 2859185187, 1178047807, 4143592971, 474200751, 2479294970, 3458189926, 2958515542, 3390157570, 655621366, 3452127468, 731936127, 965549820, 696308956, 3618942324, 3060592103, 3208226987, 1458369932, 450263943, 3210981296, 429952163, 633404900, 3431152044, 3312758705, 4120122317, 3243748643, 2758870998, 909971369, 4222827718, 2945391278, 896844990, 1461037402, 845231038, 924846397, 2703022441, 3304527948, 1711591385, 32468466, 374194316, 1795202259, 3046401902, 4127758961, 3376874185, 25572811, 1237814209, 3148793953, 3557375974, 2885214338, 2797300164, 633062618, 3550817421, 992379327, 3489770668, 289899003, 2979570990, 1256783106, 2088820639, 1347945594, 2576240922, 1456471422, 1836002614, 3077459883, 2794986751, 3747823600, 4265524694, 4200146616, 4206691453, 52951948, 1397176203, 1602171089, 2169454320, 1861426731, 1075080128, 1026567356, 773251132, 2480838650, 2205375290, 2388484024, 2864100746, 1343368018, 1383794899, 4287158246, 1121881505, 2365856027, 1998231886, 4085600818, 1213490535, 2356518834, 4133093993, 2564509824, 2508722593, 3136484805, 661060110, 2804464765, 3292308236, 1653123702, 246692618, 3317091362, 1358439685, 3803480875, 1562820177, 4271966746, 856494699, 3958340569, 3450562660, 960356947, 4254428737, 2087029639, 980682775, 3940286194, 4021745880, 34657352, 53733553, 3800517375, 3777717597, 1147824925, 2231463737, 3508517190, 3564009078, 1012455921, 2750709375, 3968771982, 683626821, 1407403347, 2096571978, 3242206788, 584353310, 272487860, 1872139367, 557424844, 2030353707, 1572677067, 631051976, 1829727158, 4112313817, 738637328, 3348493153, 2022794774, 3617160483, 2272422790, 1872430247, 3616996565, 2102045038, 1152800742, 2261770569, 2698304663, 3724619400, 2199269943, 2790044320, 3741041946, 1922974969, 983812649, 3491848858, 3552455965, 1659542537, 2738756745, 2195805908, 33218927, 2031873379, 1217339883, 2657623222, 4048962227, 563228394, 379757977, 331906297, 2409893002, 185031443, 2202318688, 2163517554, 2095121589, 2698941754, 3521959518, 2379529975, 213600492, 629413413, 3638585980, 3320340804, 2867785893, 3118929624, 4088450821, 1561631959, 389596354, 402556183, 561076286, 736862458, 3867783469, 791207400, 4041716154, 3281350316, 1361011919, 563029504, 4129809360, 4215044840, 11190910, 2736180646, 3670038577, 1017362643, 2039994678, 3080017806, 1900635992, 288672837, 3130803305, 917601274, 3397663956, 3561191778, 1756006020, 3084850281, 1083596555, 4201781260, 3034540053, 267943055, 2179428087, 2736008359, 216234279, 2100446484, 249502540, 2250571704, 2895152479, 932884153, 2064275997, 4066577459, 1152853601, 1096917726, 4243098578, 1709128183, 2624764018, 3309980151, 288724747, 2694438804, 4004373874, 460086837, 1820884696, 3083107825, 286398908, 3307176203, 1834812185, 2392634101, 2104778700, 2188255916, 1106429925, 2169723710, 2454834057, 2770570967, 3944596763, 3950869818, 891468839, 2526920851, 967384663, 1967898245, 1112772340, 2401471792, 2574854711, 3672103573, 1758098491, 3810481984, 3333163245, 3911248738, 1698145122, 855019511, 1678211329, 1355991168, 1140204435, 2534230158, 3626060745, 3384961022, 3843763936, 2626749873, 1976910934, 423851305, 1827987034, 4110942803, 710842907, 1782381432, 647309050, 3639932770, 1054619954, 1747128277, 1534128216, 4151504117, 3288003342, 2739703723, 2614347892, 255160547, 2659756785, 1782481896, 4172601753, 4140579766, 1837074982, 361648134, 1428088168, 2077013968, 363427080, 1947134723, 1250623276, 3874838508, 3805003464, 958040680, 3027261913, 2226287708, 399479526, 896470797, 2597010758, 2007391494, 2893731291, 1972825579, 1187552609, 395081443, 2700444295, 1165547192, 3710931686, 666424663, 1167514918, 2850829322, 4077849210, 2038784772, 2157927914, 1781494991, 2038168977, 564044933, 2482583123, 521184425, 79668476, 350761927, 3406003085, 3559547578, 4279268530, 980836884, 908881301, 1702198746, 1622865633, 1820445998, 1465843819, 211505401, 3636251865, 521253443, 4110843191, 506258535, 1761089294, 1351780322, 1953941943, 883624533, 421829877, 4184009557, 3712774150, 2650261922, 2237654051, 2547910603, 1188768161, 3081512244, 1277836915, 37869804, 4054094970, 2006474175, 3655313485, 3760808669, 2232509031, 2119716165, 3434489404, 1183808328, 812879878, 2147570209, 1108578909, 3608416766, 587190541, 1585433794, 2436814801, 3266915151, 2287907524, 2185228891, 1096781945, 3843215951, 3714325583, 902846348, 2584696460, 457411796, 708237965, 4089321295, 2631773973, 2074423482, 1512294518, 4044358030, 2867545589, 4089784532, 2312932160, 4050165596, 3318239864, 3072116252, 1008493200, 2543858978, 1834740527, 626235105, 3802760473, 1069635571, 678215146, 3413070070, 966079344, 4205469531, 3628862212, 1403637473, 3633952531, 3958081244, 1217507732, 3307524795, 2131313758, 2281525225, 3064080344, 92644040, 3066098544, 3925441409, 3689893789, 4230968917, 2294655088, 4180809906, 19603598, 673564351, 2441220950, 3554633439, 882887218, 206275593, 2877322366, 951261864, 488570614, 2442664629, 3428158149, 2900818070, 3804266884, 3407080487, 112269588, 2115659766, 3825516493, 1075503584, 322599557, 217467693, 45228623, 2148762225, 3196153459, 2182956719, 34635993, 3829234365, 1180099414, 1773237760, 2333672262, 1299268138, 358233999, 3123401160, 141526855, 3157823266, 3986696440, 3549429033, 1970183592, 2871289733, 3214774875, 846768151, 831022562, 2140465215, 3256722839, 176513431, 43926812, 3511108814, 4059377654, 3346851466, 2752731016, 3060850345, 1152814488, 3050284442, 1598816741, 4011257468, 2989433410, 1777693317, 1362151586, 3116294046, 4208051422, 1154377432, 3601807373, 2192868428, 2985753950, 4122130237, 2217597166, 1897915667, 297736574, 1345704857, 66], null]}

adapters/retrieval/README.md DELETED Viewed

@@ -1,202 +0,0 @@
----
-base_model: jinaai/jina-embeddings-v5-text-0.6B-pair-ckpt
-library_name: peft
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.15.2

adapters/retrieval/extra_layers.bin DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5dd1b34a3178bb3efbc28d4afbe28196a71948d2526dc102a7242a7407c7ffd4
-size 45619980

adapters/text-matching/README.md DELETED Viewed

@@ -1,202 +0,0 @@
----
-base_model: jinaai/jina-embeddings-v5-text-0.6B-pair-ckpt
-library_name: peft
----
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
-### Framework versions
-- PEFT 0.15.2

adapters/text-matching/extra_layers.bin DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:357036645e2b7a4a7edbd827dc5b276b258055f2f9e70ec8b160dfed23dd8f80
-size 45619980