Super-squash branch 'main' using huggingface_hub

Co-authored-by: autoevaluator <autoevaluator@users.noreply.huggingface.co>
Co-authored-by: SFconvertbot <SFconvertbot@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,28 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1 @@
+checkpoint-*/

README.md

@@ -0,0 +1,568 @@
+---
+license:
+- apache-2.0
+- bsd-3-clause
+tags:
+- summarization
+- led
+- summary
+- longformer
+- booksum
+- long-document
+- long-form
+datasets:
+- kmfoda/booksum
+metrics:
+- rouge
+widget:
+- text: large earthquakes along a given fault segment do not occur at random intervals
+    because it takes time to accumulate the strain energy for the rupture. The rates
+    at which tectonic plates move and accumulate strain at their boundaries are approximately
+    uniform. Therefore, in first approximation, one may expect that large ruptures
+    of the same fault segment will occur at approximately constant time intervals.
+    If subsequent main shocks have different amounts of slip across the fault, then
+    the recurrence time may vary, and the basic idea of periodic mainshocks must be
+    modified. For great plate boundary ruptures the length and slip often vary by
+    a factor of 2. Along the southern segment of the San Andreas fault the recurrence
+    interval is 145 years with variations of several decades. The smaller the standard
+    deviation of the average recurrence interval, the more specific could be the long
+    term prediction of a future mainshock.
+  example_title: earthquakes
+- text: ' A typical feed-forward neural field algorithm. Spatiotemporal coordinates
+    are fed into a neural network that predicts values in the reconstructed domain.
+    Then, this domain is mapped to the sensor domain where sensor measurements are
+    available as supervision. Class and Section Problems Addressed Generalization
+    (Section 2) Inverse problems, ill-posed problems, editability; symmetries. Hybrid
+    Representations (Section 3) Computation & memory efficiency, representation capacity,
+    editability: Forward Maps (Section 4) Inverse problems Network Architecture (Section
+    5) Spectral bias, integration & derivatives. Manipulating Neural Fields (Section
+    6) Edit ability, constraints, regularization. Table 2: The five classes of techniques
+    in the neural field toolbox each addresses problems that arise in learning, inference,
+    and control. (Section 3). We can supervise reconstruction via differentiable forward
+    maps that transform Or project our domain (e.g, 3D reconstruction via 2D images;
+    Section 4) With appropriate network architecture choices, we can overcome neural
+    network spectral biases (blurriness) and efficiently compute derivatives and integrals
+    (Section 5). Finally, we can manipulate neural fields to add constraints and regularizations,
+    and to achieve editable representations (Section 6). Collectively, these classes
+    constitute a ''toolbox'' of techniques to help solve problems with neural fields
+    There are three components in a conditional neural field: (1) An encoder or inference
+    function € that outputs the conditioning latent variable 2 given an observation
+    0 E(0) =2. 2 is typically a low-dimensional vector, and is often referred to aS
+    a latent code Or feature code_ (2) A mapping function 4 between Z and neural field
+    parameters O: Y(z) = O; (3) The neural field itself $. The encoder € finds the
+    most probable z given the observations O: argmaxz P(2/0). The decoder maximizes
+    the inverse conditional probability to find the most probable 0 given Z: arg-
+    max P(Olz). We discuss different encoding schemes with different optimality guarantees
+    (Section 2.1.1), both global and local conditioning (Section 2.1.2), and different
+    mapping functions Y (Section 2.1.3) 2. Generalization Suppose we wish to estimate
+    a plausible 3D surface shape given a partial or noisy point cloud. We need a suitable
+    prior over the sur- face in its reconstruction domain to generalize to the partial
+    observations. A neural network expresses a prior via the function space of its
+    architecture and parameters 0, and generalization is influenced by the inductive
+    bias of this function space (Section 5).'
+  example_title: scientific paper
+- text: ' the big variety of data coming from diverse sources is one of the key properties
+    of the big data phenomenon. It is, therefore, beneficial to understand how data
+    is generated in various environments and scenarios, before looking at what should
+    be done with this data and how to design the best possible architecture to accomplish
+    this The evolution of IT architectures, described in Chapter 2, means that the
+    data is no longer processed by a few big monolith systems, but rather by a group
+    of services In parallel to the processing layer, the underlying data storage has
+    also changed and became more distributed This, in turn, required a significant
+    paradigm shift as the traditional approach to transactions (ACID) could no longer
+    be supported. On top of this, cloud computing is becoming a major approach with
+    the benefits of reducing costs and providing on-demand scalability but at the
+    same time introducing concerns about privacy, data ownership, etc In the meantime
+    the Internet continues its exponential growth: Every day both structured and unstructured
+    data is published and available for processing: To achieve competitive advantage
+    companies have to relate their corporate resources to external services, e.g.
+    financial markets, weather forecasts, social media, etc While several of the sites
+    provide some sort of API to access the data in a more orderly fashion; countless
+    sources require advanced web mining and Natural Language Processing (NLP) processing
+    techniques: Advances in science push researchers to construct new instruments
+    for observing the universe O conducting experiments to understand even better
+    the laws of physics and other domains. Every year humans have at their disposal
+    new telescopes, space probes, particle accelerators, etc These instruments generate
+    huge streams of data, which need to be stored and analyzed. The constant drive
+    for efficiency in the industry motivates the introduction of new automation techniques
+    and process optimization: This could not be done without analyzing the precise
+    data that describe these processes. As more and more human tasks are automated,
+    machines provide rich data sets, which can be analyzed in real-time to drive efficiency
+    to new levels. Finally, it is now evident that the growth of the Internet of Things
+    is becoming a major source of data. More and more of the devices are equipped
+    with significant computational power and can generate a continuous data stream
+    from their sensors. In the subsequent sections of this chapter, we will look at
+    the domains described above to see what they generate in terms of data sets. We
+    will compare the volumes but will also look at what is characteristic and important
+    from their respective points of view. 3.1 The Internet is undoubtedly the largest
+    database ever created by humans. While several well described; cleaned, and structured
+    data sets have been made available through this medium, most of the resources
+    are of an ambiguous, unstructured, incomplete or even erroneous nature. Still,
+    several examples in the areas such as opinion mining, social media analysis, e-governance,
+    etc, clearly show the potential lying in these resources. Those who can successfully
+    mine and interpret the Internet data can gain unique insight and competitive advantage
+    in their business An important area of data analytics on the edge of corporate
+    IT and the Internet is Web Analytics.'
+  example_title: data science textbook
+- text: 'Transformer-based models have shown to be very useful for many NLP tasks.
+    However, a major limitation of transformers-based models is its O(n^2)O(n 2) time
+    & memory complexity (where nn is sequence length). Hence, it''s computationally
+    very expensive to apply transformer-based models on long sequences n > 512n>512.
+    Several recent papers, e.g. Longformer, Performer, Reformer, Clustered attention
+    try to remedy this problem by approximating the full attention matrix. You can
+    checkout 🤗''s recent blog post in case you are unfamiliar with these models.
+
+    BigBird (introduced in paper) is one of such recent models to address this issue.
+    BigBird relies on block sparse attention instead of normal attention (i.e. BERT''s
+    attention) and can handle sequences up to a length of 4096 at a much lower computational
+    cost compared to BERT. It has achieved SOTA on various tasks involving very long
+    sequences such as long documents summarization, question-answering with long contexts.
+
+    BigBird RoBERTa-like model is now available in 🤗Transformers. The goal of this
+    post is to give the reader an in-depth understanding of big bird implementation
+    & ease one''s life in using BigBird with 🤗Transformers. But, before going into
+    more depth, it is important to remember that the BigBird''s attention is an approximation
+    of BERT''s full attention and therefore does not strive to be better than BERT''s
+    full attention, but rather to be more efficient. It simply allows to apply transformer-based
+    models to much longer sequences since BERT''s quadratic memory requirement quickly
+    becomes unbearable. Simply put, if we would have ∞ compute & ∞ time, BERT''s attention
+    would be preferred over block sparse attention (which we are going to discuss
+    in this post).
+
+    If you wonder why we need more compute when working with longer sequences, this
+    blog post is just right for you!
+
+    Some of the main questions one might have when working with standard BERT-like
+    attention include:
+
+    Do all tokens really have to attend to all other tokens? Why not compute attention
+    only over important tokens? How to decide what tokens are important? How to attend
+    to just a few tokens in a very efficient way? In this blog post, we will try to
+    answer those questions.
+
+    What tokens should be attended to? We will give a practical example of how attention
+    works by considering the sentence ''BigBird is now available in HuggingFace for
+    extractive question answering''. In BERT-like attention, every word would simply
+    attend to all other tokens.
+
+    Let''s think about a sensible choice of key tokens that a queried token actually
+    only should attend to by writing some pseudo-code. Will will assume that the token
+    available is queried and build a sensible list of key tokens to attend to.
+
+    >>> # let''s consider following sentence as an example >>> example = [''BigBird'',
+    ''is'', ''now'', ''available'', ''in'', ''HuggingFace'', ''for'', ''extractive'',
+    ''question'', ''answering'']
+
+    >>> # further let''s assume, we''re trying to understand the representation of
+    ''available'' i.e. >>> query_token = ''available'' >>> # We will initialize an
+    empty `set` and fill up the tokens of our interest as we proceed in this section.
+    >>> key_tokens = [] # => currently ''available'' token doesn''t have anything
+    to attend Nearby tokens should be important because, in a sentence (sequence of
+    words), the current word is highly dependent on neighboring past & future tokens.
+    This intuition is the idea behind the concept of sliding attention.'
+  example_title: bigbird blog intro
+- text: 'The majority of available text summarization datasets include short-form
+    source documents that lack long-range causal and temporal dependencies, and often
+    contain strong layout and stylistic biases. While relevant, such datasets will
+    offer limited challenges for future generations of text summarization systems.
+    We address these issues by introducing BookSum, a collection of datasets for long-form
+    narrative summarization. Our dataset covers source documents from the literature
+    domain, such as novels, plays and stories, and includes highly abstractive, human
+    written summaries on three levels of granularity of increasing difficulty: paragraph-,
+    chapter-, and book-level. The domain and structure of our dataset poses a unique
+    set of challenges for summarization systems, which include: processing very long
+    documents, non-trivial causal and temporal dependencies, and rich discourse structures.
+    To facilitate future work, we trained and evaluated multiple extractive and abstractive
+    summarization models as baselines for our dataset.'
+  example_title: BookSum Abstract
+inference:
+  parameters:
+    max_length: 96
+    min_length: 8
+    no_repeat_ngram_size: 3
+    early_stopping: true
+    repetition_penalty: 3.5
+    length_penalty: 0.3
+    encoder_no_repeat_ngram_size: 3
+    num_beams: 4
+model-index:
+- name: pszemraj/led-base-book-summary
+  results:
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: kmfoda/booksum
+      type: kmfoda/booksum
+      config: kmfoda--booksum
+      split: test
+    metrics:
+    - type: rouge
+      value: 33.4536
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmEzYjNkZTUxZjA0YTdmNTJkMjVkMTg2NDRjNTkzN2ZlNDlhNTBhMWQ5MTNiYWE4Mzg5YTMyMTM5YmZjNDI3OSIsInZlcnNpb24iOjF9.OWjM_HCQLQHK4AV4em70QGT3lrVk25WyZdcXA8ywest_XSx9KehJbsIMDKtXxOOMwxvkogKnScy4tbskYMQqDg
+    - type: rouge
+      value: 5.2232
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTVhOTdjZjc5YTdhMmVjZGE1NTA5MmJkYmM3Y2U3OGVlMjZmOGVlMTUzYTdiZGRhM2NmZjAzMjFkZjlkMzJmOCIsInZlcnNpb24iOjF9.qOlwWEe8dfBunmwImhbkcxzUW3ml-ESsuxjWN1fjn_o36zaUlDqlrXovMcL9GX9mVdvZDhx9W82rAR8h6410AQ
+    - type: rouge
+      value: 16.2044
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzkwOTEwYjkxYzlhMWE4ZjhlZDVjZWEwMWY2YzgwY2Q2YzJkYWFhMTQ4ODFlZmVkY2I1OWVhMTFmZThlOGY4NCIsInZlcnNpb24iOjF9.fJSr9wRQ07YIPMpb2_xv14EkHRz3gsPdZH-4LzpdviLOjVhlK1Y4gSZjp3PTEbu4Hua0umvNTMrhii8hp3DFBA
+    - type: rouge
+      value: 29.9765
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWRkYjcwMTYwODRjN2E4MDliZWQyNjczNDU1NGZkMDRkNDlhNDA1YzZiOTk1MWJjZDkyMDg3MGMxYmVhOTA5MyIsInZlcnNpb24iOjF9.tUkVmhT0bl9eY_BzAzdzEI1lo3Iyfv6HBrrsVsRHqPFh4C0Q9Zk3IXbR-F_gMDx9vDiZIkpfG7SfsIZXwhDkBw
+    - type: loss
+      value: 3.1985862255096436
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2RmYzQ1NTFiYjk3YTZjMTI3NDJlMDY0MTgyZDZlZDRmZDcwOWE1YjU0OGYyZTJlY2RkZTEzZDFlNDk2ZjgyNSIsInZlcnNpb24iOjF9.Pc5Tfu8IXYeB5ETK2JMIL4gpRIvvYXVS6w1AZdfq9dD1dm9Te2xaNhzGBHviqgEfFI9APNSJB28wna1OpYP0Dg
+    - type: gen_len
+      value: 191.9783
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmMyMDI5MzFlNzNjODNmOWQ0ZTM3MzVkNTNkYzIxNTIwZDQzMTU2MTM0YjYzNjJiMGRhOTQ0OWFhN2U4N2NjYyIsInZlcnNpb24iOjF9.AfsX-O1YwfbPxUwAD7rd1Ub7SXth7FFpTo2iNSOUWFhYmDUECkf6qtJ5pVHXXZwnpidAlfPTPg-5y3dx_BBGCA
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: samsum
+      type: samsum
+      config: samsum
+      split: test
+    metrics:
+    - type: rouge
+      value: 32
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmNhZjk3NjFlZDBhZjU2YzgzOTdhZTNkZjBkYjNjZDk2YjE2NDBmMDhiY2Y5M2EwNGI5Njk1NWU3ZDYyMzk2ZSIsInZlcnNpb24iOjF9.htkMQQLjIeFFjnpAJOwwxAdgzGZX10Und6RONubeeydXqQqb562EHqAw0K1ZlqltC4GBGKK3xslGOWXQ5AV6CA
+    - type: rouge
+      value: 10.0781
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWYzZDA1YmU5YTkzMjEwN2IzMTNhZmZmOTU2ZGUyNzdlNWQ0OGQ1Y2UxOGQ0NWUyOWVmZmZkYzFkODE3OTliNiIsInZlcnNpb24iOjF9.WVE3fmYLkOW32_neYYj4TNJ5lhrG-27DnoJd4YDUzpHYvGWGoFU9CUuIFraQFnojRr02f3KqVY7T33DG5mpzBg
+    - type: rouge
+      value: 23.6331
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYTYyOTE0ODY2Mjk0YTk5ZTY5NTZkM2JkOGZhNjQ3NjNiMjVhNTc4ZmMwYzg1ZGIxOTA2MDQxNmU3Yjc5YWY0MSIsInZlcnNpb24iOjF9.yQ8WpdsyGKSuTG8MxHXqujEAYOIrt_hoUbuHc8HnS-GjS9xJ-rKO6pP6HYbi0LC9Xqh2_QPveCpNqr9ZQMGRCg
+    - type: rouge
+      value: 28.7831
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzVkMDNlODA4NWI3OGI1OGFlNjFlNWE4YzY5ZDE1NDdhMjIwYjlkNDIxNDZjOGRiNTI1MGJkMmE0YWZiMDNhMiIsInZlcnNpb24iOjF9.qoxn2g70rbbX6sVCvm_cXzvYZf1UdTDU44vvEVdZL-4h36cJRCOx5--O1tZEVdyvlMVi-tYz1RSxLRwQd72FAw
+    - type: loss
+      value: 2.903024673461914
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZGM2M2NlY2Q3NjYxY2EyM2FkYmM5OGVhYzcyNjA3ZTFlYzc3M2M2ODNmNWVjNjZmMGNiODc4MWY5NWE2ZDMyNyIsInZlcnNpb24iOjF9.pC4UK75LbyVFFm0-fcStMtdQhbuHE37wkZHoVbSQOYSyxjI8yA46bQkPmgg5znby9FK_wIgGxC_4KOdEeN4jBw
+    - type: gen_len
+      value: 60.7411
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWEwMDFiYjgyNzRhZDVmOWIzYzZlZWU5OTFkYmU4YzI2Mjk2OTg1ZDVlNzU0YzNhOWI1MmU2NTAxZWUzZmFlOCIsInZlcnNpb24iOjF9.Zepow4AFj1sQ6zyJGoy_Dl4ICKRtzZI2nVYWlTsDnGrBDT42ak9mFUuw-BjHR8dEVHJKmOZlLk6GJ09bL7tGAA
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: cnn_dailymail
+      type: cnn_dailymail
+      config: 3.0.0
+      split: test
+    metrics:
+    - type: rouge
+      value: 30.5036
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMmFkM2M4YTcyODEwMzY1MWViYTY0NmEzNjYwNGM4OTI4MmY1ZTk2ZjVjZjMwOGUwM2JiYTA0YjdkMWRkZTQ5MyIsInZlcnNpb24iOjF9.GatKuC1oPoD1HT9pA9lGAj6GNjhe3ADSNgZ5apntAFCHETlNV1mNf1zQ-rgFH2FP-lF3qS56Jn54pFp6FMwaBw
+    - type: rouge
+      value: 13.2558
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjUwZjBmMTUzNmM3ZTRjODQ0MGFiM2I3Y2ViMDRkODQzNGI3YzM0MmJiNzU1N2UwOTZmMGFkOTQwMzNjNmFiMSIsInZlcnNpb24iOjF9.kOWpg36sB5GdPVYUZpWlS0pSKu5mKmHcLmJO1I3oUzMSiwDeUpAPLXNC0u_gJMFaFdsaNTywepDuttLdB2oBBg
+    - type: rouge
+      value: 19.0284
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTJmYzZmZWJiNTljYmJiZTllODk0NjdmNGNkZWZlZjMwMGE5YTAzMjMwNTcyNGM4MWE4MDUzYjM3NzQ5NzA2ZCIsInZlcnNpb24iOjF9.ooUqXvZC6ci_XxKrIcox2R2A0C8qyN0HP5djFMMb9SfoAaJAgdM0j6qsVQj9ccr0AgeRRIPNH_vI3gg-_lvaDw
+    - type: rouge
+      value: 28.3404
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZTcxMDg5ZGI1MDRmNzM0ZmEyZmNiZGYxZTg0NzA4N2U0YTY3MGYxMjgzMzI0NjVlNWNiYTZmNWZjMzZkMmYzNiIsInZlcnNpb24iOjF9.RbEZQB2-IPb-l6Z1xeOE42NGwX1KQjlr2wNL9VH75L1gmMxKGTPMR_Yazma84ZKK-Ai7s2YPNh-MDanNU_4GCw
+    - type: loss
+      value: 3.9438512325286865
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjQ2YmE1OTE5NDJlMTBhZGMzNDE5OThmNzMzOTRlYjEzMjc2ZDgyMDliNGY1NjFhOGQ0N2NkYmUzZGUwOGVlZiIsInZlcnNpb24iOjF9.FAwbzK-XJc-oEBFO7m8p4hkDCZDEhmU0ZSytrim-uHHcSFjRvbL-dF8rIvKVcxw5QeZ6QKZ7EkjDT7Ltt8KyCA
+    - type: gen_len
+      value: 231.0935
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTMzMTMyYjhhNjFiYjMyNDlhYzQzODM0MWNhNjkwMDVjNmFjYTk2NmQ4NzJlZjlhZjM2MGMwNWI1MjIxMGNiZCIsInZlcnNpb24iOjF9.mHDxhA2wVj6FDx7un4028-A8iGMFcPlSb5vH2DPGLPzQHBhSlvNac4-OELZf0PRmsXSb1nIqHqU-S_WUs8OSBg
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: billsum
+      type: billsum
+      config: default
+      split: test
+    metrics:
+    - type: rouge
+      value: 36.8502
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmE2ZjI4YmJkZGVjZDkzNzU5ZmI2MDYzNGZkNjE2OGM0Y2Y0Nzk1NTc1ZmUyZmFhYjIwY2RhMDVkMzQ1MWIxYyIsInZlcnNpb24iOjF9.SZjhhFkKwvRrI-Yl29psn17u1RCISsmmLVXxo2kxCjkhtMOma-EzC5YidjPDGQLb-J2nvqUworaC2pL_oeHxDQ
+    - type: rouge
+      value: 15.9147
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODgwOTJhOWIyZDQ4ZDA5YWMzYTJkZWFmMzlkNWYxNTg5OGFiNzY0MTExNTgyMTdlMTQ1N2EwYWY4OGZkNWY5YyIsInZlcnNpb24iOjF9.DS-X3eA1tGhVSuUL8uSPtJMNijODF3ugaKEtBglmPqF1OQZwIwQs-NExNYP4d6Y4Pa9d-DujD5yfyl9C8HBGCw
+    - type: rouge
+      value: 23.4762
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYTYxNTA4YzhmYTQ0YmRjMWU5ZDliZWFhMjM4ZmUyNGUyOWJhNzA1MDBhZDliYmYyYzY3NjBmZTZlYWY3YTY3ZCIsInZlcnNpb24iOjF9.o0W7dqdz0sqMPKtJbXSRpyVNsREEUypW-bGv7TW5lfJFkijfDKhVITEClFLWu5n2tIV-sXAYxgQHDf5_hpY-Dw
+    - type: rouge
+      value: 30.9597
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzEzOGNiYjk4NDkxNTFmMjA5YjM1YTQzZTk2N2JiZDgxNzAxYzFlYjliZjA3NmRjMzZlNGYyODBkNTI1NzVjNiIsInZlcnNpb24iOjF9.C_hobTR0ZY958oUZcGEKj2RoPOkyfMCTznwi4mUx-bfGRRAecMyn45bWVwwRq12glk1vThDetCjOMHA6jgSDCw
+    - type: loss
+      value: 3.878790855407715
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmYyOWM0YWQ0MjAxZDg5ZWQyNDk3MGUwNzdkOWIwZDc0OGJjYTU3YjZmOWY0YTljNDI0OWRlNTI0ZDMwZWEzOCIsInZlcnNpb24iOjF9.P01Jzfa-5jyMeoEqEsEluKOydNmtRtNy8YhwfJuYHVJTVDzCIfzY8b7iNfqTfKFKwKkZ4eTwmA6vmsPZeASDAw
+    - type: gen_len
+      value: 131.3622
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmJjN2Q5ZGNlZjQ2ODJiYTZlMzZmNWVmMzRlMGQ0ZTkxZWM3ZDQ4ZmQ1NmUyZjY4MTVhZGE5NDFiZTBhNDZiYSIsInZlcnNpb24iOjF9.DqYNc0ZCX_EqRi4zbSBAtb-js_JBHSWZkeGR9gSwEkJletKYFxPGZWd-B1ez88aj6PO775-qHd98xx3IWCHECQ
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: big_patent
+      type: big_patent
+      config: y
+      split: test
+    metrics:
+    - type: rouge
+      value: 33.7585
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiM2VmMGU5YWJlZWFlNjA3MDY2NTBmZWU3YWQxYTk3OGYzZmU5NmFmMTQ1NTVmNDQyZTJkNDMwY2E5NGRjMGU3MSIsInZlcnNpb24iOjF9.P6Rt9c3Xi_B-u8B1ug4paeZDoAO4ErGeNM0gELHGeOMj4XMjeSvyAW_-30cA9Wf23-0jGPOSZbN5pME4JpxfDA
+    - type: rouge
+      value: 9.4101
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDA0NzUxMjIwYTFjNGQ5YTA4YjE1NGU5YWMzYjhiOTk2NWE3ZGQxNDY4YTI3ZmI0ODBjYmJkZjcwYTM2OTg2MCIsInZlcnNpb24iOjF9.23hd2SuLoX3_Rygj2ykcSQccPeFsf4yLDAgvS189jx6JNln0MVR6YI2-3Yzo5g8LJk0MCbgkOp0my-nf7nMaDw
+    - type: rouge
+      value: 18.8927
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODhhMGZiZWFlNmZkYmYxZjJmODE1NWRiZjI2OGU1MTc4MDkyYjk1Mzk5ODFkYWVhY2ExNTViYjJmYzkzNWJhYiIsInZlcnNpb24iOjF9.SkKhf-l2cl2KcuC17oPrBtkBlZJaj2ujCgzRlfZy76rU9JtlW7N9bcy1ugnw-vRVUVVR6wUK08T45YorfuxqBg
+    - type: rouge
+      value: 28.5051
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMTgzYzA0NmQ0OTZmNzJkNGZiNTdmMzFmOTljMWE3YzM0NDg2MDY1ZDY5ZTE4MmQ5YzU1ZDFiNmE2ZjkwMjRjMiIsInZlcnNpb24iOjF9.p1TQINRxMatNe77_BMnusSg1K5FOD9f1_N4TBJDjJHNhYnyQDE4pKHfK8j6fsHGg58DHVQjmm8g96SK4uMF6DA
+    - type: loss
+      value: 5.162865161895752
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWM1YTQ4MjVmMDkyZDI3OWJmODhmOWE2MDYyMDA4OGRmYzhiY2YzZjVmMTZkMTI4NjBlY2MwMDY3ZDE5ZjlmMyIsInZlcnNpb24iOjF9.Czh4TOG-QIqyc_-GJ3wc1TLuxc-KLwPelV5tiwEjNhZFyUZkjLH__ccOxBk9TYy2vunvh2AwdY3Mt6Fr8LhaDA
+    - type: gen_len
+      value: 222.6626
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2JjNzVkODhmOWQ5NWMwNDdlNzhkYjE5NjY3NTgwNWVmZDZlMzc4NDdmZjdlN2M2ODBkZGU5NGU0ZjMzM2Q5OCIsInZlcnNpb24iOjF9.z4hZ-uXg8PPn-THRHFrsWZpS3jgE8URk5yoLenwWtev5toTrZ2Y-DP8O30nPnzMkzA4yzo_NUKIACxoUdMqfCQ
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: multi_news
+      type: multi_news
+      config: default
+      split: test
+    metrics:
+    - type: rouge
+      value: 38.7332
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMGViMThhNTdlZDRiMTg5NTZjNGVmOThiMjI5NDEyZDMxYjU4MTU2ZTliZjZmMzAzMmRhNDIxYjViYjZmNWYwNSIsInZlcnNpb24iOjF9.SK_1Q9WlkNhu3mfsyir1l72pddjURZvJV3mcJ4jhBxS2k2q1NAR8JT_iT8v1thLiv8NUDmDr2o9Dig4A8svDBw
+    - type: rouge
+      value: 11.0072
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzkzMDU1ZGZlOWUwOGQyY2UwMWFjZTY1MDBmNzcyZGYzZTliNGVkNDZjZDVjZjA4NmE3OWVhMGIyZmE3NGE0NSIsInZlcnNpb24iOjF9.j0wvR0NPw0lqxW3ASbmBvxAbFHGikXw-Y7FjutojhzTfSs3BIs5Z8s5_h6eesvSGT5fS_qUrbnl9EEBwjrXqDg
+    - type: rouge
+      value: 18.6018
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjIwNTUzN2ZhZjU5OGFhYzRmZmEwY2NkZWVjYmYzZjRjMGIxNzNjZDY5YzIyMTg2NDJkMGYxYmViNTcwOTc5NCIsInZlcnNpb24iOjF9.rD_tFYRyb-o6VX7Z52fULvP_HQjqqshqnvbjAxWjuCM9hCn1J6oh0zAASPw0k1lWiURbiMCiaxIHxe_5BN_rAQ
+    - type: rouge
+      value: 34.5911
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2Q4MWY3NGFhNjE5YjE5NzIyODVhNTYxNWFmZDE5NjNiZTM1M2M3ZmIwNTZiOWEyMTc2MzQ0MWQ5YTdjYThlNyIsInZlcnNpb24iOjF9.R789HgYsv_k6OrjocVi0ywx0aCRlgOKpEWUiSUDca-AfoDS8ADJBtLYoEKg1wnRlR9yWoD4vtEWdKbyOOln1CA
+    - type: loss
+      value: 3.5744354724884033
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzBjZTk0YWMwMzQxNDRlY2UxZDc4NTE1MmEzNDkwM2M3ZGZhNGMzNmI4ZDU2ZTVhZDkwMjNhYTkxZTIwN2E4MyIsInZlcnNpb24iOjF9.bDQ_3-CumosWKroMwBEMwKnDAj4ENQbUnbS387hU0zAY1K5g1NOy7fKBohxYZnRVolEfiuhszifUMW9zcLjqCA
+    - type: gen_len
+      value: 192.0014
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDQxZmEwYmU5MGI1ZWE5NTIyMmM1MTVlMjVjNTg4MDQyMjJhNGE5NDJhNmZiN2Y4ZDc4ZmExNjBkMjQzMjQxMyIsInZlcnNpb24iOjF9.o3WblPY-iL1vT66xPwyyi1VMPhI53qs9GJ5HsHGbglOALwZT4n2-6IRxRNcL2lLj9qUehWUKkhruUyDM5-4RBg
+  - task:
+      type: summarization
+      name: Summarization
+    dataset:
+      name: xsum
+      type: xsum
+      config: default
+      split: test
+    metrics:
+    - type: rouge
+      value: 16.3186
+      name: ROUGE-1
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjNiYzkxNTc1M2ZiYzY4NmVhY2U4MGU0YWE1NzQ4YzQxNjM1ZThmOWU3ZjUwMWUxMWM1NTQyYzc0OWQ5MzQyZSIsInZlcnNpb24iOjF9.cDZzbzxrXaM4n-Fa-vBpUgq7ildtHg9hlO5p9pt58VYLGK3rsid3oUE2qsFH6Qk63j2cF4_hzgq93xoVlnR3Dg
+    - type: rouge
+      value: 3.0261
+      name: ROUGE-2
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjkzNzA0ODk3NWJjOGM2ZWFlY2MyZWM4NzZlYzZiMGQ2ODc0NzgzNDYzYmVlZjg2ZjBmNDMwOGViYTljYWQ2NSIsInZlcnNpb24iOjF9.ohBfAUhEktfITK6j_NusN5SOmF4XUHZWPNMpGrsGXRHTf1bUl6_UEQ0S3w58WQsgIuV3MkxWNRBU1oZAm3fbBQ
+    - type: rouge
+      value: 10.4045
+      name: ROUGE-L
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMDM2ZDZhYzBiNGM3NDdhODlmNjJhMTNlZDE3ZTZmYjM1MWU5YmE0ODMyZGFhMmM0YmMwMzNiZWU4ZDAzMDFlNiIsInZlcnNpb24iOjF9.653PFaov_0t8g_fVyVxm8DBx7uV4646yK0rtxOxC7qsnRdljdThSOklw9tND5-44WdkzipzuLyVzq1qe-TbKBA
+    - type: rouge
+      value: 12.612
+      name: ROUGE-LSUM
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmY5YzU2ZjE2OWM0ZGQwZmVjZjQwZTQ0MDNkZmNiMTdhZjFkMDA5OGFhYWQ0Y2QwZDY0YWJlNWUxZGQ0YTUwZiIsInZlcnNpb24iOjF9.RXyu1jIj_gV26WCHSGHZufWXKFEexuRaLD4gkOvlBcaXJrFoE11tttB6mYzN6Tk8qx5cvV5L_ZIUfDmOqunkAA
+    - type: loss
+      value: 3.323798179626465
+      name: loss
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjU5ZWUxMjIwMWYwNDY1YzUwMzUxNGFiZWI3ZDVhZDFlYzJhNzk3MjA1OGExNTg0NjZlOGQyYzBiZjdhN2E2YSIsInZlcnNpb24iOjF9.vFxH1vHAACKE4XcgBhuoaV38yUZuYJuNm23V3nWVbF4FwyN79srV3Y9CqPGoOiIoUSQJ9fdKZXZub5j0GuUJAA
+    - type: gen_len
+      value: 149.7551
+      name: gen_len
+      verified: true
+      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzg1ZjY5MTJkMTgzMjhiYzMxNjkyZjlmNmI2ZGU0YTRhZjU5NjQwOWE5MjczZDIxNGI1MGI4YzhhOGVkZDFkYSIsInZlcnNpb24iOjF9.S7W5-vqldJuqtC5MweC3iCK6uy-uTRe4kGqoApMl2Sn6w9sVHnY7u905yNLXzFLrLYMgjlct5LB7AAirHeEJBw
+---
+# LED-Based Summarization Model: Condensing Long and Technical Information
+
+<a href="https://colab.research.google.com/gist/pszemraj/36950064ca76161d9d258e5cdbfa6833/led-base-demo-token-batching.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+The Longformer Encoder-Decoder (LED) for Narrative-Esque Long Text Summarization is a model I fine-tuned from [allenai/led-base-16384](https://huggingface.co/allenai/led-base-16384) to condense extensive technical, academic, and narrative content in a fairly generalizable way.
+
+## Key Features and Use Cases
+
+- Ideal for summarizing long narratives, articles, papers, textbooks, and other documents.
+  - the sparknotes-esque style leads to 'explanations' in the summarized content, offering insightful output.
+- High capacity: Handles up to 16,384 tokens per batch.
+- demos: try it out in the notebook linked above or in the [demo on Spaces](https://huggingface.co/spaces/pszemraj/summarize-long-text)
+
+> **Note:** The API widget has a max length of ~96 tokens due to inference timeout constraints. 
+
+## Training Details
+
+The model was trained on the BookSum dataset released by SalesForce, which leads to the `bsd-3-clause` license. The training process involved 16 epochs with parameters tweaked to facilitate very fine-tuning-type training (super low learning rate). 
+
+Model checkpoint: [`pszemraj/led-base-16384-finetuned-booksum`](https://huggingface.co/pszemraj/led-base-16384-finetuned-booksum). 
+
+## Other Related Checkpoints
+
+This model is the smallest/fastest booksum-tuned model I have worked on. If you're looking for higher quality summaries, check out:
+
+- [Long-T5-tglobal-base](https://huggingface.co/pszemraj/long-t5-tglobal-base-16384-book-summary)
+- [BigBird-Pegasus-Large-K](https://huggingface.co/pszemraj/bigbird-pegasus-large-K-booksum)
+- [Pegasus-X-Large](https://huggingface.co/pszemraj/pegasus-x-large-book-summary)
+- [Long-T5-tglobal-XL](https://huggingface.co/pszemraj/long-t5-tglobal-xl-16384-book-summary)
+
+There are also other variants on other datasets etc on my hf profile, feel free to try them out :)
+
+---
+
+## Basic Usage
+
+I recommend using `encoder_no_repeat_ngram_size=3` when calling the pipeline object, as it enhances the summary quality by encouraging the use of new vocabulary and crafting an abstractive summary.
+
+Create the pipeline object:
+
+```python
+import torch
+from transformers import pipeline
+
+hf_name = "pszemraj/led-base-book-summary"
+
+summarizer = pipeline(
+    "summarization",
+    hf_name,
+    device=0 if torch.cuda.is_available() else -1,
+)
+```
+
+Feed the text into the pipeline object:
+
+```python
+wall_of_text = "your words here"
+
+result = summarizer(
+    wall_of_text,
+    min_length=8,
+    max_length=256,
+    no_repeat_ngram_size=3,
+    encoder_no_repeat_ngram_size=3,
+    repetition_penalty=3.5,
+    num_beams=4,
+    do_sample=False,
+    early_stopping=True,
+)
+print(result[0]["generated_text"])
+```
+
+## Simplified Usage with TextSum
+
+To streamline the process of using this and other models, I've developed [a Python package utility](https://github.com/pszemraj/textsum) named `textsum`. This package offers simple interfaces for applying summarization models to text documents of arbitrary length. 
+
+Install TextSum:
+
+```bash
+pip install textsum
+```
+
+Then use it in Python with this model:
+
+```python
+from textsum.summarize import Summarizer
+
+model_name = "pszemraj/led-base-book-summary"
+summarizer = Summarizer(
+    model_name_or_path=model_name,  # you can use any Seq2Seq model on the Hub
+    token_batch_length=4096,  # how many tokens to batch summarize at a time
+)
+long_string = "This is a long string of text that will be summarized."
+out_str = summarizer.summarize_string(long_string)
+print(f"summary: {out_str}")
+```
+
+Currently implemented interfaces include a Python API, a Command-Line Interface (CLI), and a shareable demo/web UI. 
+
+For detailed explanations and documentation, check the [README](https://github.com/pszemraj/textsum) or the [wiki](https://github.com/pszemraj/textsum/wiki)
+
+---

config.json

@@ -0,0 +1,60 @@
+{
+  "_name_or_path": "pszemraj/led-base-book-summary",
+  "activation_dropout": 0.0,
+  "activation_function": "gelu",
+  "architectures": [
+    "LEDForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "attention_window": [
+    1024,
+    1024,
+    1024,
+    1024,
+    1024,
+    1024
+  ],
+  "bos_token_id": 0,
+  "classif_dropout": 0.0,
+  "classifier_dropout": 0.0,
+  "d_model": 768,
+  "decoder_attention_heads": 12,
+  "decoder_ffn_dim": 3072,
+  "decoder_layerdrop": 0.0,
+  "decoder_layers": 6,
+  "decoder_start_token_id": 2,
+  "dropout": 0.1,
+  "early_stopping": true,
+  "encoder_attention_heads": 12,
+  "encoder_ffn_dim": 3072,
+  "encoder_layerdrop": 0.0,
+  "encoder_layers": 6,
+  "eos_token_id": 2,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "length_penalty": 0.8,
+  "max_decoder_position_embeddings": 1024,
+  "max_encoder_position_embeddings": 16384,
+  "max_length": 1024,
+  "min_length": 8,
+  "model_type": "led",
+  "no_repeat_ngram_size": 3,
+  "num_beams": 4,
+  "num_hidden_layers": 6,
+  "pad_token_id": 1,
+  "repetition_penalty": 3.5,
+  "torch_dtype": "float32",
+  "transformers_version": "4.19.2",
+  "use_cache": true,
+  "vocab_size": 50265
+}

evals-outputs/GAUNTLET.md

@@ -0,0 +1,292 @@
+# gauntlet results
+
+These are are this model's output results on my "summarization gauntlet". You can find more info about that [here on my dropbox for it](https://www.dropbox.com/sh/axu1xlscrrexy55/AADAm01-4Zs3POyHQrgbDAsda?dl=0) or at [this dataset](https://huggingface.co/datasets/pszemraj/summcomparer-gauntlet-v0p1).
+
+- if you aren't familiar with it, one thing to note is some of the docs **purposefully** are "messy"/have spelling errors etc.
+
+parameters
+
+```json
+{
+  "model_name_or_path": "pszemraj/led-base-book-summary",
+  "use_cuda": true,
+  "token_batch_length": 16384,
+  "batch_stride": 16,
+  "max_length_ratio": 0.25,
+  "load_in_8bit": false,
+  "compile_model": true,
+  "optimum_onnx": false,
+  "device": "cuda",
+  "inference_params": {
+    "min_length": 8,
+    "max_length": 4096,
+    "no_repeat_ngram_size": 3,
+    "encoder_no_repeat_ngram_size": 4,
+    "repetition_penalty": 2.5,
+    "num_beams": 10,
+    "num_beam_groups": 1,
+    "length_penalty": 1.0,
+    "early_stopping": true,
+    "do_sample": false
+  },
+  "textsum_version": "0.2.0"
+}
+```
+
+- Created: `2023-11-28T20:10:42.701696`
+
+## ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853561_0_part1_summary
+
+Well, he's going to talk a little bit about the basic issues of linguistics in two separate talks. First, he wants to talk about how we're trying to understand language. Specifically, what do we mean by "real" explanations? Theoretically, there are three main things: 1) an authentic explanation; 2) a genuine solution; and 3) a kind of combinatorial approach to understanding language. Brain scientists have been working on this for quite some time, but they haven't yet come up with a really good explanation. In the early 1900s, people like Otto Jesperson, Leonard Bloomfield, Robert Ademian, and others all tried to figure out how the structure of language comes into existence in their minds. They all thought that it was just a bunch of random bits of information floating around in their brains. But now science is starting to use machine learning as a way to understand languages.
+
+---
+
+Section Scores for ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853561_0_part1_summary:
+
+ - -0.9902
+
+---
+
+## ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853631_0_part2_summary
+
+Adam explains the concept merge, which he calls a "superior version of merge" because it has only two elements: an external and an internal merge. In order to explain how one can combine multiple objects, however, one must first have access to all of them. This is particularly important in the context of machine learning, where there are so many different kinds of concurrent operations that it becomes very difficult to understand what each operation accomplishes.
+Pisa argues that the problem is not just a problem with the structure of the examples but also with the problems inherent in the field. The problems are "the kinds of problems" that arise when one tries to give an adequate explanation of what's happening in the real world. For example, if you create a new object and then merge it with another object, you don't get exactly what you wanted; instead, you end up using the same paradigm as all the examples before. In other words, there isn't a single solution to the problem because there aren't enough problems to solve in a principled way.
+
+---
+
+Section Scores for ASR-whisper-rpunctuated_Noam Chomsky, Fundam_1669853631_0_part2_summary:
+
+ - -0.9733
+
+ - -0.9339
+
+---
+
+## ASRnlp_law_lecture_week_1_v_2_c_transcription_1_summary
+
+This is the first chapter of a new course called "Natural Language Processing", which focuses on natural language processing. The course will focus on using machine learning and artificial intelligence to understand legal and social science applications. It will also take place in person rather than in a web-based environment. There will be no overtaken by artificial intelligence or machine learning, but there will certainly be hybrid learning and hybrid learning.
+
+---
+
+Section Scores for ASRnlp_law_lecture_week_1_v_2_c_transcription_1_summary:
+
+ - -0.9214
+
+---
+
+## ASRnlp_law_lecture_week_2_v_2_c_transcription_2_summary
+
+Gulliver explains the changes that have been made to the course in order to better prepare students for the next generation of jobs. For example, there is now more room available for people to join remotely. There are also new questions asking if homework emissions can be counted as gas or non-acronym courses and how they should be counted. The first question is whether homework emissions must be copy-and-pasted from the note; this is important because it will allow students to know what's going on in their own notebooks at the time of joining. Next up is a survey of corporate and dictionary methods. This paper focuses on using machine learning to identify which documents are most likely to be classified as being classified as having "minor" or "high dimensionality". It also looks at ways to reduce the amount of time that companies spend on these documents.
+
+---
+
+Section Scores for ASRnlp_law_lecture_week_2_v_2_c_transcription_2_summary:
+
+ - -1.0658
+
+---
+
+## ASRnlp_law_lecture_week_3_part_1_v_2_c_transcription_3_summary
+
+The hybrid learning experts discuss immigrants and how they can reduce the number of words that are used in the English language. They begin with a discussion about immigrants and then move to a discussion of "reconcilable differences" between the different types of immigrants. This paper focuses on three methods for achieving this goal. The first method is called tariff similarity; the second method is Energetic Analysis. The third method is termed elastic search. The fourth method is time-series analysis. It uses machine learning to predict the frequency of words, phrases, and events in order to identify which words are most likely to be associated with default.
+
+---
+
+Section Scores for ASRnlp_law_lecture_week_3_part_1_v_2_c_transcription_3_summary:
+
+ - -0.9206
+
+---
+
+## Emie_dissertation_cleansed_summary
+
+Szemraj's dissertation is a "Master of Philosophy" in film and screen studies. It focuses on three subjects: urban space, materiality, and movement in post-war American and British films noir; act of violence; and the man between. Act of Violence takes place in Los Angeles during the aftermath of the German invasion of the United States in World War II. The focus of the film is on how the characters move through the city as if they are moving from physical reality to metaphysically real reality. In Act 2, for example, Enley walks through Bunker Hill with his dog, Parkson. When he reaches the train tracks, he sees that they are part of the wreckage of the Berlin Airlift.
+The Man Between and Act of Violence are also notable for their use of "physical correspondences" to tell the story of Susanne's captivity. Ivo attempts to persuade Susanne to go to the opera that evening, but Susanne misinterprets him as a renegade accomplice in a plot to destroy her identity. When Ivo finally succeeds in capturing the attention of the border patrol officers, he runs away into the snow. The sequence culminates with Ivo's pursuit of an errand boy who has hidden behind a laundry truck. In this sequence, Reed uses "material reality" to heighten and reveal how much Susanne is afraid of Ivo. As the action moves forward, it becomes increasingly difficult for Ivo to escape the physical reality of Berlin. Both Enley and Parkon move through suburbia or hotel lobbies in order to shed their past and present identities; however, they cannot escape the sense of alienation felt by the world around them. Thus, both films emphasize the importance of "reconciling material reality" with "artificiality."
+
+---
+
+Section Scores for Emie_dissertation_cleansed_summary:
+
+ - -0.9783
+
+ - -0.9207
+
+---
+
+## OCR_ML4HLecture02image__summary
+
+This paper focuses on machine learning for medical image analysis. Machine Learning is used to predict the volume and length of abdominal organs and focal points in order to predict disease, stroke, and neurodegeneration. Dr. Barlow demonstrates how machine learning can outperform human-to-machine translation in predicting disease outcomes.
+
+---
+
+Section Scores for OCR_ML4HLecture02image__summary:
+
+ - -0.8923
+
+---
+
+## OCR_ML4HLecture04RepresentationLearning.pptx__summary
+
+Next, we look at how machine learning can improve the accuracy of human-caused disease prediction using time series. we use sequence-based representation learning to predict disease outcomes using real-world medical time series and compare it favorably with competitor approaches. Next, we examine how "reconcoder and forecaster" models can outperform each other in predicting disease outcomes.
+
+---
+
+Section Scores for OCR_ML4HLecture04RepresentationLearning.pptx__summary:
+
+ - -0.8706
+
+---
+
+## OCR_ML4HLecture05-NLP.pptx__summary
+
+Artificial Intelligence (AI) and Natural Language Processing (NLP) we examine the use of artificial intelligence to predict disease-causing behavior in real-time. Specifically, we look at how NLP tasks can be scaled down based on the frequency of occurrences of certain terms in the document. For example, if you're trying to figure out which word is most likely to show up in a given sentence, you'll find that it's actually showing up in the next sentence. The same thing happens with speech recognition.
+
+---
+
+Section Scores for OCR_ML4HLecture05-NLP.pptx__summary:
+
+ - -0.9732
+
+---
+
+## OCR_PAPER_Hong et al. - 2022 - CogVideo Large-scale Pretraining for Text-to-Video Generation via Transformers-annotated__summary
+
+Cog' Video, a 9B-rate transformer trained to automatically predict the future frames of text in Chinese, uses Cog' video as an example. The model outperforms its peers in both machine and human evaluation. the Tsinghua University Department of Electrical and Computer Engineering (IBEA) conducts a detailed assessment of the performance of these transformers. It is shown that they outperform all publicly available transformers when compared to other types of transformers used for text generation. Next, the team presents Cog 'Video, which is the world's first fully-fledged pretrained neural network capable of predicting future frames from real-time information. To better understand the temporal relationships between images and text, the task is further advanced by using artificial intelligence techniques such as transcoding and image generation.
+
+---
+
+Section Scores for OCR_PAPER_Hong et al. - 2022 - CogVideo Large-scale Pretraining for Text-to-Video Generation via Transformers-annotated__summary:
+
+ - -1.1301
+
+---
+
+## OCR_PAPER_Kandpal, Nieto, Jin - 2022 - Music Enhancement via Image Translation and Vocoding-annotated__summary
+
+Kandpal explains how he and his co-workers can improve the sound quality of music by using a combination of machine learning, maelstrom modeling, and vocoding. In particular, they demonstrate that their approach outperforms all other methods currently used to improve music quality. For example, they train a MelzMel, Diffwave, and Mel2Mel models on top of each other in order to optimally compare different levels of musical quality. The results are even better than those obtained from purely independent training.
+
+---
+
+Section Scores for OCR_PAPER_Kandpal, Nieto, Jin - 2022 - Music Enhancement via Image Translation and Vocoding-annotated__summary:
+
+ - -0.9349
+
+---
+
+## OCR_PAPER_dall-e-2-annotated__summary
+
+The next step in this study is to develop contrastive image generation using CLIP. To do so, the team uses diffusion models and autoregressive models. In order to achieve these results, they train multiple decoders to produce multiple images of the same image. They then use diffusion models to predict which image will be used for image generation. The final result is a high-quality dataset with over 1,000 samples from production versions of the paper.
+
+---
+
+Section Scores for OCR_PAPER_dall-e-2-annotated__summary:
+
+ - -0.8975
+
+---
+
+## The Most Dangerous Game--Richard Connell_summary
+
+Rainsford swears that they have reached a large island, which is rumored to be inhabited by cannibals. When Rainsford reaches the island, he feels a sudden chill and a sense of uneasiness. The ship's captain, Captain Nielsen, had warned him that the island was "an evil place" and asked him not to feel anything. He also remembers seeing an old Swede who had told him that sailors should not feel anything because the air about them was poisonous. After swimming for a few minutes in the dark, RainsfORD hears shots off in the distance. He swears to go back to sleep. On his way home, he stops at a dinner party where General Zaroff invites him to dine with him. During the course of their conversation, general Zaroff reveals that he has found a new species of deer--the Cossack--and plans to kill him if he does not find him by 3:00 p.m. the next day.
+
+---
+
+Section Scores for The Most Dangerous Game--Richard Connell_summary:
+
+ - -1.0244
+
+---
+
+## gpt_peter_testing_group_exemplars_summary
+
+In order to prepare for the hackathon that is to take place on Thursday evenings, Peter and his co-workers decide to create a meme about Cheesy Poofs, an American made product banned by the U.S. government because of its graphic content. The mothers of the boys ban the sale of the American-made cheese poofs in order to discourage the boys from buying them. They also issue an ultimatum to the boys regarding what they should do with their money if they want to make a profit. In order to avoid being labeled as a "hard worker," Peter changes the form of the application so that it conforms to the needs of the class. He also adds more soldiers to respond to Paul's new proposal. Lastly, the group decides to ask each other questions about their day's activities, their hobbies, and their belief in Solipsism. At the end of their training, they will discuss how they can improve the algorithm for saving the world.
+
+---
+
+Section Scores for gpt_peter_testing_group_exemplars_summary:
+
+ - -0.9597
+
+---
+
+## navy seals copy pasta_summary
+
+I will wipe you out with precision, mark my words. I will show you everything from the navy seals to the entire US marine corps. I will use your pathetic little life as an opportunity to wipe it all out with "unleasurable" vengeance. You're fucking dead.
+
+---
+
+Section Scores for navy seals copy pasta_summary:
+
+ - -1.0145
+
+---
+
+## script_findingnemo_summary
+
+FINDING Nemo: The First Day of School This is a work in progress transcript of the first part of the movie. It's not 100% accurate, and some mistakes may be made, but it's open for corrections if you have any. Marlin says he likes the neighborhood, but he doesn't want to name all of his kids after a clown. He wants to name them all "Marlin Jr." and "Coral Jr.". Brain Snack: A clownfish is a fish that swims from one place to another. In this scene, the clownfish are introduced as Dory and Bekker. They're trying to figure out how to get past the jellyfish when they run into each other on the way to meet their new families. When they finally find their families, though, they can't communicate with each other because they don't recognize each other. Everyone runs off to find their parents.
+Nemo's dad, Gulliver, shows up with the news that Nemo has taken on three giant sharks. Brain Snack: Sharkbait isn't just looking for you, it's also predicting that Darla will arrive in Sydney sometime in the next 48 hours or so. In other words, they don't have much time to clean up before Darla arrives--they only have a little over 48 hours until Darla is due to arrive. The boys scramble to find their exit buddy, Squirt, who'll give them some tips on how to get out of this sticky situation. They decide to play a game of hide-and-seek in order to figure out where Dory is and what he's doing. And guess what? There's a fish floating in the water. It's called the AquaScum 2,000, which is basically a two-hundred-thousand-year-old biodegradable fish tank. Everyone freaks out except for Dory, who tries to talk to the fish by pretending to be a pelican. But Dory convinces him to try talking to Nemo instead. At the end of the chapter, Dory runs off to find his son.
+
+---
+
+Section Scores for script_findingnemo_summary:
+
+ - -1.0351
+
+ - -0.9937
+
+---
+
+## script_frozendisney_summary
+
+In this short scene, the ice men work on a frozen lake. They sing a song about love and fear as they push their ice blocks through the water. An ice floe approaches them threateningly, but they fight it off. The workers pile onto a horsedrawn ice sled and head off into the Northern Lights. At night, the girls sleep in their beds. In the morning, they wake up to find Anna asleep on their bed. She has been injured by a piece of ice that accidentally knocks her over. When she wakes up, she realizes that there is no way for her to get out of the ice. A group of people from Arendelle gathers to watch the coronation of the king and queen. J. Lee introduces his newest addition to the party: Grand Pabbie. He changes all of Anna's magical memories to mundane ones like snowflakes and ice cream. As he instructs the reindeer how to control their icy hands, Anna skips down a snowy slope and lands on an ice wall. On top of her, she runs into a giant snowstorm. It freezes her whole body before she can get back up. During the celebration, Bjorn and Jaquenetta are introduced.
+On the way to the ice palace, Olaf tries to kiss Anna but accidentally slams his head against a puddle of ice. He falls unconscious. Bjorn and Jaquen decide to go to the love experts who will help them find Anna. On the way home, they run into J. Lee and Olaf at an ice palace. They try to get away, but when they do, they are caught in a huge snowball-jacking. Everyone runs away except for Bjorn, who manages to save Anna's life. In the end, Bjorn is sentenced to death.
+As the storm continues, Anna and Olaf rush down the hall in a panic. The walls of the building crack under the force of the storm, preventing them from escaping. When they reach the top of the wall, however, they are struck by ice spikes that block their path. They fall to the ground as a giant snowball. During this brief scene, it is revealed that Anna's sister is dead because of her. It is also revealed that Kristoff has risked his life to rescue Anna. As he runs towards her, he accidentally hits her with his sword. He falls overboard. Kai, the dockmaster, tells the guards that Arendelle will no longer conduct any dealings with HANS. The guards take him off the ship and send him back to Weselton. On the way home, Anna pulls a blindfold over her face and embraces Sven. She kisses him on the lips. At the end of the scene, Olaf skates through the crowd and helps Anna into the sleigh.
+
+---
+
+Section Scores for script_frozendisney_summary:
+
+ - -1.0835
+
+ - -1.0709
+
+ - -0.9543
+
+---
+
+## script_strangersonatrain_summary
+
+Bruno Haines, a young man in expensive clothes, and Anthony, a troubled young man, appear on the train platform. They are greeted by Bruno, who apologizes for his over-the-top smoking and remarks that he is too busy to attend the doubles match between Evans and Faraday. The conversation then moves to news of people divorcing or getting remarried. In this brief scene, they are interrupted by the arrival of a police car. As they enter the main part of the city, they witness the arrest of two men whom they believe have plotted to murder one another. At the same time, they also witness the arrests of three other men accused of plotting against them.
+At the police station, Barbra tells Guy that she saw him on the train from New York City to Washington, D. C., and that he should be able to tell them where he was at nine:00 a.m. That night, Guy goes to the office of Captain Turley, who is supposed to be there. When he enters, he finds Professor Collins waiting behind him. He explains that he has been able to identify the man who tried to murder Mrs. Cunningham in the book The Murder of Mr. Metcalf by using his own name. At the same time, Haines notes that one of the Burtons' friends, Antony, has helped mastermind the murder of Dr. Calf. As they are leaving the apartment, Bertha notices that Guy is staring at her. She realizes that he is trying to talk to her about the murder and decides to leave the apartment for a few days so that she can figure out how to get rid of him. While she is gone, Anne overhears a conversation between Bruno and Miss Darville.
+Bruno and Anne are in the midst of a conversation about Guy. He tells his mother that Guy should not have sent her on an errand like this, because Guy is irresponsible and will do anything for Lady Macbeth's reputation. When Anne gets up to leave, she sees Guy's lighter in her pocket. The police need something to prove that he was at the murder scene--something to prove they were at the crime scene that night. As soon as the game begins, Guy and Reynolds win the first set with a flourish. At the end of the third set, though, Guy smashes one of the horses' heads against the wall. In the fourth set, Guy kills another horseman who tries to get underneath the top of the second horse. A bystander catches a glimpse of the two men getting into a fight over a broken wheelbarrow. An onlooker rushes in to see what has happened. It turns out that Haines beat the man who tried to steal his lighter during the tournament. After the game, Ganymede and Hennessy take their cab back to the station.
+
+---
+
+Section Scores for script_strangersonatrain_summary:
+
+ - -1.0462
+
+ - -1.0986
+
+ - -1.1284
+
+---
+
+## script_sunsetblvd._summary
+
+On March 21, 1949, the Los Angeles Daily News publishes a picture of Sunset Boulevard as it was before the shooting began. Two men enter the scene and give Gillis a business card to use to get the car keys. They explain that they need to jack up the car and haul it away because they have a court order. The car is a 1946 Plymouth convertible with two shots in its back and one in its stomach. It has been loaned to a man named Sheldrake who lives next door to him. He had lent the car to a guy named Bill Demarest who took it to Palm Springs for his health. At the same time, he has borrowed three hundred dollars to pay for another car. When the men leave, Gillis' VOICE is heard on the other end of the room -- Norma's -- asking him if he can borrow some money to write a piece about her. He does not want to do anything more than talk to her. In fact, he tells her that he will send her a copy of Salome.
+Gillis is embarrassed at the way Norma treats him. He tries to tell her that he doesn't want her to love him, but she tells him that he's just trying to get her to think of someone else. Gillis goes back to his apartment and finds a box with two bars of music on it. When he opens the box, there's a note saying "To Joe from Northma," which says "Mad About the Boy." The next day, Gillis goes off to New Year's Eve party at Artie Green's apartment. At the end of the night, he calls in to see how things are going. It turns out that Artie has been working on a script for a play called Norma's Salome. Afterward, they head over to De Mille's office to have a chat. There's a knock on the door as soon as they get there. In the meantime, we learn that Norma has had an affair with a guy named Gordon Cole. We also learn that de Mille wants to hire one of his assistants to help him finish his script.
+Gillis tells Betty that Artie has sent her a telegram asking her to come to Arizona with him. It only costs two dollars to get married there, so it would be nice for them to have a honeymoon together. As they are finishing the story, Norma's bedroom doorbell rings and Gillis comes in to talk to her. He tries to give her some more clues but she is too scared to do so. She shoots both men in rapid succession and he falls into the pool. At the end of the scene, DeMille rushes out to save the life of Norma.
+
+---
+
+Section Scores for script_sunsetblvd._summary:
+
+ - -1.0416
+
+ - -1.0423
+
+ - -0.8555
+
+---
+

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a08ab8231e58381a51d97153e6715d14b2e56271fb830ca66f385c100c8e6f1
+size 647614118

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8c74d469611ef8013b779aefc5cab0cf0cd16bd65dfe5c90d4aeb4dd20e57ec
+size 647616935

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"errors": "replace", "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "trim_offsets": true, "model_max_length": 16384, "special_tokens_map_file": "/root/.cache/huggingface/transformers/05da652a7fca41c1c18027c1201e473217bb373e370d1283e3de49d5880cbf0c.cb2244924ab24d706b02fd7fcedaea4531566537687a539ebb94db511fd122a0", "name_or_path": "pszemraj/led-base-book-summary", "tokenizer_class": "LEDTokenizer"}

trainer_state.json

@@ -0,0 +1,781 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.9994069974303224,
+  "global_step": 632,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03,
+      "learning_rate": 1.6398077091987075e-05,
+      "loss": 2.639,
+      "step": 5
+    },
+    {
+      "epoch": 0.06,
+      "learning_rate": 2.3460344492986226e-05,
+      "loss": 2.6696,
+      "step": 10
+    },
+    {
+      "epoch": 0.09,
+      "learning_rate": 2.7591506092636183e-05,
+      "loss": 2.6473,
+      "step": 15
+    },
+    {
+      "epoch": 0.13,
+      "learning_rate": 3e-05,
+      "loss": 2.6481,
+      "step": 20
+    },
+    {
+      "epoch": 0.16,
+      "learning_rate": 3e-05,
+      "loss": 2.641,
+      "step": 25
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 3e-05,
+      "loss": 2.663,
+      "step": 30
+    },
+    {
+      "epoch": 0.22,
+      "learning_rate": 3e-05,
+      "loss": 2.6205,
+      "step": 35
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 3e-05,
+      "loss": 2.6176,
+      "step": 40
+    },
+    {
+      "epoch": 0.28,
+      "learning_rate": 3e-05,
+      "loss": 2.6764,
+      "step": 45
+    },
+    {
+      "epoch": 0.32,
+      "learning_rate": 3e-05,
+      "loss": 2.6393,
+      "step": 50
+    },
+    {
+      "epoch": 0.35,
+      "learning_rate": 3e-05,
+      "loss": 2.6457,
+      "step": 55
+    },
+    {
+      "epoch": 0.38,
+      "learning_rate": 3e-05,
+      "loss": 2.6594,
+      "step": 60
+    },
+    {
+      "epoch": 0.41,
+      "learning_rate": 3e-05,
+      "loss": 2.6146,
+      "step": 65
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 3e-05,
+      "loss": 2.6564,
+      "step": 70
+    },
+    {
+      "epoch": 0.47,
+      "learning_rate": 3e-05,
+      "loss": 2.6465,
+      "step": 75
+    },
+    {
+      "epoch": 0.51,
+      "learning_rate": 3e-05,
+      "loss": 2.6293,
+      "step": 80
+    },
+    {
+      "epoch": 0.54,
+      "learning_rate": 3e-05,
+      "loss": 2.6427,
+      "step": 85
+    },
+    {
+      "epoch": 0.57,
+      "learning_rate": 3e-05,
+      "loss": 2.6203,
+      "step": 90
+    },
+    {
+      "epoch": 0.6,
+      "learning_rate": 3e-05,
+      "loss": 2.6286,
+      "step": 95
+    },
+    {
+      "epoch": 0.63,
+      "learning_rate": 3e-05,
+      "loss": 2.6523,
+      "step": 100
+    },
+    {
+      "epoch": 0.66,
+      "learning_rate": 3e-05,
+      "loss": 2.6239,
+      "step": 105
+    },
+    {
+      "epoch": 0.7,
+      "learning_rate": 3e-05,
+      "loss": 2.6053,
+      "step": 110
+    },
+    {
+      "epoch": 0.73,
+      "learning_rate": 3e-05,
+      "loss": 2.6772,
+      "step": 115
+    },
+    {
+      "epoch": 0.76,
+      "learning_rate": 3e-05,
+      "loss": 2.6096,
+      "step": 120
+    },
+    {
+      "epoch": 0.79,
+      "learning_rate": 3e-05,
+      "loss": 2.6703,
+      "step": 125
+    },
+    {
+      "epoch": 0.82,
+      "learning_rate": 3e-05,
+      "loss": 2.6278,
+      "step": 130
+    },
+    {
+      "epoch": 0.85,
+      "learning_rate": 3e-05,
+      "loss": 2.6324,
+      "step": 135
+    },
+    {
+      "epoch": 0.89,
+      "learning_rate": 3e-05,
+      "loss": 2.6098,
+      "step": 140
+    },
+    {
+      "epoch": 0.92,
+      "learning_rate": 3e-05,
+      "loss": 2.6498,
+      "step": 145
+    },
+    {
+      "epoch": 0.95,
+      "learning_rate": 3e-05,
+      "loss": 2.6319,
+      "step": 150
+    },
+    {
+      "epoch": 0.98,
+      "learning_rate": 3e-05,
+      "loss": 2.6507,
+      "step": 155
+    },
+    {
+      "epoch": 1.01,
+      "learning_rate": 3e-05,
+      "loss": 2.6601,
+      "step": 160
+    },
+    {
+      "epoch": 1.04,
+      "learning_rate": 3e-05,
+      "loss": 2.5753,
+      "step": 165
+    },
+    {
+      "epoch": 1.08,
+      "learning_rate": 3e-05,
+      "loss": 2.6221,
+      "step": 170
+    },
+    {
+      "epoch": 1.11,
+      "learning_rate": 3e-05,
+      "loss": 2.5597,
+      "step": 175
+    },
+    {
+      "epoch": 1.14,
+      "learning_rate": 3e-05,
+      "loss": 2.5934,
+      "step": 180
+    },
+    {
+      "epoch": 1.17,
+      "learning_rate": 3e-05,
+      "loss": 2.5633,
+      "step": 185
+    },
+    {
+      "epoch": 1.2,
+      "learning_rate": 3e-05,
+      "loss": 2.5715,
+      "step": 190
+    },
+    {
+      "epoch": 1.23,
+      "learning_rate": 3e-05,
+      "loss": 2.5809,
+      "step": 195
+    },
+    {
+      "epoch": 1.27,
+      "learning_rate": 3e-05,
+      "loss": 2.5561,
+      "step": 200
+    },
+    {
+      "epoch": 1.3,
+      "learning_rate": 3e-05,
+      "loss": 2.5633,
+      "step": 205
+    },
+    {
+      "epoch": 1.33,
+      "learning_rate": 3e-05,
+      "loss": 2.5526,
+      "step": 210
+    },
+    {
+      "epoch": 1.36,
+      "learning_rate": 3e-05,
+      "loss": 2.6323,
+      "step": 215
+    },
+    {
+      "epoch": 1.39,
+      "learning_rate": 3e-05,
+      "loss": 2.5679,
+      "step": 220
+    },
+    {
+      "epoch": 1.42,
+      "learning_rate": 3e-05,
+      "loss": 2.5805,
+      "step": 225
+    },
+    {
+      "epoch": 1.46,
+      "learning_rate": 3e-05,
+      "loss": 2.5833,
+      "step": 230
+    },
+    {
+      "epoch": 1.49,
+      "learning_rate": 3e-05,
+      "loss": 2.5859,
+      "step": 235
+    },
+    {
+      "epoch": 1.52,
+      "learning_rate": 3e-05,
+      "loss": 2.5947,
+      "step": 240
+    },
+    {
+      "epoch": 1.55,
+      "learning_rate": 3e-05,
+      "loss": 2.6159,
+      "step": 245
+    },
+    {
+      "epoch": 1.58,
+      "learning_rate": 3e-05,
+      "loss": 2.5969,
+      "step": 250
+    },
+    {
+      "epoch": 1.61,
+      "learning_rate": 3e-05,
+      "loss": 2.5649,
+      "step": 255
+    },
+    {
+      "epoch": 1.65,
+      "learning_rate": 3e-05,
+      "loss": 2.5304,
+      "step": 260
+    },
+    {
+      "epoch": 1.68,
+      "learning_rate": 3e-05,
+      "loss": 2.5628,
+      "step": 265
+    },
+    {
+      "epoch": 1.71,
+      "learning_rate": 3e-05,
+      "loss": 2.5857,
+      "step": 270
+    },
+    {
+      "epoch": 1.74,
+      "learning_rate": 3e-05,
+      "loss": 2.5914,
+      "step": 275
+    },
+    {
+      "epoch": 1.77,
+      "learning_rate": 3e-05,
+      "loss": 2.5572,
+      "step": 280
+    },
+    {
+      "epoch": 1.8,
+      "learning_rate": 3e-05,
+      "loss": 2.6037,
+      "step": 285
+    },
+    {
+      "epoch": 1.83,
+      "learning_rate": 3e-05,
+      "loss": 2.6095,
+      "step": 290
+    },
+    {
+      "epoch": 1.87,
+      "learning_rate": 3e-05,
+      "loss": 2.5787,
+      "step": 295
+    },
+    {
+      "epoch": 1.9,
+      "learning_rate": 3e-05,
+      "loss": 2.5881,
+      "step": 300
+    },
+    {
+      "epoch": 1.93,
+      "learning_rate": 3e-05,
+      "loss": 2.5706,
+      "step": 305
+    },
+    {
+      "epoch": 1.96,
+      "learning_rate": 3e-05,
+      "loss": 2.5636,
+      "step": 310
+    },
+    {
+      "epoch": 1.99,
+      "learning_rate": 3e-05,
+      "loss": 2.576,
+      "step": 315
+    },
+    {
+      "epoch": 2.03,
+      "learning_rate": 3e-05,
+      "loss": 2.5955,
+      "step": 320
+    },
+    {
+      "epoch": 2.06,
+      "learning_rate": 3e-05,
+      "loss": 2.5293,
+      "step": 325
+    },
+    {
+      "epoch": 2.09,
+      "learning_rate": 3e-05,
+      "loss": 2.5107,
+      "step": 330
+    },
+    {
+      "epoch": 2.12,
+      "learning_rate": 3e-05,
+      "loss": 2.5378,
+      "step": 335
+    },
+    {
+      "epoch": 2.15,
+      "learning_rate": 3e-05,
+      "loss": 2.5262,
+      "step": 340
+    },
+    {
+      "epoch": 2.18,
+      "learning_rate": 3e-05,
+      "loss": 2.5485,
+      "step": 345
+    },
+    {
+      "epoch": 2.22,
+      "learning_rate": 3e-05,
+      "loss": 2.5123,
+      "step": 350
+    },
+    {
+      "epoch": 2.25,
+      "learning_rate": 3e-05,
+      "loss": 2.509,
+      "step": 355
+    },
+    {
+      "epoch": 2.28,
+      "learning_rate": 3e-05,
+      "loss": 2.4993,
+      "step": 360
+    },
+    {
+      "epoch": 2.31,
+      "learning_rate": 3e-05,
+      "loss": 2.5133,
+      "step": 365
+    },
+    {
+      "epoch": 2.34,
+      "learning_rate": 3e-05,
+      "loss": 2.5346,
+      "step": 370
+    },
+    {
+      "epoch": 2.37,
+      "learning_rate": 3e-05,
+      "loss": 2.5174,
+      "step": 375
+    },
+    {
+      "epoch": 2.4,
+      "learning_rate": 3e-05,
+      "loss": 2.5282,
+      "step": 380
+    },
+    {
+      "epoch": 2.44,
+      "learning_rate": 3e-05,
+      "loss": 2.5489,
+      "step": 385
+    },
+    {
+      "epoch": 2.47,
+      "learning_rate": 3e-05,
+      "loss": 2.4989,
+      "step": 390
+    },
+    {
+      "epoch": 2.5,
+      "learning_rate": 3e-05,
+      "loss": 2.5243,
+      "step": 395
+    },
+    {
+      "epoch": 2.53,
+      "learning_rate": 3e-05,
+      "loss": 2.5228,
+      "step": 400
+    },
+    {
+      "epoch": 2.56,
+      "learning_rate": 3e-05,
+      "loss": 2.5013,
+      "step": 405
+    },
+    {
+      "epoch": 2.59,
+      "learning_rate": 3e-05,
+      "loss": 2.509,
+      "step": 410
+    },
+    {
+      "epoch": 2.63,
+      "learning_rate": 3e-05,
+      "loss": 2.566,
+      "step": 415
+    },
+    {
+      "epoch": 2.66,
+      "learning_rate": 3e-05,
+      "loss": 2.5291,
+      "step": 420
+    },
+    {
+      "epoch": 2.69,
+      "learning_rate": 3e-05,
+      "loss": 2.5421,
+      "step": 425
+    },
+    {
+      "epoch": 2.72,
+      "learning_rate": 3e-05,
+      "loss": 2.5454,
+      "step": 430
+    },
+    {
+      "epoch": 2.75,
+      "learning_rate": 3e-05,
+      "loss": 2.5324,
+      "step": 435
+    },
+    {
+      "epoch": 2.78,
+      "learning_rate": 3e-05,
+      "loss": 2.5457,
+      "step": 440
+    },
+    {
+      "epoch": 2.82,
+      "learning_rate": 3e-05,
+      "loss": 2.5365,
+      "step": 445
+    },
+    {
+      "epoch": 2.85,
+      "learning_rate": 3e-05,
+      "loss": 2.5395,
+      "step": 450
+    },
+    {
+      "epoch": 2.88,
+      "learning_rate": 3e-05,
+      "loss": 2.5458,
+      "step": 455
+    },
+    {
+      "epoch": 2.91,
+      "learning_rate": 3e-05,
+      "loss": 2.4935,
+      "step": 460
+    },
+    {
+      "epoch": 2.94,
+      "learning_rate": 3e-05,
+      "loss": 2.5347,
+      "step": 465
+    },
+    {
+      "epoch": 2.97,
+      "learning_rate": 3e-05,
+      "loss": 2.5287,
+      "step": 470
+    },
+    {
+      "epoch": 3.01,
+      "learning_rate": 3e-05,
+      "loss": 2.5784,
+      "step": 475
+    },
+    {
+      "epoch": 3.04,
+      "learning_rate": 3e-05,
+      "loss": 2.4889,
+      "step": 480
+    },
+    {
+      "epoch": 3.07,
+      "learning_rate": 3e-05,
+      "loss": 2.4622,
+      "step": 485
+    },
+    {
+      "epoch": 3.1,
+      "learning_rate": 3e-05,
+      "loss": 2.4628,
+      "step": 490
+    },
+    {
+      "epoch": 3.13,
+      "learning_rate": 3e-05,
+      "loss": 2.5204,
+      "step": 495
+    },
+    {
+      "epoch": 3.16,
+      "learning_rate": 3e-05,
+      "loss": 2.4543,
+      "step": 500
+    },
+    {
+      "epoch": 3.2,
+      "learning_rate": 3e-05,
+      "loss": 2.4717,
+      "step": 505
+    },
+    {
+      "epoch": 3.23,
+      "learning_rate": 3e-05,
+      "loss": 2.5113,
+      "step": 510
+    },
+    {
+      "epoch": 3.26,
+      "learning_rate": 3e-05,
+      "loss": 2.4838,
+      "step": 515
+    },
+    {
+      "epoch": 3.29,
+      "learning_rate": 3e-05,
+      "loss": 2.5224,
+      "step": 520
+    },
+    {
+      "epoch": 3.32,
+      "learning_rate": 3e-05,
+      "loss": 2.4858,
+      "step": 525
+    },
+    {
+      "epoch": 3.35,
+      "learning_rate": 3e-05,
+      "loss": 2.5227,
+      "step": 530
+    },
+    {
+      "epoch": 3.39,
+      "learning_rate": 3e-05,
+      "loss": 2.4824,
+      "step": 535
+    },
+    {
+      "epoch": 3.42,
+      "learning_rate": 3e-05,
+      "loss": 2.4836,
+      "step": 540
+    },
+    {
+      "epoch": 3.45,
+      "learning_rate": 3e-05,
+      "loss": 2.4968,
+      "step": 545
+    },
+    {
+      "epoch": 3.48,
+      "learning_rate": 3e-05,
+      "loss": 2.5004,
+      "step": 550
+    },
+    {
+      "epoch": 3.51,
+      "learning_rate": 3e-05,
+      "loss": 2.4756,
+      "step": 555
+    },
+    {
+      "epoch": 3.54,
+      "learning_rate": 3e-05,
+      "loss": 2.5068,
+      "step": 560
+    },
+    {
+      "epoch": 3.58,
+      "learning_rate": 3e-05,
+      "loss": 2.468,
+      "step": 565
+    },
+    {
+      "epoch": 3.61,
+      "learning_rate": 3e-05,
+      "loss": 2.4764,
+      "step": 570
+    },
+    {
+      "epoch": 3.64,
+      "learning_rate": 3e-05,
+      "loss": 2.4927,
+      "step": 575
+    },
+    {
+      "epoch": 3.67,
+      "learning_rate": 3e-05,
+      "loss": 2.4993,
+      "step": 580
+    },
+    {
+      "epoch": 3.7,
+      "learning_rate": 3e-05,
+      "loss": 2.4786,
+      "step": 585
+    },
+    {
+      "epoch": 3.73,
+      "learning_rate": 3e-05,
+      "loss": 2.489,
+      "step": 590
+    },
+    {
+      "epoch": 3.77,
+      "learning_rate": 3e-05,
+      "loss": 2.4271,
+      "step": 595
+    },
+    {
+      "epoch": 3.8,
+      "learning_rate": 3e-05,
+      "loss": 2.4502,
+      "step": 600
+    },
+    {
+      "epoch": 3.83,
+      "learning_rate": 3e-05,
+      "loss": 2.4682,
+      "step": 605
+    },
+    {
+      "epoch": 3.86,
+      "learning_rate": 3e-05,
+      "loss": 2.4702,
+      "step": 610
+    },
+    {
+      "epoch": 3.89,
+      "learning_rate": 3e-05,
+      "loss": 2.4722,
+      "step": 615
+    },
+    {
+      "epoch": 3.92,
+      "learning_rate": 3e-05,
+      "loss": 2.5132,
+      "step": 620
+    },
+    {
+      "epoch": 3.96,
+      "learning_rate": 3e-05,
+      "loss": 2.446,
+      "step": 625
+    },
+    {
+      "epoch": 3.99,
+      "learning_rate": 3e-05,
+      "loss": 2.4824,
+      "step": 630
+    },
+    {
+      "epoch": 4.0,
+      "step": 632,
+      "total_flos": 2.1853297764099686e+17,
+      "train_loss": 2.5587786006022104,
+      "train_runtime": 54071.5154,
+      "train_samples_per_second": 0.748,
+      "train_steps_per_second": 0.012
+    }
+  ],
+  "max_steps": 632,
+  "num_train_epochs": 4,
+  "total_flos": 2.1853297764099686e+17,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b45e019c7ec05993cda59a7c7816ca58ae002e1bb6428693619fe1f07541b7e
+size 4399