| --- |
| library_name: setfit |
| tags: |
| - setfit |
| - absa |
| - sentence-transformers |
| - text-classification |
| - generated_from_setfit_trainer |
| metrics: |
| - accuracy |
| widget: |
| - text: louder and the mouse didnt break:I wish the volume could be louder and the |
| mouse didnt break after only a month. |
| - text: + + (sales, service,:BEST BUY - 5 STARS + + + (sales, service, respect for |
| old men who aren't familiar with the technology) DELL COMPUTERS - 3 stars DELL |
| SUPPORT - owes a me a couple |
| - text: back and my built-in webcam and built-:I got it back and my built-in webcam |
| and built-in mic were shorting out anytime I touched the lid, (mind you this was |
| my means of communication with my fiance who was deployed) but I suffered thru |
| it and would constandly have to reset the computer to be able to use my cam and |
| mic anytime they went out. |
| - text: after i install Mozzilla firfox i love every:the only fact i dont like about |
| apples is they generally use safari and i dont use safari but after i install |
| Mozzilla firfox i love every single bit about it. |
| - text: in webcam and built-in mic were shorting out:I got it back and my built-in |
| webcam and built-in mic were shorting out anytime I touched the lid, (mind you |
| this was my means of communication with my fiance who was deployed) but I suffered |
| thru it and would constandly have to reset the computer to be able to use my cam |
| and mic anytime they went out. |
| pipeline_tag: text-classification |
| inference: false |
| base_model: sentence-transformers/all-mpnet-base-v2 |
| model-index: |
| - name: SetFit Polarity Model with sentence-transformers/all-mpnet-base-v2 |
| results: |
| - task: |
| type: text-classification |
| name: Text Classification |
| dataset: |
| name: tomaarsen/setfit-absa-semeval-laptops |
| type: unknown |
| split: test |
| metrics: |
| - type: accuracy |
| value: 0.7007874015748031 |
| name: Accuracy |
| --- |
| |
| # SetFit Polarity Model with sentence-transformers/all-mpnet-base-v2 |
|
|
| This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Aspect Based Sentiment Analysis (ABSA). This SetFit model uses [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. In particular, this model is in charge of classifying aspect polarities. |
|
|
| The model has been trained using an efficient few-shot learning technique that involves: |
|
|
| 1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. |
| 2. Training a classification head with features from the fine-tuned Sentence Transformer. |
|
|
| This model was trained within the context of a larger system for ABSA, which looks like so: |
|
|
| 1. Use a spaCy model to select possible aspect span candidates. |
| 2. Use a SetFit model to filter these possible aspect span candidates. |
| 3. **Use this SetFit model to classify the filtered aspect span candidates.** |
|
|
| ## Model Details |
|
|
| ### Model Description |
| - **Model Type:** SetFit |
| - **Sentence Transformer body:** [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) |
| - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance |
| - **spaCy Model:** en_core_web_sm |
| - **SetFitABSA Aspect Model:** [joshuasundance/setfit-absa-all-MiniLM-L6-v2-laptops-aspect](https://huggingface.co/joshuasundance/setfit-absa-all-MiniLM-L6-v2-laptops-aspect) |
| - **SetFitABSA Polarity Model:** [joshuasundance/setfit-absa-all-mpnet-base-v2-laptops-polarity](https://huggingface.co/joshuasundance/setfit-absa-all-mpnet-base-v2-laptops-polarity) |
| - **Maximum Sequence Length:** 384 tokens |
| - **Number of Classes:** 4 classes |
| <!-- - **Training Dataset:** [tomaarsen/setfit-absa-semeval-laptops](https://huggingface.co/datasets/tomaarsen/setfit-absa-semeval-laptops) --> |
| <!-- - **Language:** Unknown --> |
| <!-- - **License:** Unknown --> |
| |
| ### Model Sources |
| |
| - **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) |
| - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) |
| - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) |
| |
| ### Model Labels |
| | Label | Examples | |
| |:---------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | neutral | <ul><li>'skip taking the cord with me because:I charge it at night and skip taking the cord with me because of the good battery life.'</li><li>'The tech guy then said the:The tech guy then said the service center does not do 1-to-1 exchange and I have to direct my concern to the "sales" team, which is the retail shop which I bought my netbook from.'</li><li>'all dark, power light steady, hard:\xa0One night I turned the freaking thing off after using it, the next day I turn it on, no GUI, screen all dark, power light steady, hard drive light steady and not flashing as it usually does.'</li></ul> | |
| | positive | <ul><li>'of the good battery life.:I charge it at night and skip taking the cord with me because of the good battery life.'</li><li>'is of high quality, has a:it is of high quality, has a killer GUI, is extremely stable, is highly expandable, is bundled with lots of very good applications, is easy to use, and is absolutely gorgeous.'</li><li>'has a killer GUI, is extremely:it is of high quality, has a killer GUI, is extremely stable, is highly expandable, is bundled with lots of very good applications, is easy to use, and is absolutely gorgeous.'</li></ul> | |
| | negative | <ul><li>'then said the service center does not do:The tech guy then said the service center does not do 1-to-1 exchange and I have to direct my concern to the "sales" team, which is the retail shop which I bought my netbook from.'</li><li>'concern to the "sales" team, which is:The tech guy then said the service center does not do 1-to-1 exchange and I have to direct my concern to the "sales" team, which is the retail shop which I bought my netbook from.'</li><li>'on, no GUI, screen all:\xa0One night I turned the freaking thing off after using it, the next day I turn it on, no GUI, screen all dark, power light steady, hard drive light steady and not flashing as it usually does.'</li></ul> | |
| | conflict | <ul><li>'-No backlit keyboard, but not:-No backlit keyboard, but not an issue for me.'</li><li>"to replace the battery once, but:I did have to replace the battery once, but that was only a couple months ago and it's been working perfect ever since."</li></ul> | |
| |
| ## Evaluation |
| |
| ### Metrics |
| | Label | Accuracy | |
| |:--------|:---------| |
| | **all** | 0.7008 | |
| |
| ## Uses |
| |
| ### Direct Use for Inference |
| |
| First install the SetFit library: |
| |
| ```bash |
| pip install setfit |
| ``` |
| |
| Then you can load this model and run inference. |
| |
| ```python |
| from setfit import AbsaModel |
| |
| # Download from the 🤗 Hub |
| model = AbsaModel.from_pretrained( |
| "joshuasundance/setfit-absa-all-MiniLM-L6-v2-laptops-aspect", |
| "joshuasundance/setfit-absa-all-mpnet-base-v2-laptops-polarity", |
| spacy_model="en_core_web_sm", |
| ) |
| # Run inference |
| preds = model("This laptop meets every expectation and Windows 7 is great!") |
| ``` |
| |
| <!-- |
| ### Downstream Use |
|
|
| *List how someone could finetune this model on their own dataset.* |
| --> |
|
|
| <!-- |
| ### Out-of-Scope Use |
|
|
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| --> |
|
|
| <!-- |
| ## Bias, Risks and Limitations |
|
|
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| --> |
|
|
| <!-- |
| ### Recommendations |
|
|
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| --> |
|
|
| ## Training Details |
|
|
| ### Training Set Metrics |
| | Training set | Min | Median | Max | |
| |:-------------|:----|:--------|:----| |
| | Word count | 3 | 25.5873 | 48 | |
|
|
| | Label | Training Sample Count | |
| |:---------|:----------------------| |
| | conflict | 2 | |
| | negative | 45 | |
| | neutral | 30 | |
| | positive | 49 | |
|
|
| ### Training Hyperparameters |
| - batch_size: (128, 128) |
| - num_epochs: (5, 5) |
| - max_steps: -1 |
| - sampling_strategy: oversampling |
| - body_learning_rate: (2e-05, 1e-05) |
| - head_learning_rate: 0.01 |
| - loss: CosineSimilarityLoss |
| - distance_metric: cosine_distance |
| - margin: 0.25 |
| - end_to_end: False |
| - use_amp: True |
| - warmup_proportion: 0.1 |
| - seed: 42 |
| - eval_max_steps: -1 |
| - load_best_model_at_end: True |
|
|
| ### Training Results |
| | Epoch | Step | Training Loss | Validation Loss | |
| |:----------:|:------:|:-------------:|:---------------:| |
| | 0.0120 | 1 | 0.2721 | - | |
| | **0.6024** | **50** | **0.0894** | **0.2059** | |
| | 1.2048 | 100 | 0.0014 | 0.2309 | |
| | 1.8072 | 150 | 0.0006 | 0.2359 | |
| | 2.4096 | 200 | 0.0005 | 0.2373 | |
| | 3.0120 | 250 | 0.0004 | 0.2364 | |
| | 3.6145 | 300 | 0.0003 | 0.2371 | |
|
|
| * The bold row denotes the saved checkpoint. |
| ### Framework Versions |
| - Python: 3.11.7 |
| - SetFit: 1.0.3 |
| - Sentence Transformers: 2.3.0 |
| - spaCy: 3.7.2 |
| - Transformers: 4.37.2 |
| - PyTorch: 2.1.2+cu118 |
| - Datasets: 2.16.1 |
| - Tokenizers: 0.15.1 |
|
|
| ## Citation |
|
|
| ### BibTeX |
| ```bibtex |
| @article{https://doi.org/10.48550/arxiv.2209.11055, |
| doi = {10.48550/ARXIV.2209.11055}, |
| url = {https://arxiv.org/abs/2209.11055}, |
| author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, |
| keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, |
| title = {Efficient Few-Shot Learning Without Prompts}, |
| publisher = {arXiv}, |
| year = {2022}, |
| copyright = {Creative Commons Attribution 4.0 International} |
| } |
| ``` |
|
|
| <!-- |
| ## Glossary |
|
|
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
|
|
| <!-- |
| ## Model Card Authors |
|
|
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
|
|
| <!-- |
| ## Model Card Contact |
|
|
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
