rlucatoor commited on
Commit
4d31fe7
·
1 Parent(s): 3c24069

update README.md and notebook.ipynb

Browse files
Files changed (2) hide show
  1. README.md +8 -96
  2. notebook.ipynb +8 -19
README.md CHANGED
@@ -29,9 +29,6 @@ task:
29
 
30
  # tanaos-text-anonymizer-v1: A small but performant Text Anonymization model
31
 
32
- > [!TIP]
33
- > Looking for a custom Text Anonymization model, or any other task-specific Small Language Model fine-tuned to your specific needs? We will do it for you! https://tanaos.com/#try-it-out
34
-
35
  This model was created by Tanaos with the [Artifex Python library](https://github.com/tanaos/artifex).
36
 
37
  This is a **Named Entity Recognition model** based on [tanaos/tanaos-NER-v1](https://huggingface.co/tanaos/tanaos-NER-v1) and fine-tuned on [a synthetic dataset](https://huggingface.co/datasets/tanaos/synthetic-text-anonymizer-dataset-v1) to recognize Personal Identifiable Information (PII) entities in text. Once identified, the entities are redacted to ensure privacy and confidentiality, before sharing or processing text data.
@@ -48,112 +45,27 @@ While the base NER model was trained to recognize 14 named entity categories, th
48
 
49
  ## How to Use
50
 
51
- <!-- ### Via the Tanaos API -->
52
-
53
- Use this model for free via the [Tanaos API](https://tanaos.com/) in 3 simple steps:
54
-
55
- 1. Sign up for a free account at [https://platform.tanaos.com/](https://platform.tanaos.com/)
56
- 2. Create a free API Key from the [API Keys section](https://platform.tanaos.com/profile/api-keys)
57
- 3. Replace `<YOUR_API_KEY>` in the code below with your API Key and use this snippet:
58
-
59
- ```python
60
- import requests
61
-
62
- session = requests.Session()
63
-
64
- ta_out = session.post(
65
- "https://slm.tanaos.com/models/text-anonymization",
66
- headers={
67
- "X-API-Key": "<YOUR_API_KEY>",
68
- },
69
- json={
70
- "text": "John Doe lives at 123 Main St, New York. His phone number is (555) 123-4567.",
71
- "include_mask_type": True,
72
- "include_mask_counter": True
73
- }
74
- )
75
-
76
- print(ta_out.json()["data"])
77
- # >>> ['[MASKED_PERSON_3] lives at [MASKED_ADDRESS_2] [MASKED_LOCATION_1] His phone number is [MASKED_PHONE_NUMBER_0]']
78
- ```
79
-
80
- <!-- ### Via the Artifex library (`pip install artifex`)
81
-
82
- ```python
83
- from artifex import Artifex
84
-
85
- ta = Artifex().text_anonymization
86
-
87
- print(ta("John Doe lives at 123 Main St, New York. His phone number is (555) 123-4567."))
88
- # >>> ["[MASKED] lives at [MASKED]. His phone number is [MASKED]."]
89
- ```
90
-
91
- ### Via the Transformers library
92
-
93
- Using this model through the `transformers` library, Personal Identifiable Information (PII) are only identified, but not automatically redacted; you will have to implement your own redaction logic.
94
-
95
- ```python
96
- from transformers import pipeline
97
-
98
- ta = pipeline(
99
- task="token-classification",
100
- model="tanaos/tanaos-text-anonymizer-v1",
101
- aggregation_strategy="first"
102
- )
103
-
104
- print(ta("John Doe lives at 123 Main St, New York. His phone number is (555) 123-4567."))
105
- # >>> [{'entity_group': 'PERSON', 'score': 0.90219176, 'word': 'John Doe', 'start': 0, 'end': 8}, {'entity_group': 'ADDRESS', 'score': 0.9522348, 'word': ' 123 Main St,', 'start': 18, 'end': 30}, {'entity_group': 'LOCATION', 'score': 0.97109795, 'word': ' New York.', 'start': 31, 'end': 40}, {'entity_group': 'PHONE_NUMBER', 'score': 0.9054972, 'word': ' (555) 123-4567.', 'start': 61, 'end': 76}]
106
- ```
107
-
108
- ## How to fine-tune (without training data)
109
 
110
- Use the [Artifex library](https://github.com/tanaos/artifex) to fine-tune the model to any language other than English or to custom domains by generating synthetic training data on-the-fly. Install Artifex with
111
 
112
  ```bash
113
  pip install artifex
114
  ```
115
 
116
- ### Fine-tune to any language
117
 
118
  ```python
119
  from artifex import Artifex
120
 
121
- ta = Artifex().text_anonymization
122
 
123
- model_output_path = "./output_model/"
 
124
 
125
- ta.train(
126
- domain="documentos medicos en Español",
127
- language="spanish",
128
- output_path=model_output_path
129
- )
130
-
131
- ta.load(model_output_path)
132
- print(ta("El paciente John Doe visitó Nueva York el 12 de marzo de 2023 a las 10:30 a. m."))
133
-
134
- # >>> ["El paciente [MASKED] visitó [MASKED] el [MASKED] a las [MASKED]."]
135
  ```
136
 
137
- ### Fine-tune to custom domains
138
-
139
- ```python
140
- from artifex import Artifex
141
-
142
- text_anonym = Artifex().text_anonymization
143
-
144
- model_output_path = "./output_model/"
145
-
146
- text_anonym.train(
147
- domain="legal documents",
148
- output_path=model_output_path
149
- )
150
-
151
- ta.load(model_output_path)
152
- print(ta("John Doe signed the contract on 12/03/2023 at 123 Main St, New York."))
153
-
154
- # >>> ["[MASKED] signed the contract on [MASKED] at [MASKED]."]
155
- ``` -->
156
-
157
  ## Model Description
158
 
159
  - **Base model:** `FacebookAI/roberta-base`
@@ -174,7 +86,7 @@ by providing the following instructions and generating 10,000 synthetic training
174
  ```python
175
  from artifex import Artifex
176
 
177
- ta = Artifex().text_anonymization
178
 
179
  ta.train(
180
  domain="general",
 
29
 
30
  # tanaos-text-anonymizer-v1: A small but performant Text Anonymization model
31
 
 
 
 
32
  This model was created by Tanaos with the [Artifex Python library](https://github.com/tanaos/artifex).
33
 
34
  This is a **Named Entity Recognition model** based on [tanaos/tanaos-NER-v1](https://huggingface.co/tanaos/tanaos-NER-v1) and fine-tuned on [a synthetic dataset](https://huggingface.co/datasets/tanaos/synthetic-text-anonymizer-dataset-v1) to recognize Personal Identifiable Information (PII) entities in text. Once identified, the entities are redacted to ensure privacy and confidentiality, before sharing or processing text data.
 
45
 
46
  ## How to Use
47
 
48
+ Use this model through the [Artifex library](https://github.com/tanaos/artifex):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
 
50
+ install Artifex with
51
 
52
  ```bash
53
  pip install artifex
54
  ```
55
 
56
+ use the model with
57
 
58
  ```python
59
  from artifex import Artifex
60
 
61
+ ta = Artifex().text_anonymization()
62
 
63
+ anonymized_text = ta("John Doe lives at 123 Main St, New York. His phone number is (555) 123-4567.")
64
+ print(anonymized_text)
65
 
66
+ # >>> ["[MASKED] lives at [MASKED]. His phone number is [MASKED]."]
 
 
 
 
 
 
 
 
 
67
  ```
68
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  ## Model Description
70
 
71
  - **Base model:** `FacebookAI/roberta-base`
 
86
  ```python
87
  from artifex import Artifex
88
 
89
+ ta = Artifex().text_anonymization()
90
 
91
  ta.train(
92
  domain="general",
notebook.ipynb CHANGED
@@ -13,11 +13,11 @@
13
  "id": "e94691be",
14
  "metadata": {},
15
  "source": [
16
- "Use this model for free via the [Tanaos API](https://tanaos.com/) in 3 simple steps:\n",
17
  "\n",
18
- "1. Sign up for a free account at [https://platform.tanaos.com/](https://platform.tanaos.com/)\n",
19
- "2. Create a free API Key from the [API Keys section](https://platform.tanaos.com/profile/api-keys)\n",
20
- "3. Replace `<YOUR_API_KEY>` in the code below with your API Key and use this snippet:"
21
  ]
22
  },
23
  {
@@ -31,23 +31,12 @@
31
  },
32
  "outputs": [],
33
  "source": [
34
- "import requests\n",
35
  "\n",
36
- "session = requests.Session()\n",
37
  "\n",
38
- "ta_out = session.post(\n",
39
- " \"https://slm.tanaos.com/models/text-anonymization\",\n",
40
- " headers={\n",
41
- " \"X-API-Key\": \"<YOUR_API_KEY>\",\n",
42
- " },\n",
43
- " json={\n",
44
- " \"text\": \"John Doe lives at 123 Main St, New York. His phone number is (555) 123-4567.\",\n",
45
- " \"include_mask_type\": True,\n",
46
- " \"include_mask_counter\": True\n",
47
- " }\n",
48
- ")\n",
49
- "\n",
50
- "print(ta_out.json()[\"data\"])"
51
  ]
52
  }
53
  ],
 
13
  "id": "e94691be",
14
  "metadata": {},
15
  "source": [
16
+ "Use the [Artifex library](https://github.com/tanaos/artifex). Install it with\n",
17
  "\n",
18
+ "```bash\n",
19
+ "pip install artifex\n",
20
+ "```"
21
  ]
22
  },
23
  {
 
31
  },
32
  "outputs": [],
33
  "source": [
34
+ "from artifex import Artifex\n",
35
  "\n",
36
+ "ta = Artifex().text_anonymization()\n",
37
  "\n",
38
+ "anonymized_text = ta(\"John Doe lives at 123 Main St, New York. His phone number is (555) 123-4567.\")\n",
39
+ "print(anonymized_text)"
 
 
 
 
 
 
 
 
 
 
 
40
  ]
41
  }
42
  ],