Update README.md

da223bb verified 8 months ago

12.3 kB

	---
	language:
	- id
	- en
	library_name: custom
	license: mit
	tags:
	- indonesian
	- skill-extraction
	- ner
	- job-market
	- nlp
	- rule-based
	- skills-taxonomy
	- indonesian-nlp
	pipeline_tag: token-classification
	---

	# Indonesian Skill Extractor v1.0

	A production-ready, rule-based NER system for extracting and categorizing technical and soft skills from Indonesian job postings.

	## 🎯 Model Description

	This model specializes in identifying and categorizing skills from Indonesian job market texts. It uses a comprehensive skill taxonomy with 200+ predefined skills across 7 categories, combined with intelligent pattern matching and normalization.

	### Key Features

	- ✅ Zero Dependencies: Pure Python, no ML frameworks required
	- ✅ 200+ Skills: Comprehensive taxonomy across 7 categories
	- ✅ Bilingual: Handles English and Indonesian (including code-switching)
	- ✅ Skill Normalization: Maps aliases to canonical forms (js→javascript, etc.)
	- ✅ Proficiency Detection: Identifies beginner/intermediate/expert levels
	- ✅ Fast & Deterministic: 1000+ docs/sec, reproducible results
	- ✅ Production Ready: Lightweight (~20 KB), easy integration

	### Skill Categories

	\| Category \| Count \| Examples \|
	\|----------\|-------\|----------\|
	\| programming \| 30+ \| Python, Java, JavaScript, TypeScript, PHP, C++, Go, Rust \|
	\| frontend \| 40+ \| React, Vue, Angular, Next.js, HTML, CSS, Tailwind, Webpack \|
	\| backend \| 30+ \| Node.js, Django, Laravel, Spring Boot, Express, FastAPI \|
	\| database \| 25+ \| MySQL, PostgreSQL, MongoDB, Redis, Elasticsearch, Oracle \|
	\| cloud \| 35+ \| AWS, Azure, GCP, Docker, Kubernetes, Jenkins, Terraform \|
	\| data_science \| 30+ \| Pandas, TensorFlow, PyTorch, Tableau, Power BI, Spark \|
	\| soft_skills \| 20+ \| Communication, Leadership, Teamwork, Problem Solving \|

	Total: 200+ skills with 40+ aliases and variations

	## 🚀 Quick Start

	### Installation

	No installation required! Just download the single Python file:

	```python
	# Download skill_extractor.py from this repository
	# Place it in your project directory
	from skill_extractor import IndonesianSkillExtractor

	# Or use convenience function
	from skill_extractor import extract_skills
	```

	### Basic Usage

	```python
	from skill_extractor import IndonesianSkillExtractor

	# Initialize
	extractor = IndonesianSkillExtractor()

	# Extract skills from text
	text = "Menguasai Python, React, MySQL, dan komunikasi yang baik"
	result = extractor.extract(text)

	print(result)
	# Output:
	# {
	# 'skills': [
	# {'original': 'Python', 'normalized': 'python', 'category': 'programming', 'proficiency': None},
	# {'original': 'React', 'normalized': 'react', 'category': 'frontend', 'proficiency': None},
	# {'original': 'MySQL', 'normalized': 'mysql', 'category': 'database', 'proficiency': None},
	# {'original': 'komunikasi', 'normalized': 'komunikasi', 'category': 'soft_skills', 'proficiency': None}
	# ],
	# 'total_count': 4,
	# 'unique_count': 4,
	# 'by_category': {
	# 'programming': [...],
	# 'frontend': [...],
	# 'database': [...],
	# 'soft_skills': [...]
	# }
	# }
	```

	### Simple Extraction

	```python
	from skill_extractor import extract_skills

	# Quick extraction (returns list of skill names)
	skills = extract_skills("Python, React, MySQL, AWS")
	print(skills)
	# Output: ['python', 'react', 'mysql', 'aws']
	```

	### Batch Processing

	```python
	extractor = IndonesianSkillExtractor()

	texts = [
	"Python, Django, PostgreSQL",
	"React, TypeScript, Node.js",
	"AWS, Docker, Kubernetes"
	]

	results = extractor.batch_extract(texts)

	for i, result in enumerate(results):
	print(f"Text {i+1}: {result['total_count']} skills, {len(result['by_category'])} categories")
	```

	### Get Top Skills

	```python
	extractor = IndonesianSkillExtractor()

	job_descriptions = [
	"Python, Django, React...",
	"Java, Spring, MySQL...",
	"Python, FastAPI, PostgreSQL..."
	]

	top_skills = extractor.get_top_skills(job_descriptions, top_n=5)
	print(top_skills)
	# Output: [('python', 2), ('react', 1), ('django', 1), ...]
	```

	## 📊 Features

	### 1. Skill Normalization

	Handles variations and aliases:

	```python
	extractor = IndonesianSkillExtractor()

	# These all normalize to the same skill
	texts = ["JS", "js", "JavaScript", "javascript"]
	for text in texts:
	skills = extract_skills(text)
	print(skills) # All output: ['javascript']
	```

	40+ Aliases Supported:
	- js → javascript
	- ts → typescript
	- py → python
	- reactjs, react.js → react
	- nodejs → node.js
	- pg, postgres → postgresql
	- mongo → mongodb
	- k8s → kubernetes

	### 2. Proficiency Detection

	Extracts skill levels from text:

	```python
	text = "Expert in Python, Advanced React, Basic MySQL"
	result = extractor.extract(text)

	for skill in result['skills']:
	print(f"{skill['normalized']}: {skill['proficiency']}")

	# Output:
	# python: expert
	# react: expert (advanced maps to expert)
	# mysql: beginner (basic maps to beginner)
	```

	Proficiency Keywords:
	- Expert: expert, advanced, mahir, ahli, mastery
	- Intermediate: intermediate, menengah, competent
	- Beginner: beginner, basic, pemula, dasar

	### 3. Indonesian Language Support

	Handles Indonesian skill names and code-switching:

	```python
	text = "Komunikasi yang baik, kerja sama tim, kepemimpinan, Python"
	result = extractor.extract(text)

	for skill in result['skills']:
	print(f"{skill['original']} → {skill['category']}")

	# Output:
	# Komunikasi → soft_skills
	# kerja sama tim → soft_skills
	# kepemimpinan → soft_skills (leadership)
	# Python → programming
	```

	### 4. Comprehensive Parsing

	Handles multiple formats:

	```python
	# Comma-separated
	extract_skills("Python, React, MySQL")

	# Semicolon-separated
	extract_skills("Python; React; MySQL")

	# Bullet points
	extract_skills("• Python • React • MySQL")

	# Newline-separated
	extract_skills("Python\nReact\nMySQL")

	# Mixed with proficiency
	extract_skills("Python (Expert), React (2 years), MySQL")
	```

	## 📈 Performance

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Speed \| 1000+ docs/second \|
	\| Model Size \| ~20 KB (pure Python) \|
	\| Dependencies \| None (stdlib only) \|
	\| Skills Covered \| 200+ \|
	\| Categories \| 7 \|
	\| Aliases \| 40+ \|
	\| Languages \| Indonesian + English \|

	### Comparison with ML Models

	\| Feature \| Skill Extractor \| BERT-based NER \|
	\|---------\|----------------\|----------------\|
	\| Training Data \| Not required \| Required (1000+ samples) \|
	\| Model Size \| 20 KB \| 300+ MB \|
	\| Speed \| 1000+ docs/sec \| 50 docs/sec \|
	\| Deterministic \| ✅ Yes \| ❌ No \|
	\| Explainable \| ✅ Yes \| ❌ No \|
	\| Easy to Update \| ✅ Just edit dict \| ❌ Requires retraining \|

	## 🎯 Use Cases

	### 1. Job-Candidate Matching

	```python
	# Extract skills from job posting
	job_skills = extract_skills(job_description)

	# Extract skills from resume
	candidate_skills = extract_skills(resume_text)

	# Calculate match percentage
	matching_skills = set(job_skills) & set(candidate_skills)
	match_score = len(matching_skills) / len(job_skills) * 100
	```

	### 2. Skills Gap Analysis

	```python
	# Get market demand
	market_skills = extractor.get_top_skills(job_postings, top_n=20)

	# Get candidate pool skills
	candidate_skills = extractor.get_top_skills(resumes, top_n=20)

	# Find gaps
	in_demand = set(s[0] for s in market_skills)
	available = set(s[0] for s in candidate_skills)
	skill_gaps = in_demand - available
	```

	### 3. Trend Analysis

	```python
	from collections import Counter

	# Group by time period
	skills_by_month = {}
	for job in jobs:
	month = job['month']
	skills = extract_skills(job['requirements'])

	if month not in skills_by_month:
	skills_by_month[month] = []
	skills_by_month[month].extend(skills)

	# Analyze trends
	for month, skills in skills_by_month.items():
	top_5 = Counter(skills).most_common(5)
	print(f"{month}: {top_5}")
	```

	### 4. Resume Screening

	```python
	required_skills = ['python', 'django', 'postgresql']
	nice_to_have = ['react', 'docker', 'aws']

	def score_resume(resume_text):
	candidate_skills = set(extract_skills(resume_text))

	# Required skills (2 points each)
	required_score = len(candidate_skills & set(required_skills)) * 2

	# Nice to have (1 point each)
	bonus_score = len(candidate_skills & set(nice_to_have)) * 1

	return required_score + bonus_score

	# Rank candidates
	candidates = [...]
	ranked = sorted(candidates, key=lambda c: score_resume(c['resume']), reverse=True)
	```

	## 🔧 API Reference

	### `IndonesianSkillExtractor`

	Main class for skill extraction.

	#### Methods:

	`extract(text: str) -> Dict`
	- Full extraction with metadata
	- Returns: skills, counts, categories, proficiency

	`extract_simple(text: str) -> List[str]`
	- Simple extraction returning skill names
	- Returns: List of normalized skill strings

	`batch_extract(texts: List[str]) -> List[Dict]`
	- Process multiple texts
	- Returns: List of extraction results

	`get_top_skills(texts: List[str], top_n: int) -> List[Tuple]`
	- Get most frequent skills across texts
	- Returns: List of (skill, count) tuples

	`get_stats() -> Dict`
	- Get model statistics
	- Returns: version, total_skills, categories, etc.

	### Convenience Functions

	`extract_skills(text: str) -> List[str]`
	- Quick one-line extraction
	- Creates extractor instance automatically

	## 📄 License

	This model is released under the MIT License.

	Citation:
	```bibtex
	@software{indonesian_skill_extractor_2024,
	author = {Herlambang Haryo Putro},
	title = {Indonesian Skill Extractor v1.0},
	year = {2024},
	publisher = {Hugging Face},
	url = {https://huggingface.co/herlambangharyoputro/indonesian-skill-extractor-v1}
	}
	```

	## 🤝 Contributions

	Part of the Job Market Intelligence Platform project.

	Repository: [GitHub - job-market-intelligence-platform](https://github.com/herlambangharyoputro/job-market-intelligence-platform)

	Related Datasets:
	- [Indonesian Job Market - Raw 2024](https://huggingface.co/datasets/herlambangharyoputro/indonesian-job-market-raw-2024)
	- [Indonesian Job Market - Tokenized 2024](https://huggingface.co/datasets/herlambangharyoputro/indonesian-job-market-tokenized-2024)

	Contributions welcome! If you:
	- Find missing skills or categories
	- Have suggestions for improvements
	- Want to add more language support
	- Build interesting projects using this model

	Please open an issue or pull request on GitHub.

	## 📧 Contact

	- Author: Herlambang Haryo Putro
	- Email: herlambangharyoputro@gmail.com
	- GitHub: [@herlambangharyoputro](https://github.com/herlambangharyoputro)
	- Project: Job Market Intelligence Platform

	## 🔄 Version History

	- v1.0.0 (December 2024): Initial release
	- 200+ skills across 7 categories
	- 40+ aliases for normalization
	- Proficiency level detection
	- Indonesian language support
	- Zero dependencies

	## ⚠️ Limitations

	### Coverage
	- Limited to predefined skill taxonomy (200+ skills)
	- New/emerging skills may be categorized as 'other'
	- Domain-specific skills may not be recognized

	### Language
	- Primarily optimized for Indonesian job market
	- May not capture all regional variations
	- English technical terms preferred over Indonesian equivalents

	### Accuracy
	- Rule-based approach may miss context-dependent skills
	- Acronyms can be ambiguous (e.g., "AI" = Artificial Intelligence or Adobe Illustrator)
	- Proficiency detection based on keywords only

	### Recommendations
	- Best for structured skill lists (bullets, commas)
	- Review 'other' category for domain-specific additions
	- Combine with manual review for critical applications
	- Consider ML-based approach for unstructured text

	## 🎯 Future Improvements

	Planned features for v2.0:
	- Expanded skill taxonomy (300+ skills)
	- Industry-specific categories
	- Skill clustering and relationships
	- Confidence scoring
	- Multi-language support (Javanese, Sundanese)
	- Experience year extraction
	- Certification detection

	---

	Last Updated: December 2024
	Model Version: 1.0.0
	Status: ✅ Production Ready
	Type: Rule-based NER

	For questions or collaboration, visit [GitHub](https://github.com/herlambangharyoputro/job-market-intelligence-platform).