---
license: mit
tags:
  - anomaly-detection
  - lstm-autoencoder
  - rest-api
  - network-security
  - intrusion-detection
  - pytorch
  - unsupervised-learning
library_name: pytorch
pipeline_tag: other
---

# LSTM-Autoencoder for REST API Anomaly Detection

Reconstruction-based **LSTM-Autoencoder** for detecting anomalous REST API traffic. Trained exclusively on normal (benign) traffic, so it needs no labelled attack data and can generalize to sequence-dependent attacks — such as Broken Object Level Authorization (BOLA) — that request-isolated, signature-based defences (WAFs) typically miss.

## Model Details

- **Architecture**: 2-layer LSTM encoder (hidden size 64) → 64-dimensional bottleneck → 2-layer LSTM decoder + linear projection
- **Parameters**: ~124,000
- **Framework**: PyTorch 2.x
- **Input modes**:
  - **Embedding mode** (for HTTP-log/URL traffic, e.g. CSIC 2010): abstracted URL-template tokens → 32-d learned embedding → LSTM
  - **Numeric mode** (for network-flow traffic, e.g. CIC-IDS2018, UNSW-NB15): standardized float feature vectors fed directly to the LSTM
- **Sequence length**: sliding-window sessions of `W = 5` consecutive requests/flows
- **Dropout**: 0.2 between LSTM layers
- **Training objective**: minimize mean-squared reconstruction error on normal traffic only
- **Detection rule**: a session is flagged anomalous if its reconstruction error exceeds a threshold calibrated at the 95th percentile of normal-traffic reconstruction errors; the threshold can be further refined via an Active Learning (uncertainty-sampling) loop without retraining the model

This repository provides separate checkpoints per dataset, since each uses a different input mode / feature space (see "Checkpoints" below).

## Intended Use

Detecting anomalous/attack sessions in REST API or network-flow traffic in settings where labelled attack data is unavailable (the common zero-day scenario). Designed for real-time or near-real-time deployment: inference latency is ~0.044–0.050 ms per session on CPU.

**Not intended for**: single-request (non-sequential) anomaly detection, traffic types substantially different from the training distributions below without retraining/fine-tuning, or as a sole security control (best used alongside, not instead of, a WAF).

## Training Data

Trained separately, per dataset, on **normal traffic only**:

| Dataset | Type | Normal sessions (train) |
|---|---|---|
| CSIC 2010 | HTTP logs | 35,996 |
| CIC-IDS2018 | Network flows | 1,676,966 |
| UNSW-NB15 | Network flows | 74,396 |

See the accompanying dataset card for full dataset details and licensing.

## Evaluation

| Dataset | Precision | Recall | F1 | FPR | Latency (ms/session) |
|---|---|---|---|---|---|
| CSIC 2010 | 0.946 | **1.000** | **0.972** | 0.040 | 0.050 |
| CIC-IDS2018 | 0.037† | 0.988 | 0.071† | 0.057 | 0.044 |
| UNSW-NB15 | **0.979** | 0.667 | **0.794** | 0.128 | 0.044 |

† CIC-IDS2018 precision is limited by extreme session-level class imbalance (~454:1 normal:attack) in the test set, not a model deficiency — recall stays high (0.988).

On UNSW-NB15, the model outperforms an Isolation Forest baseline (F1 = 0.663) while running 6.1–6.8× faster.

### Ablation: why normal-only training?

A supplementary ablation trained the same architecture on balanced normal/attack data (reconstruction objective and direct supervision). Balanced reconstruction training degraded F1 by 43–78% relative to the normal-only model across all three datasets, confirming that training exclusively on normal traffic is necessary — not incidental — for reconstruction-based anomaly detection to work as intended. Direct supervision achieved near-perfect scores only because it requires labelled attack data at training time, which defeats the zero-day use case this model targets.

## How to Use

```python
import torch
from model import LSTMAutoencoder  # see repo for model.py

model = LSTMAutoencoder(mode="numeric", input_dim=23, hidden_size=64)
state_dict = torch.load("lstm_ae_cicids2018.pt", map_location="cpu")
model.load_state_dict(state_dict)
model.eval()

# x: tensor of shape (batch, W=5, input_dim), pre-scaled with the matching StandardScaler
with torch.no_grad():
    x_hat = model(x)
    error = ((x - x_hat) ** 2).mean(dim=(1, 2))  # per-session MSE anomaly score

threshold = 0.0136  # example — use the calibrated threshold shipped for each checkpoint
is_anomalous = error > threshold
```

For CSIC 2010 (embedding mode), inputs are integer token sequences produced by the matching vocabulary/abstraction pipeline in the source repository — using the model without that exact preprocessing will produce unreliable results, since the abstraction step is load-bearing for this model (see Limitations).

## Limitations

- **CIC-IDS2018 precision** is fundamentally constrained by the dataset's extreme class imbalance at the session level, independent of model quality.
- **Preprocessing-sensitive**: the URL-abstraction / feature-scaling pipeline used at inference time must exactly match the one used to build the training vocabulary/scaler. Divergent implementations of the same logic (observed during development) silently inflate false positives.
- **Session ordering matters** for flow-based datasets: sessions must be constructed in timestamp order, not raw file order, to match training conditions.
- **BOLA** is a motivating attack class for the sequence-aware design but was not evaluated with a dedicated, labelled BOLA dataset.
- Overlapping sliding-window sessions carry a theoretical train/test leakage risk (adjacent windows share requests), acknowledged as a limitation rather than resolved.
- Latency figures are per-session batched inference, not single live-request latency.

## Citation

```
Kandanaarachchi, K.A.D.S.D. "Detecting Anomalous REST API Traffic."
BSc thesis, University of Vavuniya, 2020/ICT/19.
Supervised by Dr. S. Kirushanth.
```

## License

MIT (code/weights). Underlying datasets retain their original licenses — see the dataset card.