Duplicate from AllGPTORG/PiSA-Lite

Co-authored-by: Arnold <Loewolf@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,331 @@
+---
+language:
+- en
+license: other
+pipeline_tag: image-to-image
+tags:
+- super-resolution
+- image-upscaling
+- image-restoration
+- qnn
+- onnx
+- snapdragon
+- qualcomm
+- mobile
+- npu
+- android
+- generative-ai
+library_name: custom
+---
+
+# PiSA-Lite
+
+**PiSA-Lite is a lightweight, mobile-optimized version of PiSA-SR for Snapdragon-powered smartphones. It is designed to preserve high-quality textures and semantic image details while running through Qualcomm's NPU.**
+
+> PiSA-Lite is an unofficial optimization based on PiSA-SR. It is not affiliated with or endorsed by the original PiSA-SR authors.
+
+## Overview
+
+PiSA-Lite keeps the original PiSA-SR architecture and its semantic image-restoration behavior while preparing the model for mobile deployment.
+
+Unlike small super-resolution models that mainly sharpen edges, PiSA-Lite aims to preserve PiSA-SR's ability to reconstruct material-aware details such as:
+
+- wood grain
+- grass and vegetation
+- metal reflections
+- fabric textures
+- hair and fine surface details
+- building and object structure
+
+The current release includes:
+
+- precompiled Qualcomm QNN Context Binaries for Snapdragon 8 Gen 3
+- ONNX source models for compiling separate builds for other supported Snapdragon chips
+- a fixed 4× super-resolution pipeline
+- an FP16/W8A16 quality configuration
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| Base project | PiSA-SR |
+| Task | Generative image super-resolution |
+| Input | 128 × 128 RGB image |
+| Output | 512 × 512 RGB image |
+| Upscale factor | 4× |
+| Latent shape | `1 × 4 × 64 × 64` |
+| Target runtime | Qualcomm QNN / HTP NPU |
+| Current target SoC | Snapdragon 8 Gen 3 / SM8650 |
+| Current target device family | Samsung Galaxy S24 Family |
+| Deployment format | QNN Context Binary |
+| Source export format | ONNX |
+
+## Files
+
+### Snapdragon 8 Gen 3 QNN Models
+
+The included QNN binaries were compiled specifically for Snapdragon 8 Gen 3 / SM8650:
+
+```text
+pisa_encoder_quality.bin
+pisa_denoiser_quality.bin
+pisa_decoder_quality.bin
+```
+
+| File | Purpose | Precision | Approximate size |
+|---|---|---:|---:|
+| `pisa_encoder_quality.bin` | Converts the image into latent space | FP16 | 74 MiB |
+| `pisa_denoiser_quality.bin` | Restores PiSA textures and semantic details | W8A16 | 791 MiB |
+| `pisa_decoder_quality.bin` | Converts the restored latent into an image | FP16 | 104 MiB |
+
+Total package size is approximately **970 MiB**.
+
+### ONNX Models
+
+The ONNX files are source models for creating separate QNN builds for other supported Snapdragon chips:
+
+```text
+encoder.onnx
+denoiser.onnx
+decoder.onnx
+```
+
+The ONNX files are **not** pre-optimized universal mobile models. They must be compiled for the intended Snapdragon target using Qualcomm AI Hub, QAIRT, or another compatible Qualcomm QNN toolchain.
+
+## Hardware Compatibility
+
+The supplied `.bin` files are compiled for:
+
+```text
+Qualcomm Snapdragon 8 Gen 3
+SoC: SM8650
+Samsung Galaxy S24 Family
+Android 14
+```
+
+QNN Context Binaries are hardware-specific.
+
+Do not assume that the supplied Snapdragon 8 Gen 3 binaries will work on:
+
+- Snapdragon 8 Gen 2
+- Snapdragon 8 Elite
+- Snapdragon 7-series devices
+- Exynos devices
+- MediaTek devices
+- desktop CPUs or GPUs
+
+For another supported Snapdragon chip, use the ONNX models to compile a separate QNN package for that target.
+
+## Pipeline
+
+```text
+128 × 128 input image
+        ↓
+Resize to 512 × 512
+        ↓
+PiSA VAE Encoder
+        ↓
+Latent sampling
+        ↓
+PiSA Denoiser
+        ↓
+PiSA VAE Decoder
+        ↓
+Color correction
+        ↓
+512 × 512 output image
+```
+
+All three model components must be executed in order.
+
+## Precision Configuration
+
+The current quality release uses:
+
+```text
+Encoder:  FP16
+Denoiser: W8A16
+Decoder:  FP16
+```
+
+This reduces the size of the largest PiSA component while keeping the texture-sensitive VAE encoder and decoder in FP16.
+
+## Android Integration
+
+The QNN files are not standalone applications and cannot be opened directly.
+
+An Android application must load them through Qualcomm QAIRT/QNN, typically through a native C++ layer:
+
+```text
+Kotlin / Java UI
+        ↓
+JNI
+        ↓
+C++ QNN runner
+        ↓
+QNN HTP backend
+        ↓
+Encoder → Denoiser → Decoder
+```
+
+Recommended private storage layout:
+
+```text
+/data/user/0/<application-id>/files/models/pisa_sm8650/
+├── pisa_encoder_quality.bin
+├── pisa_denoiser_quality.bin
+└── pisa_decoder_quality.bin
+```
+
+Because the complete model package is large, downloading the files after installation is generally preferable to embedding them directly inside the APK.
+
+## Compiling for Another Snapdragon Chip
+
+Use the ONNX models as source graphs and compile each component for the selected target device:
+
+```text
+encoder.onnx
+denoiser.onnx
+decoder.onnx
+        ↓
+Qualcomm AI Hub / QAIRT / QNN compiler
+        ↓
+target-specific QNN Context Binaries
+```
+
+A separate set of binaries should be generated for each supported Snapdragon family.
+
+The application should detect the device SoC before downloading or loading a model package.
+
+```text
+SM8650 / Snapdragon 8 Gen 3
+→ Load the included SM8650 package
+
+Another supported Snapdragon chip
+→ Download a separately compiled package
+
+Unsupported hardware
+→ Use a smaller GPU or CPU fallback model
+```
+
+## Intended Use
+
+PiSA-Lite is intended for:
+
+- low-resolution photo restoration
+- experimental mobile photography
+- restoring vegetation and environmental details
+- improving material textures
+- enhancing compressed images
+- improving game screenshots
+- research into mobile generative super-resolution
+
+## Out-of-Scope Use
+
+PiSA-Lite is not recommended for:
+
+- forensic image analysis
+- identity verification
+- medical imaging
+- document or evidence recovery
+- exact text reconstruction
+- license-plate recovery
+- recovering factual details that are not visible in the source image
+
+## Limitations
+
+PiSA-Lite is a generative super-resolution model and may create visually plausible details that were not present in the original low-resolution input.
+
+Possible failure cases include:
+
+- invented textures
+- incorrect small text
+- altered faces
+- changed logos or symbols
+- inaccurate fine patterns
+- unstable results on heavily degraded inputs
+- high memory use compared with small CNN upscalers
+- slower inference than models such as SPAN
+- hardware-specific deployment requirements
+
+Generated output should not be treated as factual evidence.
+
+## Current Status
+
+- [x] PiSA-SR quality preserved in local testing
+- [x] Weight-optimized PiSA-Lite package created
+- [x] ONNX models exported
+- [x] QNN Context Binaries compiled
+- [x] Snapdragon 8 Gen 3 NPU inference completed
+- [ ] Public Android runtime example
+- [ ] On-device speed and memory benchmarks
+- [ ] Additional Snapdragon targets
+- [ ] Larger calibration dataset
+- [ ] Hugging Face demo Space
+
+## Comparison
+
+| Model | Sharpness | Semantic texture reconstruction | Mobile suitability |
+|---|---:|---:|---:|
+| SPAN | Good | Limited | High |
+| TinySR | Very good | Medium | Medium |
+| PiSA-SR | Very good | Very high | Low |
+| PiSA-Lite | Very good | Very high in current tests | Targeted at Snapdragon NPU |
+
+The PiSA-Lite quality claim is based on local visual testing and should be validated on a larger public benchmark set.
+
+## Credits
+
+PiSA-Lite is based on the original **PiSA-SR** project and research.
+
+All credit for the original architecture, training method, pretrained model, and research belongs to the original PiSA-SR authors.
+
+PiSA-Lite focuses on:
+
+- mobile deployment
+- weight optimization
+- fixed-shape inference
+- ONNX export
+- Qualcomm QNN compilation
+- Snapdragon NPU execution
+
+## License and Redistribution
+
+The metadata uses `license: other` because redistribution rights may depend on multiple upstream components.
+
+Before redistributing model weights or binaries, review and comply with:
+
+- the original PiSA-SR license
+- the Stable Diffusion 2.1 base-model license
+- all pretrained-model licenses
+- Qualcomm AI Hub and QNN terms
+- any checkpoint or dataset restrictions
+
+Uploading this repository does not automatically grant rights beyond the relevant upstream licenses.
+
+## Disclaimer
+
+This project is experimental and provided without warranty.
+
+The maintainers are not responsible for:
+
+- hallucinated or inaccurate reconstructed details
+- unsupported-device crashes
+- excessive memory usage
+- incorrect Android integration
+- redistribution outside upstream license terms
+- damage or data loss caused by use of the model
+
+Use PiSA-Lite at your own risk.
+
+## Repository
+
+GitHub:
+
+```text
+https://github.com/LoewolfERSTELLER/PiSA-Lite
+```
+
+## Short Description
+
+> PiSA-Lite is an unofficial, mobile-optimized PiSA-SR upscaler for Snapdragon smartphones, designed to preserve high-quality textures and semantic image details through Qualcomm's NPU.

deployment_manifest.json

@@ -0,0 +1,80 @@
+{
+  "status": "SNAPDRAGON-NPU-MODELL FERTIG",
+  "target_device": "Samsung Galaxy S24 (Family)",
+  "mode": "quality",
+  "source_model": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_lite_w8\\pisa_lite_quality_w8_fp16.pt",
+  "input": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\bild_128x128.png",
+  "npu_test_output": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\bild_pisa_snapdragon_npu_512x512.png",
+  "local_test_output": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\local_test\\pisa_lite_local_reference.png",
+  "fixed_shapes": {
+    "encoder_input": [
+      1,
+      3,
+      512,
+      512
+    ],
+    "encoder_outputs": [
+      [
+        1,
+        4,
+        64,
+        64
+      ],
+      [
+        1,
+        4,
+        64,
+        64
+      ]
+    ],
+    "denoiser_input_output": [
+      1,
+      4,
+      64,
+      64
+    ],
+    "decoder_output": [
+      1,
+      3,
+      512,
+      512
+    ]
+  },
+  "components": {
+    "encoder": {
+      "onnx_folder": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\onnx\\encoder.onnx",
+      "onnx_size_bytes": 68428607,
+      "qnn_context_binary": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\qnn\\pisa_encoder_quality.bin",
+      "qnn_size_bytes": 77496320,
+      "compile_options": "--target_runtime qnn_context_binary --qnn_options default_graph_htp_precision=FLOAT16",
+      "compile_job_url": "https://workbench.aihub.qualcomm.com/jobs/jgl11dmj5/"
+    },
+    "denoiser": {
+      "onnx_folder": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\onnx\\denoiser.onnx",
+      "onnx_size_bytes": 1732703600,
+      "qnn_context_binary": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\qnn\\pisa_denoiser_quality.bin",
+      "qnn_size_bytes": 829493248,
+      "compile_options": "--target_runtime qnn_context_binary --quantize_full_type w8a16",
+      "compile_job_url": "https://workbench.aihub.qualcomm.com/jobs/jgl11dvj5/"
+    },
+    "decoder": {
+      "onnx_folder": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\onnx\\decoder.onnx",
+      "onnx_size_bytes": 99084465,
+      "qnn_context_binary": "C:\\Users\\aarno\\Documents\\Projekte\\irgendwas\\pisa_snapdragon_npu\\qnn\\pisa_decoder_quality.bin",
+      "qnn_size_bytes": 108863488,
+      "compile_options": "--target_runtime qnn_context_binary --qnn_options default_graph_htp_precision=FLOAT16",
+      "compile_job_url": "https://workbench.aihub.qualcomm.com/jobs/jgdzzkoz5/"
+    }
+  },
+  "npu_inference_job_urls": {
+    "encoder": "https://workbench.aihub.qualcomm.com/jobs/jgl11dmj5/",
+    "denoiser": "https://workbench.aihub.qualcomm.com/jobs/jgl11dvj5/",
+    "decoder": "https://workbench.aihub.qualcomm.com/jobs/jgdzzkoz5/"
+  },
+  "metrics_npu_vs_local": {
+    "mse": 9.212180157192051e-05,
+    "mae": 0.0046744453720748425,
+    "psnr_db": 40.3563757738287
+  },
+  "created_at": "2026-07-12 18:17:10"
+}

onnx/decoder.onnx/decoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6746bd6393309743723b319be38a427e588ab07c5cb4962d4574c45f5495ba7
+size 99084465

onnx/denoiser.onnx/denoiser.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9523f6f45799ae401c028c55f57f4e067e50486f68f250804e0694d94ffae8a0
+size 1732703600

onnx/encoder.onnx/encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:287c3493e28a2097ee0128ea18a31aa2a14b71bc213c952f97e3d18b2d035fc3
+size 68428607

qnn/pisa_decoder_quality.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d48fd9d7b29b1087098a38665a5711479edbb6a1ea7881f91a1a382805a620f4
+size 108863488

qnn/pisa_denoiser_quality.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c793d4b63bf07c7465c88266db17e84d042b5385b3026a6a78ef2eb074265b69
+size 829493248

qnn/pisa_encoder_quality.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c891813cedbb9c63364b42f13493325aede1e070b2799d8c6617ae7c70a3a06
+size 77496320