Security features persistence

README.md

@@ -4,7 +4,7 @@
 [![License](https://img.shields.io/badge/License-UNLICENSE-blue.svg)](https://raw.githubusercontent.com/7i/shorter/master/UNLICENSE)
 [![License](https://img.shields.io/badge/License-0BSD-blue.svg)](https://raw.githubusercontent.com/7i/shorter/master/LICENSE)
 # shorter
-URL shortener with pastebin and file upload functions
+URL shortener with pastebin and QR code support
 
 
 ## WIP
@@ -27,55 +27,156 @@ go get github.com/7i/shorter
 shorter /path/to/config
 ```
 
-## Examples
-A deployed version of shorter is accessable at [7i.se](http://7i.se)
+## Features
+
+### URL shortening
+
+Create a short link via the web UI or the quick-add GET syntax:
 
-create a temporary link to "https://www.example.com" via a GET request that is as short as possible:
 ```bash
+# Quick-add: shortest available key
 7i.se?https://www.example.com
-or
-7i.se/?https://www.example.com
+
+# Quick-add: custom key
+7i.se/mykey?https://www.example.com
 ```
-create a temporary link to "https://www.example.com" via a GET request using the key "KeyToExample":
-```bash
-7i.se/KeyToExample?https://www.example.com
-or
-7i.se/KeyToExample/?https://www.example.com
+
+Four key buckets, each with a configurable timeout:
+
+| Length | Example | Default timeout |
+|--------|---------|-----------------|
+| 1 char | `7i.se/a` | 24 h |
+| 2 chars | `7i.se/ab` | 7 d |
+| 3 chars | `7i.se/abc` | 60 d |
+| Custom (4–64 chars) | `7i.se/mykey` | 30 d |
+
+Append `~` to any key to preview where it points without consuming an access (`7i.se/a~`).
+
+### Pastebin
+
+Submit a text blob via the web UI (`requestType=text`). Large blobs are transparently gzip-compressed before storage.
+
+### QR codes
+
+```
+GET /qr/{key}
+```
+
+Returns a 256×256 PNG QR code encoding the full short URL for the given key.
+
+### JSON API
+
+#### Shorten a URL
+
+```
+POST /api/v1/shorten
+Content-Type: application/json
+
+{
+  "url":    "https://www.example.com",
+  "len":    "1",          // "1", "2", "3", or "custom"
+  "key":    "mykey",      // optional, required when len=custom
+  "x_times": 5            // optional: delete after N accesses (0 = unlimited)
+}
+```
+
+Response `201 Created`:
+```json
+{
+  "key":       "a",
+  "short_url": "https://7i.se/a",
+  "expires":   "Mon 2025-01-01 12:00 UTC"
+}
+```
+
+#### Look up a key (no access consumed)
+
+```
+GET /api/v1/lookup/{key}
 ```
 
+Response `200 OK`:
+```json
+{
+  "key":             "a",
+  "link_type":       "url",
+  "url":             "https://www.example.com",
+  "expires":         "Mon 2025-01-01 12:00 UTC",
+  "access_count":    42,
+  "times_remaining": -1
+}
+```
+
+`times_remaining: -1` means unlimited accesses.
+
+### Persistence
+
+Links are stored in a [bbolt](https://github.com/etcd-io/bbolt) embedded database (`shorterdata/shorter.db`). They survive server restarts and are pruned automatically on startup when expired.
+
+### Admin endpoint
+
+```
+GET /listactive~
+```
+
+HTTP Basic Auth required. Password is verified as `sha256(password + Salt) == HashSHA256` using constant-time comparison.
+
+### Rate limiting
+
+POST requests are rate-limited to **30 per minute per IP** (sliding window). Exceeding the limit returns `429 Too Many Requests`.
+
+### Blocklist
+
+Domains can be blocked via:
+- `BlockedDomains` list in the config file
+- An optional newline-delimited `BlocklistFile` (lines starting with `#` are comments)
+
+Blocked URLs return `403 Forbidden`.
+
+## Security
+
+- TLS 1.2+ with AEAD-only cipher suites (AES-GCM, ChaCha20-Poly1305), X25519 curve preferred
+- `X-Content-Type-Options: nosniff` and `Referrer-Policy: no-referrer` on all responses
+- Optional `Strict-Transport-Security`, `Content-Security-Policy`, and `Report-To` headers
+- Decompression bomb protection: 20 MiB hard cap on gzip decompression
+- All URL inputs validated; only `http://` and `https://` schemes accepted
+- Concurrent-safe link storage with no data races (verified with `-race`)
+
 ## TODO
 - [x] Implement shortening of URLs
-   - [x] 1 char long - configurabe timeout
-   - [x] 2 chars long - configurabe timeout
-   - [x] 3 chars long - configurabe timeout
+   - [x] 1 char long - configurable timeout
+   - [x] 2 chars long - configurable timeout
+   - [x] 3 chars long - configurable timeout
    - [x] make timeouts configurable
    - [x] temporary word bindings (7i.se/coolthing)
-   - [x] quick add link via get request with syntax 7i.se?https://example.com
-   - [x] quick add word bindings link via get request with syntax 7i.se/coolthing?https://example.com where coolthing is the key
-   - [ ] optional removal of link after N accesses
-- [x] Add functionality to print where a link is pointing by adding ~ at the end of the link e.g. 7i.se/a~ will display where 7i.se/a is pointing to
+   - [x] quick add link via GET request with syntax 7i.se?https://example.com
+   - [x] quick add word bindings link via GET request with syntax 7i.se/coolthing?https://example.com
+   - [x] optional removal of link after N accesses (x_times)
+- [x] Add functionality to print where a link is pointing by adding ~ at the end of the link
 - [x] Add config file that specifies relevant options
 - [x] Pastebin functionality with same timeouts as above
-- [x] Move to ssl with Let's Encrypt
-- [ ] Save all active links in a database file instead of gob files 
-- [ ] Add support for subdomains with diffrent configs e.g. d1.7i.se
-   - [ ] Add password/client cert protected subdomain management e.g. d1.7i.se/admin
-   - [ ] Let the user managing a subdomain specify generic links and set timeouts, including "no timeout" for the shortened links, text-blobs and files.
+- [x] Move to SSL with Let's Encrypt
+- [x] Save all active links in a database file (bbolt)
+- [x] JSON REST API for programmatic shortening and lookup
+- [x] QR code generation per short link
+- [x] Click analytics (access count per link)
+- [x] URL deduplication (repeated submissions return existing key)
+- [x] Per-IP rate limiting
+- [x] Blocklist support (config + file)
 - [x] Enable CSP
-   - [x] Move all js and css to seperate files and modify html/template files to use these
+   - [x] Move all js and css to separate files and modify html/template files to use these
    - [ ] Setup a CSP report collector
-- [ ] Use blocklists for known malware sites, integrate with:
+- [ ] Add support for subdomains with different configs e.g. d1.7i.se
+   - [ ] Add password/client cert protected subdomain management
+   - [ ] Let the user managing a subdomain specify generic links and set timeouts
+- [ ] Integrate with external malware/blocklist feeds:
    - [ ] https://www.stopbadware.org/firefox
    - [ ] https://www.malwaredomainlist.com
    - [ ] https://isc.sans.edu/suspicious_domains.html
-   - [ ] https://zeltser.com/malicious-ip-blocklists/
-   - [ ] if linking to a page that redirects, follow redirects only for 5 levels and display error if redirected more times
-- [ ] Include report form to take down links that breaks terms of usage
-   - [ ] implement capcha for submitting reports to take down links
-- [x] Create Terms of usage
+- [ ] Include report form to take down links that break terms of use
+- [x] Create Terms of use
 
 
 ## License
 
 The `shorter` project is dual-licensed to the [public domain](UNLICENSE) and under a [zero-clause BSD license](LICENSE). You may choose either license to govern your use of `shorter`.
-

api.go

@@ -0,0 +1,178 @@
+package main
+
+import (
+	"encoding/json"
+	"net/http"
+	"strings"
+	"time"
+)
+
+// API request / response types
+
+type apiShortenRequest struct {
+	URL    string `json:"url"`
+	Key    string `json:"key"`    // optional custom key (4-64 chars)
+	Len    string `json:"len"`    // "1", "2", "3", or "custom"
+	XTimes int    `json:"x_times"` // max accesses; omit or 0 for unlimited
+}
+
+type apiShortenResponse struct {
+	Key      string `json:"key"`
+	ShortURL string `json:"short_url"`
+	Expires  string `json:"expires"`
+}
+
+type apiLookupResponse struct {
+	Key            string `json:"key"`
+	LinkType       string `json:"link_type"`
+	URL            string `json:"url,omitempty"`
+	Expires        string `json:"expires"`
+	AccessCount    int64  `json:"access_count"`
+	TimesRemaining int    `json:"times_remaining"` // -1 = unlimited
+}
+
+func handleAPI(mux *http.ServeMux) {
+	mux.HandleFunc("/api/v1/shorten", apiShorten)
+	mux.HandleFunc("/api/v1/lookup/", apiLookup)
+}
+
+func writeJSON(w http.ResponseWriter, status int, v interface{}) {
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(status)
+	json.NewEncoder(w).Encode(v)
+}
+
+// apiShorten handles POST /api/v1/shorten
+// Body: {"url":"https://...","len":"1","x_times":0}
+// Returns: {"key":"ab","short_url":"https://host/ab","expires":"..."}
+func apiShorten(w http.ResponseWriter, r *http.Request) {
+	if !validRequest(r) {
+		writeJSON(w, http.StatusForbidden, map[string]string{"error": "forbidden"})
+		return
+	}
+	if r.Method != http.MethodPost {
+		writeJSON(w, http.StatusMethodNotAllowed, map[string]string{"error": "method not allowed"})
+		return
+	}
+	if !rateLimitAllow(r.RemoteAddr) {
+		writeJSON(w, http.StatusTooManyRequests, map[string]string{"error": "rate limit exceeded"})
+		return
+	}
+
+	var req apiShortenRequest
+	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+		writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON body"})
+		return
+	}
+
+	if !validURL(req.URL) {
+		writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid URL; only http and https are allowed"})
+		return
+	}
+	if isBlocklisted(req.URL) {
+		writeJSON(w, http.StatusForbidden, map[string]string{"error": "URL is not allowed"})
+		return
+	}
+
+	scheme := "http"
+	if r.TLS != nil {
+		scheme = "https"
+	}
+
+	// Dedup: return existing unlimited short link for the same URL if no custom key requested.
+	if req.Key == "" && req.XTimes <= 0 {
+		if existing := findExistingURL(r.Host, req.URL); existing != nil {
+			writeJSON(w, http.StatusOK, apiShortenResponse{
+				Key:      existing.Key,
+				ShortURL: scheme + "://" + r.Host + "/" + existing.Key,
+				Expires:  existing.Timeout.Format(dateFormat),
+			})
+			return
+		}
+	}
+
+	// Choose the target LinkLen bucket.
+	var ll *LinkLen
+	switch req.Len {
+	case "2":
+		ll = &domainLinkLens[r.Host].LinkLen2
+	case "3":
+		ll = &domainLinkLens[r.Host].LinkLen3
+	case "custom":
+		if !validate(req.Key) || len(req.Key) < 4 || len(req.Key) > maxKeyLen {
+			writeJSON(w, http.StatusBadRequest, map[string]string{"error": errInvalidCustomKey})
+			return
+		}
+		ll = &domainLinkLens[r.Host].LinkCustom
+	default:
+		ll = &domainLinkLens[r.Host].LinkLen1
+	}
+
+	xTimes := req.XTimes
+	if xTimes < 1 {
+		xTimes = -1
+	} else if xTimes > config.LinkAccessMaxNr {
+		xTimes = config.LinkAccessMaxNr
+	}
+
+	ll.Mutex.RLock()
+	timeout := ll.Timeout
+	ll.Mutex.RUnlock()
+
+	lnk := &Link{
+		Key:      req.Key,
+		LinkType: "url",
+		Data:     req.URL,
+		Times:    xTimes,
+		Timeout:  time.Now().Add(timeout),
+	}
+	key, err := ll.Add(lnk)
+	if err != nil {
+		writeJSON(w, http.StatusInternalServerError, map[string]string{"error": err.Error()})
+		return
+	}
+
+	writeJSON(w, http.StatusCreated, apiShortenResponse{
+		Key:      key,
+		ShortURL: scheme + "://" + r.Host + "/" + key,
+		Expires:  lnk.Timeout.Format(dateFormat),
+	})
+}
+
+// apiLookup handles GET /api/v1/lookup/{key}
+// Returns metadata for the key without consuming an access.
+func apiLookup(w http.ResponseWriter, r *http.Request) {
+	if !validRequest(r) {
+		writeJSON(w, http.StatusForbidden, map[string]string{"error": "forbidden"})
+		return
+	}
+	if r.Method != http.MethodGet {
+		writeJSON(w, http.StatusMethodNotAllowed, map[string]string{"error": "method not allowed"})
+		return
+	}
+
+	key := strings.TrimPrefix(r.URL.Path, "/api/v1/lookup/")
+	key = strings.TrimSuffix(key, "/")
+	if !validate(key) || len(key) == 0 {
+		writeJSON(w, http.StatusBadRequest, map[string]string{"error": errInvalidKey})
+		return
+	}
+
+	lnk, _ := lookupLink(r.Host, key)
+	if lnk == nil {
+		writeJSON(w, http.StatusNotFound, map[string]string{"error": "key not found"})
+		return
+	}
+
+	resp := apiLookupResponse{
+		Key:            key,
+		LinkType:       lnk.LinkType,
+		Expires:        lnk.Timeout.Format(dateFormat),
+		AccessCount:    lnk.AccessCount,
+		TimesRemaining: lnk.Times,
+	}
+	if lnk.LinkType == "url" {
+		resp.URL = lnk.Data
+	}
+	writeJSON(w, http.StatusOK, resp)
+}