Upstram merge

.profile

@@ -0,0 +1,14 @@
+# Sourced by Heroku before the dyno's command runs, for every process type —
+# web, worker, the release phase, and one-off `heroku run` dynos.
+#
+# glibc hands each thread its own malloc arena, up to 8 x nproc, and a dyno
+# reports the host's core count rather than its own share. Arenas are never
+# returned to the OS, so on a process with real thread churn — the publisher
+# builds a fresh ThreadPoolExecutor every 15s, and boto3's managed transfer
+# adds ten threads per download — RSS ratchets upward and never comes back.
+# That is what put the 512MB worker at 110% of quota with 3081 R14s in a day.
+#
+# Set as a DEFAULT, not an override: .profile is sourced after config vars are
+# injected, so a bare assignment would silently stomp a value set from the
+# dashboard and make this untunable without a deploy.
+export MALLOC_ARENA_MAX="${MALLOC_ARENA_MAX:-2}"

Procfile

@@ -1,3 +1,3 @@
 release: python manage.py migrate
 web: gunicorn config.wsgi:application --bind 0.0.0.0:$PORT --workers 1 --threads 4
-worker: python manage.py process_tasks
+worker: python manage.py process_tasks --duration 3600

README.md

@@ -311,6 +311,10 @@ All platforms with ephemeral filesystems require `STORAGE_BACKEND=s3` - see `.en
 
 **Memory sizing.** Importing the app costs roughly 100 MB before it serves a request, and a warmed Gunicorn worker settles near 200 MB - so the shipped command runs a single threaded worker (`--workers 1 --threads 4`), which fits a 512 MB dyno with room for media handling. Raise `--workers` only when you raise the memory to match; a rule of thumb is 250 MB per worker. Do **not** add `--max-requests`: gthread stops heartbeating at the start of the request that trips the counter, so the arbiter kills the worker mid-request once `--timeout` (30s) passes - which drops whichever upload happened to be that request, and uploads here can be up to 1 GB. The `worker` process needs the same headroom: it downloads videos to disk and streams them to the platform, and if it is killed mid-publish (Heroku's R15, an OOM kill, a deploy) the affected post is failed by the confirmation sweep rather than left in limbo.
 
+**Why the worker runs with `--duration 3600`.** `process_tasks` is a single non-forking process that runs every task in the same heap and never restarts, so a peak allocation raises RSS permanently - CPython and glibc keep the freed pages in their own arenas. Left alone it ratchets: on a 512 MB Basic dyno it climbed from ~280 MB after a deploy to 566 MB over 15 hours and sat at 110% of quota. `--duration` is checked at the *top* of the run loop, so a task in flight always finishes; the process then exits 0 between tasks and the platform restarts it at its floor. Nothing is lost - a recycle cannot interrupt a publish, and `confirm_pending_publishes` settles anything in flight regardless. Don't go below ~1800s, where Heroku's crash cool-off starts to engage. Other deploy targets keep the plain command: `docker-compose.yml` has no `restart:` policy on the worker, so there a clean exit would simply stop it.
+
+**`MALLOC_ARENA_MAX`.** glibc gives each thread its own arena (up to 64 MB) capped at `8 x nproc`, and containers report the host's core count, so the cap is effectively unbounded. This app has real thread churn - the publisher builds a fresh pool every 15s and boto3's managed transfer adds ten threads per download - and those arenas are never returned to the OS. On Heroku this is already set to 2 by the repo's `.profile`, which applies to web, worker, the release phase and one-off `heroku run` dynos alike; it is written as a default rather than an override, so a config var still wins if you want to tune it. Deploy targets that build from the `Dockerfile` (Render, Railway, docker-compose) do not read `.profile` - set it in their own environment config if the host is memory-tight.
+
 See `architecture.md` for detailed per-platform instructions and cost breakdowns.
 
 ## Project Structure

apps/accounts/views.py

@@ -5,7 +5,6 @@
 from django.shortcuts import redirect, render
 from django.utils import timezone
 from django.views.decorators.http import require_http_methods
-from PIL import Image
 
 
 def health_check(request):
@@ -110,9 +109,17 @@ def _handle_photo_update(request, user):
         return
 
     # Validate minimum dimensions (180x180)
+    #
+    # PIL is imported here rather than at module scope because config.urls
+    # imports this module for ``health_check``, so a top-level import loaded
+    # Pillow's _imaging extension into every web AND worker process at boot,
+    # whether or not an avatar was ever touched. Matches how the rest of the
+    # codebase imports PIL (apps.media_library.services, apps.intelligence.views).
+    from PIL import Image
+
     try:
-        img = Image.open(avatar)
-        width, height = img.size
+        with Image.open(avatar) as img:
+            width, height = img.size
         if width < 180 or height < 180:
             messages.error(request, "Photo must be at least 180×180 pixels.")
             return

apps/analytics/services.py

@@ -14,6 +14,7 @@
 from datetime import timedelta
 from typing import Any, NamedTuple
 
+from django.db import connections
 from django.utils import timezone
 
 from apps.composer.models import PlatformPost
@@ -293,25 +294,27 @@ def account_analytics_bundle(account: SocialAccount, days: int) -> dict[str, Any
     start = end - timedelta(days=2 * days - 1)
     platform_metrics = PLATFORM_METRICS.get(account.platform, [])
 
-    rows = list(
-        AccountInsightsSnapshot.objects.filter(
-            social_account=account,
-            metric_key__in=platform_metrics,
-            date__gte=start,
-            date__lte=end,
-        )
-    )
+    # Four columns, not model instances: AccountInsightsSnapshot carries ``raw``
+    # and ``errors`` JSONFields that Django decodes on hydration and that
+    # nothing below reads. See ``_latest_post_stats`` for the same reasoning at
+    # the scale where it actually hurt.
+    rows = AccountInsightsSnapshot.objects.filter(
+        social_account=account,
+        metric_key__in=platform_metrics,
+        date__gte=start,
+        date__lte=end,
+    ).values_list("metric_key", "date", "value", "captured_at")
     by_metric: dict[str, dict[dt_date, float]] = defaultdict(dict)
     captured_by_metric: dict[str, Any] = {}
     max_captured: Any = None
     metrics_with_account_data: set[str] = set()
-    for r in rows:
-        by_metric[r.metric_key][r.date] = r.value
-        metrics_with_account_data.add(r.metric_key)
-        if r.metric_key not in captured_by_metric or r.captured_at > captured_by_metric[r.metric_key]:
-            captured_by_metric[r.metric_key] = r.captured_at
-        if max_captured is None or r.captured_at > max_captured:
-            max_captured = r.captured_at
+    for metric_key, day, value, captured_at in rows:
+        by_metric[metric_key][day] = value
+        metrics_with_account_data.add(metric_key)
+        if metric_key not in captured_by_metric or captured_at > captured_by_metric[metric_key]:
+            captured_by_metric[metric_key] = captured_at
+        if max_captured is None or captured_at > max_captured:
+            max_captured = captured_at
 
     # Hybrid fallback: for content-attribution metrics, derive a daily series
     # by summing per-post deltas so platforms without ``get_account_metrics``
@@ -566,7 +569,7 @@ def all_posts_for(
 
     posts: list[PlatformPost] = list(qs)
     metrics = post_metrics_for(account.platform)
-    stats_by_post = _latest_post_stats(posts, metrics)
+    stats_by_post = _latest_post_stats([p.id for p in posts], metrics)
 
     rows: list[dict[str, Any]] = []
     for p in posts:
@@ -633,7 +636,7 @@ def post_detail(post: PlatformPost) -> dict[str, Any]:
     """
     account = post.social_account
     metrics = post_metrics_for(account.platform)
-    stats = _latest_post_stats([post], metrics).get(post.id, {})
+    stats = _latest_post_stats([post.id], metrics).get(post.id, {})
     sparklines_by_metric, max_captured = _post_sparklines_with_freshness(post, metrics)
     return {
         "post": post,
@@ -667,46 +670,75 @@ def _label(metric_key: str) -> str:
     return METRICS.get(metric_key, {}).get("label", metric_key.replace("_", " ").title())
 
 
-def _latest_post_stats(posts: Iterable[PlatformPost], metrics: list[str]) -> dict[Any, dict[str, float]]:
-    """For each post, return ``{metric_key: latest value}``."""
-    post_ids = [p.id for p in posts]
+def _latest_post_stats(post_ids: Iterable[Any], metrics: list[str]) -> dict[Any, dict[str, float]]:
+    """For each post id, return ``{metric_key: latest value}``.
+
+    Three columns, never model instances. ``PostInsightsSnapshot`` is one row
+    per (post, metric, day) and carries two JSONFields — ``raw`` is the entire
+    provider response — and Django decodes both eagerly while hydrating a row.
+    Pulling these as models therefore ran a ``json.loads`` over every payload
+    in the table for the account, twice per row, to read three numbers none of
+    which are in the JSON. On an account with 300 posts and 90 days of history
+    that is 216k instances and 432k needless decodes, in a web process with
+    ~60 MB of headroom.
+
+    Where the backend supports it, ``DISTINCT ON`` also does the dedup in
+    Postgres rather than in Python, so the query returns one row per
+    (post, metric) instead of one per day. The ``order_by`` prefix it requires
+    is the ordering this needs anyway.
+
+    That is an optimization, not a requirement: README documents SQLite for
+    local development and small deployments, and SQLite inherits Django's base
+    ``distinct_sql``, which raises ``NotSupportedError`` the moment any field is
+    passed. So the clause is applied only when the backend advertises it, and
+    everything else dedups the same rows in Python. The ``values_list`` above
+    is where nearly all of the saving comes from and it works everywhere.
+    """
+    post_ids = list(post_ids)
     if not post_ids:
         return {}
-    rows = PostInsightsSnapshot.objects.filter(platform_post_id__in=post_ids, metric_key__in=metrics).order_by(
-        "platform_post_id", "metric_key", "-date"
+    rows = (
+        PostInsightsSnapshot.objects.filter(platform_post_id__in=post_ids, metric_key__in=metrics)
+        .order_by("platform_post_id", "metric_key", "-date")
+        .values_list("platform_post_id", "metric_key", "value")
     )
+
     out: dict[Any, dict[str, float]] = defaultdict(dict)
+    if connections[rows.db].features.can_distinct_on_fields:
+        for post_id, metric_key, value in rows.distinct("platform_post_id", "metric_key"):
+            out[post_id][metric_key] = value
+        return out
+
+    # Same ordering, so the first row for each (post, metric) is still the
+    # newest; ``iterator`` keeps the untrimmed result set from being cached.
     seen: set[tuple[Any, str]] = set()
-    for r in rows:
-        key = (r.platform_post_id, r.metric_key)
+    for post_id, metric_key, value in rows.iterator(chunk_size=2000):
+        key = (post_id, metric_key)
         if key in seen:
             continue
         seen.add(key)
-        out[r.platform_post_id][r.metric_key] = r.value
+        out[post_id][metric_key] = value
     return out
 
 
-def _post_sparklines(post: PlatformPost, metrics: list[str]) -> dict[str, list[float]]:
-    """Daily history per metric since publish — for the detail-drawer sparkline."""
-    return _post_sparklines_with_freshness(post, metrics)[0]
-
-
 def _post_sparklines_with_freshness(post: PlatformPost, metrics: list[str]) -> tuple[dict[str, list[float]], Any]:
-    """Same as :func:`_post_sparklines` but also returns the max ``captured_at``.
+    """Daily history per metric since publish, plus the max ``captured_at``.
 
-    Used by :func:`post_detail` so the freshness side-channel
-    (:func:`apps.analytics.freshness.post_freshness`) doesn't need its own
-    ``Max("captured_at")`` aggregate against the same rows.
+    Feeds the detail-drawer sparkline. The freshness value rides along so the
+    side-channel (:func:`apps.analytics.freshness.post_freshness`) doesn't need
+    its own ``Max("captured_at")`` aggregate against the same rows.
     """
-    rows = PostInsightsSnapshot.objects.filter(platform_post=post, metric_key__in=metrics).order_by(
-        "metric_key", "date"
+    rows = (
+        PostInsightsSnapshot.objects.filter(platform_post=post, metric_key__in=metrics)
+        .order_by("metric_key", "date")
+        .values_list("metric_key", "value", "captured_at")
     )
     out: dict[str, list[float]] = defaultdict(list)
     max_captured: Any = None
-    for r in rows:
-        out[r.metric_key].append(r.value)
-        if max_captured is None or r.captured_at > max_captured:
-            max_captured = r.captured_at
+    for metric_key, value, captured_at in rows:
+        out[metric_key].append(value)
+        if max_captured is None or captured_at > max_captured:
+            max_captured = captured_at
     return dict(out), max_captured
 
 

apps/analytics/tests/test_services.py

@@ -99,3 +99,157 @@ def test_account_bundle_keeps_account_reach_instead_of_summing_post_reach(facebo
     series = account_analytics_bundle(facebook_account, 7)["series_map"]["reach"]
 
     assert series[-1] == 10
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_takes_the_newest_row_per_metric(facebook_account):
+    """The dedup moved from a Python ``seen`` set into Postgres ``DISTINCT ON``.
+
+    The ordering prefix is what makes that correct — ``date DESC`` inside the
+    ORDER BY is the only reason the newest row survives — so pin it here rather
+    than trusting the clause to keep meaning what it means.
+    """
+    from apps.analytics.models import PostInsightsSnapshot
+    from apps.analytics.services import _latest_post_stats
+
+    post = _published_platform_post(facebook_account)
+    today = timezone.now().date()
+    for offset, value in ((2, 10.0), (1, 20.0), (0, 30.0)):
+        PostInsightsSnapshot.objects.create(
+            platform_post=post,
+            metric_key="likes",
+            date=today - timedelta(days=offset),
+            value=value,
+        )
+
+    assert _latest_post_stats([post.id], ["likes"]) == {post.id: {"likes": 30.0}}
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_keeps_metrics_and_posts_separate(facebook_account):
+    from apps.analytics.models import PostInsightsSnapshot
+    from apps.analytics.services import _latest_post_stats
+
+    first = _published_platform_post(facebook_account)
+    second = _published_platform_post(facebook_account)
+    today = timezone.now().date()
+    for post, metric, value in (
+        (first, "likes", 1.0),
+        (first, "comments", 2.0),
+        (second, "likes", 3.0),
+    ):
+        PostInsightsSnapshot.objects.create(platform_post=post, metric_key=metric, date=today, value=value)
+
+    assert _latest_post_stats([first.id, second.id], ["likes", "comments"]) == {
+        first.id: {"likes": 1.0, "comments": 2.0},
+        second.id: {"likes": 3.0},
+    }
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_ignores_metrics_not_asked_for(facebook_account):
+    from apps.analytics.models import PostInsightsSnapshot
+    from apps.analytics.services import _latest_post_stats
+
+    post = _published_platform_post(facebook_account)
+    today = timezone.now().date()
+    PostInsightsSnapshot.objects.create(platform_post=post, metric_key="likes", date=today, value=1.0)
+    PostInsightsSnapshot.objects.create(platform_post=post, metric_key="shares", date=today, value=9.0)
+
+    assert _latest_post_stats([post.id], ["likes"]) == {post.id: {"likes": 1.0}}
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_short_circuits_on_no_posts(django_assert_num_queries):
+    from apps.analytics.services import _latest_post_stats
+
+    with django_assert_num_queries(0):
+        assert _latest_post_stats([], ["likes"]) == {}
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_runs_one_query_regardless_of_history(facebook_account, django_assert_num_queries):
+    """No JSON is decoded and no model is hydrated: three columns, one query.
+
+    ``PostInsightsSnapshot`` carries two JSONFields — ``raw`` is the whole
+    provider response — that Django decodes eagerly on hydration. Reading these
+    as models ran a ``json.loads`` per payload per row to fetch three numbers
+    that are not in the JSON, which is what pushed the web dyno to 88% of a
+    512 MB quota.
+    """
+    from apps.analytics.models import PostInsightsSnapshot
+    from apps.analytics.services import _latest_post_stats
+
+    post = _published_platform_post(facebook_account)
+    today = timezone.now().date()
+    bulky = {"payload": "x" * 2000}
+    for offset in range(30):
+        PostInsightsSnapshot.objects.create(
+            platform_post=post,
+            metric_key="likes",
+            date=today - timedelta(days=offset),
+            value=float(offset),
+            raw=bulky,
+        )
+
+    with django_assert_num_queries(1):
+        result = _latest_post_stats([post.id], ["likes"])
+    assert result == {post.id: {"likes": 0.0}}
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_dedups_without_distinct_on(facebook_account, monkeypatch):
+    """SQLite has no DISTINCT ON, and the README supports SQLite deployments.
+
+    Django's base ``distinct_sql`` raises ``NotSupportedError`` as soon as a
+    field is passed, so the whole analytics surface 500s on SQLite if the
+    clause is applied unconditionally. Force the fallback and assert it picks
+    the same rows.
+    """
+    from django.db import connections
+
+    from apps.analytics.models import PostInsightsSnapshot
+    from apps.analytics.services import _latest_post_stats
+
+    post = _published_platform_post(facebook_account)
+    today = timezone.now().date()
+    for offset, value in ((2, 10.0), (1, 20.0), (0, 30.0)):
+        PostInsightsSnapshot.objects.create(
+            platform_post=post,
+            metric_key="likes",
+            date=today - timedelta(days=offset),
+            value=value,
+        )
+    PostInsightsSnapshot.objects.create(platform_post=post, metric_key="comments", date=today, value=7.0)
+
+    monkeypatch.setattr(connections["default"].features, "can_distinct_on_fields", False, raising=False)
+
+    assert _latest_post_stats([post.id], ["likes", "comments"]) == {post.id: {"likes": 30.0, "comments": 7.0}}
+
+
+@pytest.mark.django_db
+def test_latest_post_stats_agrees_across_both_dedup_paths(facebook_account, monkeypatch):
+    """The two branches must be interchangeable, not merely both plausible."""
+    from django.db import connections
+
+    from apps.analytics.models import PostInsightsSnapshot
+    from apps.analytics.services import _latest_post_stats
+
+    first = _published_platform_post(facebook_account)
+    second = _published_platform_post(facebook_account)
+    today = timezone.now().date()
+    for post in (first, second):
+        for offset, metric in ((0, "likes"), (3, "likes"), (1, "comments")):
+            PostInsightsSnapshot.objects.create(
+                platform_post=post,
+                metric_key=metric,
+                date=today - timedelta(days=offset),
+                value=float(offset),
+            )
+
+    ids = [first.id, second.id]
+    with_distinct_on = _latest_post_stats(ids, ["likes", "comments"])
+    monkeypatch.setattr(connections["default"].features, "can_distinct_on_fields", False, raising=False)
+    without_distinct_on = _latest_post_stats(ids, ["likes", "comments"])
+
+    assert with_distinct_on == without_distinct_on