Lab 01: robust line fitting

submissions/gracelh4/lab01.ipynb

@@ -0,0 +1,83 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# Lab 01: Fitting a Line When Your Data Are Lying to You\n\n**ASTR 457, Fall 2026 — posted Thu Sep 3, due Wed Sep 9 by Noon (fork → PR, as always)**\n\n*Why this lab: fitting a line to contaminated data is the statistic behind real discoveries (it's how Gaia BH1's dark companion was found), and refereeing someone else's fit — including a machine's — is the skill you'll use in every collaboration, every review, and every AI-assisted analysis for the rest of your career.*\n\nEvery one of you has your own dataset: `data/<your netid>.csv`, with columns `x`, `y`, `sigma_y`.\nIt was generated from a straight line, $y = mx + b$, with Gaussian noise of the stated\n$\\sigma_y$ — except that some fraction of the points are **outliers** drawn from a much\nbroader distribution. I know the true $m$, $b$, and outlier fraction for your dataset.\nYou don't, and you can't look them up, and your classmates' values are different from yours.\n\n**How this is graded.** Not on whether your code runs — on whether your answers are *right*\nand your uncertainties are *honest*. Part of your grade comes from how close your reported\n$m$ and $b$ are to your truth **in units of your own reported uncertainty**. Report tiny\nerror bars you can't back up and you will lose points even if your central value is close.\nReport huge error bars to be safe and you'll lose points too. Calibration is the skill.\n\n**AI policy reminder.** Use whatever tools you like, including AI assistants — and document\nit in Part 4. You may be selected to defend this lab in person. Welcome to doing research."
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Part 1: The naive fit (20%)\n\nLoad your dataset and fit a straight line by **weighted least squares**, using the reported\n`sigma_y` and all the data points. Report $m \\pm \\sigma_m$ and $b \\pm \\sigma_b$, plot the\ndata (with error bars) and your fit, and compute the reduced $\\chi^2$.\n\nThen answer, in a few sentences: what is the reduced $\\chi^2$ telling you, and why is it\ndoing that? Would you trust these error bars? (This fit is *supposed* to be bad. Understanding\nexactly how it is bad is the point.)"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Part 1: your work here\n"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "**ANSWER (a few sentences):**\n\n*your answer goes here*"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Part 2: A fit you can defend (40%)\n\nNow deal with the outliers. Do this **two ways**:\n\n1. **Sigma-clipping**: iteratively remove points that are discrepant with the fit, refit,\n   and repeat until it converges. Be explicit about your clipping threshold and why you chose it.\n2. **A mixture model**: model each point as coming from either the line (with its stated\n   $\\sigma_y$) or from a broader outlier distribution, and maximize the appropriate likelihood\n   (see Hogg, Bovy & Lang 2010, §3, for the canonical treatment — this is the same problem\n   with different numbers).\n\nFor each method report $m \\pm \\sigma_m$, $b \\pm \\sigma_b$, and (for the mixture) your\nestimate of the outlier fraction. Then commit to a **final answer** — one set of numbers\nyou'd put in a paper — and justify the choice. State clearly how you estimated your\nuncertainties and why you believe them."
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Part 2: your work here\n"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "**FINAL ANSWER:** $m = $ ___ $\\pm$ ___ , $b = $ ___ $\\pm$ ___ , outlier fraction $\\approx$ ___\n\n**Justification and uncertainty method (a short paragraph):**\n\n*your answer goes here*"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Part 3: The referee report (30%)\n\nThe file `ai_solution.ipynb` in this directory is a complete, tidy, confident analysis of\nthis exact problem, produced by an AI assistant. It runs top to bottom without errors, the\nplots look professional, and the prose is fluent. It is also wrong — in at least **three**\ndistinct, substantive ways (cosmetic nitpicks don't count).\n\nWrite a referee report:\n\n1. **Identify** at least three substantive statistical errors.\n2. **Demonstrate** each one quantitatively on *your* dataset — show, with a number or a plot,\n   what the error does to the inference. \"This is bad practice\" is not a demonstration.\n3. **State** what the correct treatment is (you already built it in Part 2).\n\nOne of the best things you can learn this semester is that fluent, error-free-looking\nanalysis and correct analysis are different things. Referee accordingly."
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "**REFEREE REPORT:**\n\n*Flaw 1:*\n\n*Flaw 2:*\n\n*Flaw 3:*"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# Part 3: demonstrations here\n"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Part 4: AI-use appendix & verification plan (10%)\n\n1. **AI use**: which tools did you use (Copilot, Claude, ChatGPT, none, ...), for what,\n   what did they get wrong, and how did you catch it? Honesty is graded; \"I didn't use any\"\n   is fine if true.\n2. **Verification plan**: list the checks you ran before deciding to trust your Part 2\n   numbers (e.g., residual plots, posterior/parameter sanity checks, refitting on synthetic\n   data you generated with known answers). For each check: what would failure have looked like?"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "**AI-USE APPENDIX:**\n\n*your answer goes here*\n\n**VERIFICATION PLAN:**\n\n*your answer goes here*"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}