Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,16 @@ flowchart TD

Cross-year-matched rows are never sent to the ODS — they're only made available through the Runway app's API, which EDU and other external consumers query.

### Roster sources & no-ODS year selectability

A roster is the student lookup the executor matches input rows against. Source precedence:

1. **ODS** — for `sendToOds` years, the executor fetches the roster from the ODS API.
2. **EDU** — for no-ODS (`sendToOds=false`) years, if `crossYearMatchAvailable`, the executor pulls the roster from EDU (Snowflake) via `appUrls.roster` as NDJSON. EDU is preferred over the S3 file when available (the executor handles this preference).
3. **S3 roster file** — the fallback for no-ODS years when cross-year matching is unavailable (`__rosters/...jsonl`). The app omits `rosterFilePath` from the payload when `crossYearMatchAvailable` is true (it would be a dangling pointer).

A no-ODS year is **selectable** at job creation, and shows **green** ("roster loaded") on the ODS-config page, when a roster file exists **OR** the partner has cross-year matching enabled. This gate is the partner setting only (`crossYearMatchingEnabled`) — no creds/connection check, so a year can be selectable while the executor's `crossYearMatchAvailable` (which also requires `canConnect`) is false; that case fails cleanly at run time per run atomicity.

### S3 Path Structure

```
Expand Down
21 changes: 21 additions & 0 deletions app/api/integration/tests/earthbeam-api.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,27 @@ describe('Earthbeam API', () => {
expect(res.body.crossYearMatchAvailable).toBe(false);
expect(res.body.appUrls.roster).toBeUndefined();
});

it('omits rosterFilePath on a no-ODS job when cross-year matching is available (EDU is the source)', async () => {
const authService = app.get(EarthbeamApiAuthService);
const noOdsJob = await seedJob({
sendToOds: false,
schoolYearId: '2324',
bundle: bundleA,
tenant: tenantA,
});
const noOdsRun = noOdsJob.runs[0];
const noOdsToken = await authService.createAccessToken({ runId: noOdsRun.id });

const res = await request(app.getHttpServer())
.get(`/earthbeam/jobs/${noOdsRun.id}`)
.set('Authorization', `Bearer ${noOdsToken}`);

expect(res.status).toBe(200);
expect(res.body.crossYearMatchAvailable).toBe(true);
expect(res.body.appUrls.roster).toBeDefined();
expect(res.body.rosterFilePath).toBeUndefined();
});
});

// TODO: add tests for things other than descriptor mappings
Expand Down
40 changes: 40 additions & 0 deletions app/api/integration/tests/external-api.v1.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,46 @@ describe('ExternalApiV1', () => {
expect(job?.odsId).toBeNull();
expect(job?.sendToOds).toBe(false);
});

it('should create a no-ODS job without a roster file when cross-year matching is enabled', async () => {
await prisma.schoolYearConfig.create({
data: {
partnerId: partnerA.id,
schoolYearId: '2324',
isEnabled: true,
sendToOds: false,
},
});
await prisma.partner.update({
where: { id: partnerA.id },
data: { crossYearMatchingEnabled: true },
});

const doesFileExistMock = app.get(FileService).doesFileExist as jest.Mock;
doesFileExistMock.mockResolvedValue(false);

try {
const res = await request(app.getHttpServer())
.post(endpoint)
.set('Authorization', `Bearer ${token}`)
.send({
...jobInput,
schoolYear: '2024',
});

expect(res.status).toBe(201);

const job = await prisma.job.findUnique({ where: { uid: res.body.uid } });
expect(job?.odsId).toBeNull();
expect(job?.sendToOds).toBe(false);
} finally {
doesFileExistMock.mockResolvedValue(true);
await prisma.partner.update({
where: { id: partnerA.id },
data: { crossYearMatchingEnabled: false },
});
}
});
});

describe('API Client Info', () => {
Expand Down
70 changes: 70 additions & 0 deletions app/api/integration/tests/jobs.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -400,6 +400,76 @@ describe('POST /jobs', () => {
}
});

it('should create a no-ODS job without a roster file when cross-year matching is enabled', async () => {
await prisma.schoolYearConfig.create({
data: {
partnerId: tenantA.partnerId,
schoolYearId: '2324',
isEnabled: true,
sendToOds: false,
},
});
await prisma.partner.update({
where: { id: tenantA.partnerId },
data: { crossYearMatchingEnabled: true },
});

const doesFileExistMock = app.get(FileService).doesFileExist as jest.Mock;
doesFileExistMock.mockResolvedValue(false);

try {
const res = await request(app.getHttpServer())
.post(endpoint)
.set('Cookie', [sessionA.cookie])
.send({
...postJobDto,
schoolYearId: '2324',
});

expect(res.status).toBe(201);

const job = await prisma.job.findUnique({ where: { id: res.body.id } });
expect(job?.odsId).toBeNull();
expect(job?.sendToOds).toBe(false);
} finally {
doesFileExistMock.mockResolvedValue(true);
await prisma.partner.update({
where: { id: tenantA.partnerId },
data: { crossYearMatchingEnabled: false },
});
}
});

it('should reject a no-ODS job with no roster file when cross-year matching is disabled', async () => {
await prisma.schoolYearConfig.create({
data: {
partnerId: tenantA.partnerId,
schoolYearId: '2324',
isEnabled: true,
sendToOds: false,
},
});
// partner A defaults to crossYearMatchingEnabled=false

const doesFileExistMock = app.get(FileService).doesFileExist as jest.Mock;
doesFileExistMock.mockResolvedValue(false);

try {
const res = await request(app.getHttpServer())
.post(endpoint)
.set('Cookie', [sessionA.cookie])
.send({
...postJobDto,
schoolYearId: '2324',
});

expect(res.status).toBe(400);
expect(res.body.message).toContain('No roster file found');
} finally {
doesFileExistMock.mockResolvedValue(true);
}
});

it('should reject requests when the school year is not enabled', async () => {
const res = await request(app.getHttpServer())
.post(endpoint)
Expand Down
45 changes: 38 additions & 7 deletions app/api/integration/tests/school-year-config.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -286,22 +286,22 @@ describe('GET /school-year-config/tenant', () => {
expect(yearIds).toContain('2526');
expect(yearIds).not.toContain('2324');

// 2425: sendToOds=true → hasRoster is null (no S3 check), hasOds from tenant's ODS config
// 2425: sendToOds=true → hasNonOdsRoster is null (no S3 check), hasOds from tenant's ODS config
const row2425 = res.body.find((r: any) => r.schoolYearId === '2425');
expect(row2425).toMatchObject({
sendToOds: true,
hasOds: true,
hasRoster: null,
hasNonOdsRoster: null,
startYear: 2024,
endYear: 2025,
});

// 2526: sendToOds=false → hasRoster checked via S3, hasOds still reflects ODS config existence
// 2526: sendToOds=false → hasNonOdsRoster checked via S3, hasOds still reflects ODS config existence
const row2526 = res.body.find((r: any) => r.schoolYearId === '2526');
expect(row2526).toMatchObject({
sendToOds: false,
hasOds: true,
hasRoster: true,
hasNonOdsRoster: true,
startYear: 2025,
endYear: 2026,
});
Expand Down Expand Up @@ -340,19 +340,50 @@ describe('GET /school-year-config/tenant', () => {
expect(row2526.hasOds).toBe(true);
});

it('should return hasRoster=false when roster file does not exist', async () => {
it('should return hasNonOdsRoster=false when roster file does not exist', async () => {
const doesFileExistMock = app.get(FileService).doesFileExist as jest.Mock;
doesFileExistMock.mockResolvedValue(false);

try {
const res = await request(app.getHttpServer()).get(endpoint).set('Cookie', [cookieA]);

// 2526 is seeded as sendToOds=false, so hasRoster reflects the S3 check
// 2526 is seeded as sendToOds=false, so hasNonOdsRoster reflects the S3 check
const row2526 = res.body.find((r: any) => r.schoolYearId === '2526');
expect(row2526.hasRoster).toBe(false);
expect(row2526.hasNonOdsRoster).toBe(false);
} finally {
doesFileExistMock.mockResolvedValue(true);
}
});

it('should return hasNonOdsRoster=true for a no-ODS year when cross-year matching is enabled, without an S3 check', async () => {
const doesFileExistMock = app.get(FileService).doesFileExist as jest.Mock;
doesFileExistMock.mockClear();
doesFileExistMock.mockResolvedValue(false);
await global.prisma.partner.update({
where: { id: partnerA.id },
data: { crossYearMatchingEnabled: true },
});

try {
const res = await request(app.getHttpServer()).get(endpoint).set('Cookie', [cookieA]);

// 2526 is sendToOds=false; toggle on → hasNonOdsRoster true even with no file
const row2526 = res.body.find((r: any) => r.schoolYearId === '2526');
expect(row2526.hasNonOdsRoster).toBe(true);

// ODS years stay null regardless of the toggle
const row2425 = res.body.find((r: any) => r.schoolYearId === '2425');
expect(row2425.hasNonOdsRoster).toBeNull();

// Toggle short-circuits the S3 check entirely
expect(doesFileExistMock).not.toHaveBeenCalled();
} finally {
doesFileExistMock.mockResolvedValue(true);
await global.prisma.partner.update({
where: { id: partnerA.id },
data: { crossYearMatchingEnabled: false },
});
}
});
});
});
11 changes: 8 additions & 3 deletions app/api/src/earthbeam/api/earthbeam-api.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -171,9 +171,14 @@ export class EarthbeamApiService {
},
crossYearMatchAvailable,
sendToOds: job.sendToOds,
rosterFilePath: job.sendToOds
? undefined
: `s3://${this.configService.rosterBucket()}/${rosterFileKey(job, job.schoolYear)}`,
// When cross-year matching is available, the executor pulls the roster
// from EDU via appUrls.roster, so the S3 file path would be a dangling
// (often nonexistent) pointer — omit it. The executor only reads
// rosterFilePath in its non-cross-year branch.
rosterFilePath:
job.sendToOds || crossYearMatchAvailable
? undefined
: `s3://${this.configService.rosterBucket()}/${rosterFileKey(job, job.schoolYear)}`,
// odsConnection check narrows the type — the early guard ensures it's present when sendToOds
assessmentDatastore:
odsConnection && job.sendToOds
Expand Down
19 changes: 15 additions & 4 deletions app/api/src/jobs/jobs.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,21 @@ export class JobsService {
}

if (!config.sendToOds) {
const rosterKey = rosterFileKey({ partnerId: input.tenant.partnerId, tenantCode: input.tenant.code }, config.schoolYear);
const rosterExists = await this.fileService.doesFileExist(rosterKey, this.appConfig.rosterBucket());
if (!rosterExists) {
return { status: 'error', code: 'roster_file_missing' };
// A no-ODS year is valid if a roster file exists OR the partner has
// cross-year matching enabled (EDU can supply the roster). This is the
// partner setting only — no creds/connection check.
// Short-circuit the S3 check when the toggle is on; we don't need the file.
const partner = await this.prisma.partner.findUnique({
where: { id: input.tenant.partnerId },
select: { crossYearMatchingEnabled: true },
});

if (!partner?.crossYearMatchingEnabled) {
const rosterKey = rosterFileKey({ partnerId: input.tenant.partnerId, tenantCode: input.tenant.code }, config.schoolYear);
const rosterExists = await this.fileService.doesFileExist(rosterKey, this.appConfig.rosterBucket());
if (!rosterExists) {
return { status: 'error', code: 'roster_file_missing' };
}
}

return {
Expand Down
18 changes: 16 additions & 2 deletions app/api/src/school-year-config/school-year-config.controller.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,15 @@ export class SchoolYearConfigController {
@Authorize('school-year-config.read')
@Get('tenant')
async getTenantConfig(@TenantDecorator() tenant: Tenant) {
// When cross-year matching is enabled, EDU can supply the roster, so a
// no-ODS year has a roster regardless of any S3 file. Partner setting only
// — no creds/connection check.
const partner = await this.prisma.partner.findUnique({
where: { id: tenant.partnerId },
select: { crossYearMatchingEnabled: true },
});
const crossYearMatchingEnabled = partner?.crossYearMatchingEnabled ?? false;

const schoolYears = await this.prisma.schoolYear.findMany({
where: {
schoolYearConfig: {
Expand Down Expand Up @@ -73,8 +82,13 @@ export class SchoolYearConfigController {
);
}

const hasRoster = config.sendToOds
// ODS years use an ODS-fetched roster, so this is null for them. For
// no-ODS years, a roster is available from an S3 file or, when
// cross-year matching is enabled, from EDU.
const hasNonOdsRoster = config.sendToOds
? null
: crossYearMatchingEnabled
? true
: await this.fileService.doesFileExist(
rosterFileKey({ partnerId: tenant.partnerId, tenantCode: tenant.code }, schoolYear),
this.appConfig.rosterBucket(),
Expand All @@ -86,7 +100,7 @@ export class SchoolYearConfigController {
endYear: schoolYear.endYear,
sendToOds: config.sendToOds,
hasOds: schoolYear.odsConfig.length > 0,
hasRoster,
hasNonOdsRoster,
};
})
);
Expand Down
6 changes: 3 additions & 3 deletions app/fe/src/app/Pages/Home/SetupRequiredPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ export const SetupRequiredPage = () => {
const doesAnyYearSendToOds = yearConfigs.some((y) => y.sendToOds);
if (!doesAnyYearSendToOds) {
return (
<SetupMessage title="Roster File Required">
<SetupMessage title="Roster Required">
<Box textStyle="bodyLarge">
Before you can start uploading assessments, a roster file must be loaded for your
Before you can start uploading assessments, a roster must be loaded for your
district. Please contact support for assistance.
</Box>
<ContactSupport message="Roster file needs to be loaded for my district." />
<ContactSupport message="Roster needs to be loaded for my district." />
</SetupMessage>
);
}
Expand Down
6 changes: 3 additions & 3 deletions app/fe/src/app/Pages/Jobs/JobCreatePage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -209,8 +209,8 @@ export const JobCreatePage = () => {
label: `${year.startYear} - ${year.endYear} school year${
year.sendToOds && !year.hasOds
? ' (no ODS configured)'
: !year.sendToOds && year.hasRoster !== true
? ' (no roster file loaded)'
: !year.sendToOds && year.hasNonOdsRoster !== true
? ' (no roster available)'
: ''
}`,
value: year.schoolYearId,
Expand All @@ -219,7 +219,7 @@ export const JobCreatePage = () => {
const year = selectableYears.find((row) => row.schoolYearId === option.value);
if (!year) return true; // narrow .find() return type
if (year.sendToOds && !year.hasOds) return true;
if (!year.sendToOds && year.hasRoster !== true) return true;
if (!year.sendToOds && year.hasNonOdsRoster !== true) return true;
return false;
}}
></RunwaySelect>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ export const UnmatchedStudents = ({ job }: { job: GetJobDto }) => {
</>
)}
If the file already contains the correct ID, then the student likely does not exist in
{job.sendToOds ? ' the ODS' : ' the roster file'}. Contact your district administrator.
{job.sendToOds ? ' the ODS' : ' the roster'}. Contact your district administrator.
</Box>
</UnmatchedStudentStep>
<UnmatchedStudentStep>
Expand Down
Loading