Skip to content

HOWTO.md

Latest commit

 

History

History
524 lines (404 loc) · 14 KB

HOWTO.md

File metadata and controls

524 lines (404 loc) · 14 KB

HOWTO — using MMP.RollingFiles from code

This repo ships five packages that play different roles. Two of them write bytes to disk; the rest edit, transform, or display the policy that drives those writers. Pick the package that matches what you're doing.

What each package is for

flowchart LR
    subgraph Writers["Writers — append bytes to rolling files"]
        DOTNET[".NET — MMP.RollingFiles<br/>(NuGet)"]
        PYTHON["Python — mmp-rollingfiles<br/>(PyPI)"]
    end

    subgraph PolicyTools["Policy tooling — load / edit / round-trip"]
        JS["@mmpworks/rollingfiles<br/>JSON · YAML · schema"]
        VUE["@mmpworks/rollingfiles-vue<br/>Vuetify 3 components"]
    end

    subgraph Demo["Playground"]
        WEB["@mmpworks/rollingfiles-web<br/>GitHub Pages static site"]
    end

    SPEC[("spec/v1.md<br/>cross-language contract")]

    DOTNET --> SPEC
    PYTHON --> SPEC
    JS --> SPEC
    VUE --> JS
    WEB --> VUE
    WEB --> JS
Loading

Writers and editors never depend on each other — they meet at the FilesManagerPolicy shape defined in spec/v1.md. A policy written by the Vue editor round-trips byte-for-byte through the JS transformer and is consumed unchanged by the .NET or Python writer.

Which package do I need?

Task Package
Append log / event bytes to a rolling file from a .NET app MMP.RollingFiles (NuGet)
Append log / event bytes to a rolling file from Python mmp-rollingfiles (PyPI)
Parse a policy JSON or YAML file into a JS object @mmpworks/rollingfiles
Serialise a policy object back to JSON or YAML @mmpworks/rollingfiles
Show an operator-facing policy editor in a Vue 3 app @mmpworks/rollingfiles-vue
Try the editor without installing anything The playground deploy on GitHub Pages

.NET (C#) — MMP.RollingFiles

Reference implementation. Multi-targets net8.0 + net10.0, AOT-publishable, zero transitive NuGet dependencies (BCL only).

Install

dotnet add package MMP.RollingFiles

Minimal example

using MMP.RollingFiles;

var policy = new FilesManagerPolicy(
    Directory: "/var/log/myapp",
    FileNameTemplate: "events");

using var fm = new FilesManager(policy);
fm.AppendLine("hello world");

That writes /var/log/myapp/events.log with no rolling and no retention — fine for short-lived processes. Realistic deployments set at least a rolling trigger and one retention bound.

Realistic policy — daily roll, retention, compression

using MMP.RollingFiles;

var policy = new FilesManagerPolicy(
    Directory: "/var/log/myapp",
    FileNameTemplate: "events-{date}",
    Interval: RollingInterval.Daily,
    MaxBytesPerFile: 64 * 1024 * 1024,   // 64 MiB hard cap per file
    MaxRetainedFiles: 30,                // keep 30 most recent rolls
    TotalSizeCapBytes: 2L * 1024 * 1024 * 1024, // 2 GiB total budget
    RetentionDays: 90,
    Compression: CompressionMode.Gzip,
    CleanRollOnStartup: true);

using var fm = new FilesManager(policy);

for (var i = 0; i < 10_000; i++)
{
    fm.AppendLine($"event {i:D5} at {DateTimeOffset.UtcNow:O}");
}

The active file rolls on whichever trigger fires first (calendar day boundary or 64 MiB size). Sealed files get gzip'd by a background worker. The pruner enforces all three retention bounds — count, total size, and age — keeping rolled files only if they pass every one.

Hooks — observability + custom logic

FilesManagerHooks is a record of optional callbacks the library invokes at well-defined lifecycle points. All hooks run on a background thread, never on the hot append path.

using System.Threading.Tasks;
using MMP.RollingFiles;

var hooks = new FilesManagerHooks
{
    OnOpenActive = path =>
    {
        // First write to a fresh active file — perfect place to
        // write a JSON header, a CSV column row, an opening brace.
        Console.WriteLine($"opened: {path}");
        return ValueTask.CompletedTask;
    },

    OnPostRoll = path =>
    {
        // The file at `path` is now sealed and ready to be shipped
        // somewhere (S3, Azure Blob, Splunk forwarder, etc.).
        Console.WriteLine($"rolled: {path}");
        return ValueTask.CompletedTask;
    },

    OnPrune = path =>
    {
        // Return false to keep the file. Use this to defer retention
        // until a downstream consumer has acknowledged the events.
        return ValueTask.FromResult(true);
    },

    OnError = err =>
    {
        Console.Error.WriteLine($"[{err.Kind}] {err.Message} ({err.Path})");
        return ValueTask.CompletedTask;
    }
};

using var fm = new FilesManager(policy, hooks);

Loading a policy from JSON

JSON parsing is not in MMP.RollingFiles — the library stays BCL-only and dependency-free. Either deserialize with System.Text.Json directly or use the @mmpworks/rollingfiles JS package on the producing side. The on-disk JSON shape is defined in spec/v1.md.

using System.Text.Json;
using MMP.RollingFiles;

var json = File.ReadAllText("policy.json");
var policy = JsonSerializer.Deserialize<FilesManagerPolicy>(json,
    new JsonSerializerOptions { PropertyNameCaseInsensitive = true });

using var fm = new FilesManager(policy!);

Python — mmp-rollingfiles

Python port of the same contract. Same on-disk format as the .NET writer — a directory written by one is readable byte-for-byte by the other. Field names are the snake_case form of the C# PascalCase.

Install

pip install mmp-rollingfiles

Minimal example

from mmp_rollingfiles import FileManager, FileManagerPolicy

policy = FileManagerPolicy(
    directory="/var/log/myapp",
    file_name_template="events",
)

with FileManager(policy) as fm:
    fm.append_line("hello world")

Realistic policy

from mmp_rollingfiles import (
    FileManager,
    FileManagerPolicy,
    FileManagerHooks,
)

policy = FileManagerPolicy(
    directory="/var/log/myapp",
    file_name_template="events-{date}.{seq}",
    interval="Daily",
    max_bytes_per_file=64 * 1024 * 1024,
    max_retained_files=30,
    total_size_cap_bytes=2 * 1024 * 1024 * 1024,
    retention_days=90,
    compression="Gzip",
    clean_roll_on_startup=True,
)

def on_post_roll(path: str) -> None:
    print(f"rolled: {path}")

hooks = FileManagerHooks(on_post_roll=on_post_roll)

with FileManager(policy, hooks) as fm:
    for i in range(10_000):
        fm.append_line(f"event {i:05d}")

Loading a policy from JSON

The Python FileManagerPolicy is a Pydantic model — JSON in, validated model out, in one call:

import json
from mmp_rollingfiles import FileManager, FileManagerPolicy

with open("policy.json") as fh:
    raw = json.load(fh)

policy = FileManagerPolicy.model_validate(raw)

with FileManager(policy) as fm:
    fm.append_line("first event")

Unknown fields raise — the policy uses extra="forbid" so an operator's typo fails loudly rather than getting silently dropped.

JavaScript — @mmpworks/rollingfiles

Framework-neutral JS package. Does not write bytes to disk — node and browser environments use this to load / edit / save the policy that a .NET or Python writer then consumes.

Two ways to think about it:

  • Schema source of truth — POLICY_FIELDS, DEFAULTS, defaultPolicy(), isDefault() describe and produce the canonical in-memory shape.
  • Round-trip transformer — PolicyTransformer converts that shape to and from JSON or YAML text.

Install

npm install @mmpworks/rollingfiles

Load a policy from JSON or YAML

import { PolicyTransformer } from '@mmpworks/rollingfiles'

const transformer = new PolicyTransformer()

// JSON text in, validated data model out
const fromJson = transformer.fromJson(`
{
  "directory": "/var/log/myapp",
  "fileNameTemplate": "events-{date}",
  "interval": "Daily",
  "maxRetainedFiles": 30,
  "compression": "Gzip"
}
`)

// YAML text works the same way
const fromYaml = transformer.fromYaml(`
directory: /var/log/myapp
fileNameTemplate: events-{date}
interval: Daily
maxRetainedFiles: 30
compression: Gzip
`)

Serialise back to JSON or YAML

import { PolicyTransformer, defaultPolicy } from '@mmpworks/rollingfiles'

const policy = {
  ...defaultPolicy(),
  directory: '/var/log/myapp',
  fileNameTemplate: 'events-{date}',
  interval: 'Daily',
  maxRetainedFiles: 30,
  compression: 'Gzip',
}

const transformer = new PolicyTransformer()

const jsonText = transformer.toJson(policy)
const yamlText = transformer.toYaml(policy)

The transformer is stateless once constructed. Reuse the same instance for every round-trip in the process.

Manifest — operator-facing labels and tooltips

Every renderer that shows the policy to a human binds its labels, section names, and tooltips against the same PolicyManifest. That way the Vue editor, a future React form, a Blazor page, and a CLI help-text generator all render identical wording.

import { PolicyManifest } from '@mmpworks/rollingfiles'

const manifest = PolicyManifest.default()
// manifest.sectionFor('directory') → { id: 'where', label: 'Where & What', ... }
// manifest.fieldFor('compression') → { label: 'Compression', tooltip: '...', ... }

Vue — @mmpworks/rollingfiles-vue

Vue 3 + Vuetify 3 component set. The components bind to the same data model the JS transformer produces, so you can drop them into a config screen and round-trip to disk without writing any field plumbing.

vue and vuetify are peer dependencies — the consuming app provides them.

Install

npm install @mmpworks/rollingfiles-vue @mmpworks/rollingfiles

The editor — minimal wiring

<script setup>
import { ref } from 'vue'
import { FilesPolicyEditor } from '@mmpworks/rollingfiles-vue'
import { defaultPolicy } from '@mmpworks/rollingfiles'

// Source-of-truth policy the parent owns.
const policy = ref(defaultPolicy())

// The editor manages a local draft + dirty flag.
const draft = ref({})
const dirty = ref(false)

function onSave() {
  policy.value = draft.value
  // Persist policy.value somewhere — disk, settings store, API.
}

function onCancel() {
  // Re-assigning the same value re-seeds the draft and clears dirty.
  policy.value = { ...policy.value }
}
</script>

<template>
  <FilesPolicyEditor
    :policy="policy"
    v-model:draft="draft"
    v-model:dirty="dirty"
  />
  <button :disabled="!dirty" @click="onSave">Save</button>
  <button :disabled="!dirty" @click="onCancel">Cancel</button>
</template>

Wire in JSON / load / save

JsonPanel shows the live JSON next to the form (Ace-powered), and LoadSaveBar adds file-pick / download / clipboard controls that round-trip through PolicyTransformer.

<script setup>
import { ref } from 'vue'
import {
  FilesPolicyEditor,
  JsonPanel,
  LoadSaveBar,
} from '@mmpworks/rollingfiles-vue'
import { defaultPolicy } from '@mmpworks/rollingfiles'

const policy = ref(defaultPolicy())
const draft = ref({})
const dirty = ref(false)
</script>

<template>
  <LoadSaveBar v-model="policy" />
  <div class="grid">
    <FilesPolicyEditor
      :policy="policy"
      v-model:draft="draft"
      v-model:dirty="dirty"
    />
    <JsonPanel :value="draft" />
  </div>
</template>

The data flow when the user edits a field:

sequenceDiagram
    autonumber
    participant U as User
    participant E as FilesPolicyEditor
    participant P as Parent component
    participant J as JsonPanel
    participant T as PolicyTransformer

    U->>E: change a field
    E->>E: update local draft
    E->>P: emit update:draft + update:dirty
    P->>J: pass draft as :value
    J->>U: render JSON live
    U->>P: click Save
    P->>T: transformer.toJson(draft)
    T->>P: JSON text
    P->>P: persist (disk / API / settings)
Loading

Web — the playground

A static site that wires the JS + Vue packages together for interactive exploration. Deployed to GitHub Pages from release-web.yml when the workspace changes.

Run locally:

npm install
npm run -w @mmpworks/rollingfiles-web dev
# serves on http://localhost:5174

The playground is not published to npm. If your goal is to ship a config editor in your own product, depend on @mmpworks/rollingfiles-vue directly — the playground itself is about 200 LOC of glue around those components.

File lifecycle — what the writer actually does

The .NET and Python writers follow the same lifecycle. Understanding it makes the hook callbacks easier to place.

stateDiagram-v2
    [*] --> Active: AppendLine / AppendBytes
    Active --> Active: more appends
    Active --> Rolling: trigger fires<br/>(time / size / count)
    Rolling --> Sealed: atomic close + rename
    Sealed --> Compressed: background worker<br/>(if CompressionMode != None)
    Compressed --> Pruned: pruner runs<br/>(if any retention bound exceeded)
    Sealed --> Pruned: (no compression configured)
    Pruned --> [*]
    Active --> Sealed: process shutdown<br/>(via close / Dispose)
Loading

Hook callbacks fire at the boundaries — OnOpenActive when a fresh active file opens; OnPreRoll just before sealing; OnPostRoll just after sealing (compression hasn't run yet); OnPrune for each candidate deletion.

Cross-language guarantees

The five packages share one rule: the on-disk artifact is the contract. Anything you can write with the .NET package can be read with the Python package and vice versa. The spec/v1.md document is the authoritative source for:

  • The canonical in-memory shape (directory, fileNameTemplate, every optional field, defaults, value ranges)
  • The JSON wire format (camelCase keys, omit-when-default rule)
  • The YAML wire format
  • Versioning rules (additive v1.x vs structural v2)

If you're writing a tool that produces or consumes a FilesManagerPolicy outside of these five packages — for example a Go ingest agent, a Rust CLI, a Helm chart values shim — read spec/v1.md before you start. The packages here are reference implementations of that spec.