Skip to content
BranchesTags

Repository files navigation

golib

golib is a golang utility library containing common tool functions and components summarized from personal project development experience.

Components:

  • biz Business components
  • codegen Code generation tools
  • gconc Concurrent task pool component
  • configkv Configuration management component
  • dbaccess Database client components (supports MySQL, Redis, Elasticsearch)
  • dict Data dictionary component (type + code value, tree hierarchy)
  • distlock Distributed lock component (non-reentrant)
  • excel Excel read/write component
  • gast AST syntax tree tool
  • gauth Authentication component (includes jwtauth)
  • gcrypto Encryption/decryption component
  • gerror Error handling component
  • glog Logging component
  • gllm LLM access configuration layer (config → eino model)
  • gtrace OpenTelemetry Trace initialization component
  • gtree Tree structure construction tool
  • gutil Common utility functions collection
  • task Task scheduling components
  • protocol Protocol components (includes ghttp, gresty)
  • ratelimit Rate limiting component
  • storage Unified object storage component (supports S3, MinIO, OSS, COS, TOS)

Installation

go get github.com/morehao/golib

Running Tests

Some tests require real database connections (MySQL, PostgreSQL, Redis, Elasticsearch). To avoid hardcoding credentials in test code, copy .env.example to .env and configure your local connection settings:

cp .env.example .env
# edit .env with your local credentials

# Run all tests (auto-loads .env via env_test.go)
go test ./...

# Run sub-package tests (inject env manually)
source .env && go test ./codegen/...

Built-in tables: migration convention

configkv, dict, filestore, task/gcron and task/gasync own their own tables and follow one shared convention:

  1. Migrate by default. Constructors (New / Init / NewServer) run an idempotent AutoMigrate, so a caller needs no extra step. Note that GORM's generated DDL has no IF NOT EXISTS, so replicas starting cold at the same time can race and one of them may fail its first boot — use WithoutAutoMigrate() if that matters (or run the DDL from the release pipeline).
  2. WithoutAutoMigrate() when you own the DDL. The switch is a constructor option (functional options), so a shared database — or a runtime account without DDL privileges — can turn the implicit migration off and let the release pipeline run dict.Migrate(db) / configkv.Migrate(db) / filestore.Migrate(db) / gcron.AutoMigrate(db) / gasync.AutoMigrate(db) with a dedicated account. The explicit entry always runs, regardless of the switch.
  3. Every package README ships complete MySQL and PostgreSQL DDL, generated from the gorm tags by internal/ddlgen and asserted statement-by-statement in tests, so it cannot drift from the models.

GORM's own advice is "use versioned migrations in production, don't rely on AutoMigrate". That advice targets tables an application owns. This library is a reusable component owning a few fixed tables whose schema is determined by the component version, so it chooses "migrate by default + an opt-out option" and leaves the "should the app touch DDL" decision to the deployment. The precedent is casbin-gorm-adapter's TurnOffAutoMigrate(db).

The .env file is gitignored and will not be committed. If no environment variables are set, tests fall back to default values (127.0.0.1 with password 123456), so CI pipelines work without any extra setup.

For VS Code, configure .vscode/settings.json to inject environment variables:

{
  "go.testEnvVars": {
    "MYSQL_DSN": "root:123456@tcp(127.0.0.1:3306)/demo?charset=utf8mb4&parseTime=True",
    "REDIS_ADDR": "127.0.0.1:6379",
    "REDIS_PASSWORD": "123456"
  }
}

Test-specific environment variables:

Variable Description Default
MYSQL_DSN MySQL DSN for codegen tests root:123456@tcp(127.0.0.1:3306)/demo?...
POSTGRES_DSN PostgreSQL DSN for codegen tests host=127.0.0.1 user=postgres password=123456...
REDIS_ADDR Redis address 127.0.0.1:6379
REDIS_PASSWORD Redis password 123456
ELASTICSEARCH_ADDR Elasticsearch address http://localhost:9200

Components

biz

Overview

biz is a business component package providing commonly used infrastructure components for business development.

Sub-components

  • gcontext: Context utilities: request ID, user ID, tenant ID and other context key-value definitions, formatting, plus explicit tenant scope (CurrentScope/ExplicitScope/AllScope) with TenantScopeFilter as the single translator to data-layer filtering
  • gobject: Common business objects, including user authentication info (UserClaims), operator info (OperatorBaseInfo), pagination query (PageQuery)
  • gconstant: Business constant definitions, including error codes (100000 series), API versions, etc.
  • gserver: Gin server related. RouterGroups is the unique top-level route group factory (path /v1/{app}), auto-mounting otelgin & access-log middleware; business modules register routes via Register(group, ...). Routing style spec: docs/router-style.md
  • gmiddleware: Gin middleware, including JWT authentication, CORS, access logging, Token blacklist
  • gormplugin: GORM multi-tenant plugin: injects a scope condition into SELECT/UPDATE/DELETE, resolves the scope with a three-state Resolver (unset / all-tenants / by-value) and supports fail-closed MissingScope
  • genericdao: Generic DAO,封装基础的增删改查操作
  • testkit: Testing toolkit, supporting test initializer and context building

Features

  • Business scenario-oriented, ready to use
  • Unified error code specification
  • Integrated JWT authentication and multi-tenant support

codegen

Overview

codegen is a code generation tool that reads database table structures and supports generating basic CRUD code, including router, controller, service, dto, model, errorCode, etc. The generated router layer registers RESTful CRUD routes (see docs/router-style.md).

Features

  • Supports MySQL database
  • Supports PostgreSQL database
  • Supports template customization and template parameter customization
  • Supports code generation based on templates

Usage

For usage examples, refer to codegen unit tests

gconc

Overview

gconc is a unified concurrent task pool built on fixed workers and a buffered task queue.

Component

Provides a single *Pool type with:

  • non-blocking / timeout / blocking task submission
  • graceful & immediate shutdown
  • error collection & callback
  • panic-safe worker pool
  • runtime stats

Features

  • Flexible concurrency control
  • Task queue management
  • Graceful shutdown and error collection
  • Panic-safe worker pool
  • Thread-safe

Usage

For usage examples, refer to gconc usage

configkv

Overview

configkv is a configuration management component based on database key-value storage, supporting multiple data types and encryption.

Features

  • Supports json/toml/yaml/string/int/bool/float types
  • Supports encrypted storage
  • Based on GORM

dbaccess

Overview

dbaccess is a database client component collection providing encapsulation and connection management for multiple databases.

Sub-components

  • dbgorm: MySQL/PostgreSQL database client, based on GORM
  • dbredis: Redis client, based on go-redis
  • dbes: Elasticsearch client, based on official client

Features

  • Unified configuration interface
  • Integrated logging
  • Connection pool configuration support
  • Timeout control support

Usage

For usage examples, refer to dbaccess usage

dict

Overview

dict is a general-purpose data dictionary component: a type + code-value two-level model with an optional tree hierarchy and type-level / item-level extra JSON. Dictionaries become operational data instead of enums hardcoded in every service.

Features

  • Types are identified by a globally unique code, and the type table carries no grouping / namespace column at all (same as RuoYi, yudao and JeecgBoot) — in a shared database the code namespace is therefore shared, and cross-module naming relies on convention
  • Items reference their type by the natural key type_code and keep a materialized path + level, so a whole type is read in one query and assembled into a tree in memory
  • Fail-closed reads with distinguishable sentinel errors (ErrTypeNotFound / ErrTypeDisabled / ErrItemNotFound / ErrItemDisabled)
  • BatchExists validates up to 1000 code values in a single query
  • No built-in cache (every read hits the database), but a Source seam is kept for a cache decorator
  • Hard delete, a single write entry point (AdminAPI) that maintains every tree invariant, plus CheckIntegrity for drift detection
  • Tables are meant to be shared by several services in one database: dict.New auto-migrates by default (idempotent); for a shared database, or when the runtime account has no DDL privileges, pass dict.WithoutAutoMigrate() and let the release pipeline run dict.Migrate(db) instead. Since code is globally unique, each service must check naming before onboarding

Usage

For usage examples, refer to dict usage

distlock

Overview

distlock is a distributed lock component based on Redis, using redsync algorithm, supporting automatic renewal.

Features

  • Redis-based distributed lock (single or multi-node quorum)
  • Automatic renewal (lock keepalive) with jitter, lock-loss notification via Lost()
  • Non-reentrant

Usage

// 1. Create a lock factory (one per process; pass multiple clients for multi-node quorum)
factory := distlock.NewRedisStorage(redisClient)

// 2. Create a lock instance per key/TTL
lock, err := distlock.NewDistLock(factory, &distlock.Config{
	Key:         "order:pay:10086",
	TTL:         30 * time.Second,
	AutoRenewal: true,
})
if err != nil {
	// invalid config: empty Key or TTL <= 0
}

// 3. Acquire (non-blocking) -> critical section -> release
if ok, err := lock.Lock(ctx); err != nil {
	// storage failure
} else if !ok {
	// lock not acquired (normal contention, retry later)
} else {
	defer lock.Unlock(context.Background())
	// critical section...
	// when auto-renewal fails (lock lost), Lost() is closed — abort ASAP:
	// <-lock.Lost()
}

excel

Overview

excel is a simple wrapper around excelize, supporting convenient Excel file read/write through structs.

Both reading and writing Excel require defining a struct, with struct fields specifying Excel-related information through tags (ex).

Features

  • Define Excel column mapping through struct tags
  • Support reading and writing Excel files
  • Support data validation based on validator

Usage

For usage examples, refer to excel usage

gast

Overview

gast is a Go AST syntax tree operation tool, supporting AST analysis and code generation.

Features

  • Support function/method lookup
  • Support interface method addition
  • Support constant addition
  • Syntax tree traversal and manipulation

gauth

Overview

gauth is an authentication component containing JWT authentication capabilities.

Sub-components

  • jwtauth: Generic JWT signing and parsing, supports HS256 algorithm, supports renewal

Features

  • Generic JWT signing and parsing
  • Token renewal support
  • Token blacklist support

Usage

For usage examples, refer to jwtauth usage

gcrypto

Overview

gcrypto is an encryption/decryption component providing common symmetric and asymmetric encryption functions.

Sub-components

  • aes: Supports AES-128/192/256, GCM mode (recommended) and CBC mode
  • rsa: Supports encryption, decryption, signing, verification, PEM format keys
  • bcrypt: Password hashing and verification

Features

  • Environment variable configuration for keys
  • GCM mode provides authenticated encryption
  • RSA supports multiple padding modes

Usage

For usage examples, refer to gcrypto usage

gerror

Overview

gerror is an error handling component providing business error code encapsulation, supporting error chains and call stacks.

Features

  • Supports errors.Is/As
  • Error chain wrapping
  • Call stack recording
  • Business error code specification

glog

Overview

glog is a logging component exposing a unified Logger interface with pluggable drivers (zap / log/slog), providing high-performance structured logging.

Features

  • Pluggable drivers: zap (default) and log/slog, sharing one API and config
  • Console/File output with daily directories and size/backup/age rotation
  • Dual-file mode per file writer: _full (all levels) and _wf (warn and above)
  • Automatic context field extraction: OTel trace fields and custom ExtraKeys
  • Structured logging (Infow/Warnw/...)
  • Desensitization hooks: WithFieldHookFunc (fields) and WithMessageHookFunc (message)
  • High-performance log writing

Usage

See glog usage

gtrace

Overview

gtrace is an OpenTelemetry Trace initialization component supporting distributed tracing.

Features

  • OTLP gRPC/HTTP export support
  • Exporter disable mechanism
  • Integrated zap logging

Usage

For usage examples, refer to gtrace usage

gtree

Overview

gtree is a tree structure construction tool, a generic tree data structure building library supporting building trees from node lists.

Features

  • Provides TreeNode interface, only need to implement GetKey(), GetParentKey(), IsRoot() methods
  • Orphan node handling (ignore, promote to root, error)
  • Circular reference detection
  • Node sorting (ID, Name, Order or multi-level combination)
  • Pre-order traversal and level-order traversal

gutil

Overview

gutil is a collection of common utility functions providing commonly used tool functions during development.

Sub-components

  • Random number generation
  • String processing
  • Date/time operations
  • Type conversion
  • Slice/Map operations
  • File processing

task

Overview

task is a task scheduling component package containing cron and async task sub-packages, both persisting execution records via GORM and integrating glog logging and gtrace distributed tracing.

Sub-components

  • gcron: Cron tasks, based on robfig/cron/v3, supporting second-level cron, multi-instance distributed lock mutual exclusion, and execution record persistence
  • gasync: Async tasks, based on hibiken/asynq, supporting retry, timeout, delay, priority queues, execution record persistence, and cross-process trace propagation

Features

  • Second-level cron expressions and custom timezone
  • Multi-instance distributed lock mutual exclusion (with optional auto-renewal)
  • Per-task execution timeout and in-process overlap prevention (gcron)
  • Retry, timeout, retention, and multi-queue priority
  • Automatic execution record persistence (idempotent re-registration on restart)
  • Automatic TraceID/RequestID/RunID injection and logging (string primary key as task/run identifier)
  • Cross-process trace propagation
  • Graceful shutdown: Client.Close, Server.ShutdownContext (gasync), Scheduler.Stop(ctx) (gcron)

Usage

For usage examples, refer to task usage

protocol

Overview

protocol is a protocol-related component collection providing HTTP client encapsulation.

Sub-components

  • ghttp: Enhanced HTTP client, supports struct mapping, connection pool, smart retry and other features
  • gresty: HTTP client wrapper based on Resty, supports SSE (Server-Sent Events)

Features

  • Struct automatic mapping support
  • Connection pool optimization
  • Smart retry mechanism (no retry for 4xx, retry for 5xx)
  • SSE long connection support
  • Rich configuration options

Usage

For usage examples, refer to ghttp usage

storage

Overview

storage is a unified object storage component supporting multiple cloud providers with a consistent API.

Supported Providers

  • AWS S3
  • MinIO
  • Alibaba Cloud OSS
  • Tencent Cloud COS
  • Volcano Engine TOS
  • Qiniu Cloud Kodo

Features

  • Unified API across all providers
  • Multipart upload support (resumable local sessions with TTL reclamation)
  • Presigned URL generation (GET/PUT/PUT part; HMAC-signed token bound to bucket/key/op)
  • Object listing with prefix/delimiter and continuation-token pagination
  • Batch operations (delete, copy)
  • URI + PathBuilder helpers for standardized resource identifiers and public URLs
  • Local filesystem driver so a business service can act as the object storage itself

Usage

For usage examples, see storage README.

ratelimit

Overview

ratelimit is a rate limiting component: distributed rate limiting on Redis (redis_rate GCRA token bucket), with automatic degradation to a local token bucket when Redis is unavailable.

Features

  • Redis rate limiting (go-redis-rate, GCRA token bucket)
  • Automatic fail-over to in-process rate limiting (fail-open) when Redis is down, with exponential-backoff probing and automatic switch-back on recovery
  • Degradation/recovery events can be logged via WithLogger
  • Close() releases background goroutines

Notes

  • During fail-over, each process uses its own local limiter; in multi-instance deployments the aggregate limit is roughly instances × configured rate, and quotas are not shared across instances
  • Rate/Burst/Period/CleanupInterval must be positive, otherwise NewLimiter returns an error

gllm

Overview

gllm is the LLM access configuration layer: it resolves a config value object into an eino model.ToolCallingChatModel. It answers which provider, which model, what happens on failure — not how to talk to a vendor, which is already covered by eino-ext components.

Features

  • Value-object config with frozen field names (providers / models / allow_degraded), no config-file dependency
  • Call sites name the model directly: the models key is the model name, so no invented intermediate layer sits between provider and model
  • Startup-time validation via Resolve — config errors surface before the first call, not on it
  • Explicit, detectable degradation: a missing API key falls back to a built-in fake model with Model.Degraded = true instead of failing startup
  • Error classification into a reserved code range (120000-120099) with a retryability matrix
  • Driver registry with capability declarations; the core depends on eino only, never on eino-ext

Supported Drivers

  • openai — OpenAI and every OpenAI-compatible endpoint (DeepSeek, DashScope compatible mode, Ark, vLLM, Ollama, …); see gllm/driver/openai
  • fake — built-in fallback, auto-registered

Notes

  • No vendor protocol implementations; protocol lives in eino-ext, drivers only wire and register
  • No retry loop inside the library — it exposes Retryable plus recommended backoff constants so the caller or an eino callback owns retries
  • base_url is passed through verbatim — write it exactly as the vendor documents it (OpenAI's includes /v1, DeepSeek's does not)
  • Adding a driver is three things: implement gllm.Factory, call gllm.Register in init(), optionally declare capabilities / WithNoAuth

Usage

For usage examples, see gllm README. For a step-by-step adoption guide, see gllm integration guide.

About

This is a collection of tools and components summarized during the process of working with and learning Go.

Resources

Readme
MIT license
Activity

Stars

3 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages