Support calculated attribute values #82
Conversation
Applies to default, minimum, maximum properties
mikee47
changed the title
|
A primary use case for calculated defaults is using different values for debug/release builds. It would therefore be appropriate to move the generated code from However, even with this the build system cannot track such dependencies without rebuilding it every time so ultimately it's going to be up to the developer to determine if the schema needs rebuilding. |
|
I'm not particularly keen on using Security: A simplified syntax would probably be better, so instead of |
|
let me see if I understand this correctly:
so for my initial issue, could I use "logEnabled": {
"type":"boolean",
"@default":"\"true\" if os.environ.get(\"SMING_RELEASE\") !=\"1\" else \"false\"",
"ctype":"telemetryLog",
}, |
|
regarding the security vs. flexibility thoughts: there's a ton of arbitrary code already being run anyway, the code will run with the user's privileges, so I don't necessarily see hightened security concerns unless a developer pushes malicious code to the repo and a user blindly builds this. |
Yep, or simplified: "@default": "os.environ['SMING_RELEASE'] != '1'"But I'm thinking this kind of syntax shouldn't be too hard to implement and would be more intuitive and meet most use cases: "@default": "$SMING_RELEASE != '1'"dbgen just needs to perform text substitution of |
|
thinking again about the use cases, it really comes down to "doing something with one or more environment variables" doesn't it, and "something" could be as simple as conditionals and basic arithmetic. Maybe having a very simple language that just exposes |
|
for readability, I would prefer passing the true/false return implicitly is efficient but a bit unintuitive (for me) to read |
|
So I've done a very simple regex substitution so we can write stuff like |
|
Actually, expression must be |
|
I'm slightly ashamed to admit I used Gemini for this, but this seems to be rather userful: #!/usr/bin/env python3
import ast
import operator as op
import math
import os
import sys
import re
class ConfigDBEvaluator:
def __init__(self):
self.functions = {
'pow2': lambda x: 2**x,
'log2': lambda x: math.log2(x),
'align_up': lambda v, a: (v + a - 1) & ~(a - 1),
'max': max,
'min': min
}
self.operators = {
ast.Add: op.add, ast.Sub: op.sub, ast.Mult: op.mul, ast.Div: op.truediv,
ast.Eq: op.eq, ast.NotEq: op.ne, ast.Gt: op.gt, ast.Lt: op.lt,
ast.GtE: op.ge, ast.LtE: op.le, ast.LShift: op.lshift, ast.RShift: op.rshift,
ast.BitAnd: op.and_, ast.BitOr: op.or_, ast.Invert: op.invert,
ast.And: lambda a, b: a and b, ast.Or: lambda a, b: a or b,
ast.Not: lambda x: not self._is_truthy(x),
ast.USub: op.neg,
ast.In: lambda a, b: a in b,
ast.NotIn: lambda a, b: a not in b
}
def _is_truthy(self, val):
"""Factually converts common C/Env strings to booleans."""
if isinstance(val, bool): return val
if isinstance(val, (int, float)): return val != 0
if isinstance(val, str):
if val.lower() in ('false', '0', 'no', 'off', ''): return False
return True
return bool(val)
def _eval(self, node):
if isinstance(node, ast.Constant):
return node.value
elif isinstance(node, ast.Name):
if node.id == 'True': return True
if node.id == 'False': return False
val = os.environ.get(node.id)
if val is None:
raise NameError(f"Environment variable '{node.id}' is not defined.")
try:
if val.lower().startswith('0x'): return int(val, 16)
return float(val) if '.' in val else int(val)
except ValueError:
return val
elif isinstance(node, ast.BinOp):
return self.operators[type(node.op)](self._eval(node.left), self._eval(node.right))
elif isinstance(node, ast.UnaryOp):
return self.operators[type(node.op)](self._eval(node.operand))
elif isinstance(node, ast.BoolOp):
values = [self._eval(v) for v in node.values]
result = values[0]
for next_val in values[1:]:
result = self.operators[type(node.op)](result, next_val)
return result
elif isinstance(node, ast.Compare):
left_val = self._eval(node.left)
right_val = self._eval(node.comparators[0])
op_type = type(node.ops[0])
if op_type not in self.operators:
raise TypeError(f"Unsupported comparison operator: {op_type.__name__}")
return self.operators[op_type](left_val, right_val)
elif isinstance(node, ast.IfExp):
# Use truthy helper for the condition
return self._eval(node.body) if self._is_truthy(self._eval(node.test)) else self._eval(node.orelse)
elif isinstance(node, ast.Call):
if not isinstance(node.func, ast.Name):
raise TypeError(f"Unsupported function call type: {type(node.func).__name__}")
func_name = node.func.id
if func_name in self.functions:
func = self.functions[func_name]
args = [self._eval(arg) for arg in node.args]
# Factual check for argument counts
import inspect
sig = inspect.signature(func)
try:
sig.bind(*args)
except TypeError:
raise ValueError(f"Function '{func_name}' expects {len(sig.parameters)} arguments (got {len(args)})")
return func(*args)
raise ValueError(f"Forbidden function call: {func_name}")
raise TypeError(f"Unsupported syntax: {type(node).__name__}")
def run(self, expr):
if not expr or not expr.strip(): return "Error: Empty expression"
processed = expr.replace("&&", " and ").replace("||", " or ")
processed = re.sub(r'!(?!=)', ' not ', processed)
try:
tree = ast.parse(processed.strip(), mode='eval')
return self._eval(tree.body)
except Exception as e:
return f"Error: {e}"
if __name__ == "__main__":
if len(sys.argv) < 2: sys.exit("Usage: ./ConfigDBDSL.py 'expression'")
print(ConfigDBEvaluator().run(sys.argv[1]))basically this implements a pretty flexible DSL interpreter that can evaluate conditions like or it also implements basic arithmetic, bit shifts, hex numbers, bit operations etc. #!/usr/bin/env python3
import os
from ConfigDBDSL import ConfigDBEvaluator
def run_full_suite():
evaluator = ConfigDBEvaluator()
# Environment Setup
os.environ.update({
"CPU_COUNT": "4",
"ARCH": "xtensa",
"CHIP": "esp32s3",
"RAM_SIZE": "0x80000", # 512KB
"DEBUG_LEVEL": "2",
"IS_RELEASE": "False",
"FEATURES": "wifi,ble,mesh"
})
# Format: (Expression, Expected, Category)
test_cases = [
# --- Arithmetic & Bitwise ---
("CPU_COUNT * 2", 8, "Basic Arithmetic"),
("RAM_SIZE / 1024", 512.0, "Hex Handling"),
("1 << CPU_COUNT", 16, "Bit Shift"),
("~0 & 0xFF", 255, "Bitwise Invert & AND"),
("align_up(15, 8)", 16, "Custom Function"),
# --- C-Style Logical Aliases ---
("True if CPU_COUNT == 4 && ARCH == 'xtensa' else False", True, "AND alias (&&)"),
("True if ARCH == 'arm' || ARCH == 'xtensa' else False", True, "OR alias (||)"),
("True if !(ARCH == 'arm') else False", True, "NOT alias (!)"),
("True if ARCH != 'arm' else False", True, "NOT EQUAL (!=)"),
# --- Substring & Membership ---
("'esp' in CHIP", True, "Substring match (in)"),
("'mesh' in FEATURES", True, "Env string search"),
("True if '32s' in CHIP && 'wifi' in FEATURES else False", True, "Nested Logic+Member"),
("'riscv' not in ARCH", True, "Negative Member"),
# --- Nested Ternary & Math ---
("pow2(CPU_COUNT) if CPU_COUNT > 2 else 0", 16, "Function in Ternary"),
("8 if CHIP == 'esp32' else (16 if CHIP == 'esp32s3' else 32)", 16, "Nested Else-If"),
("(1 if !IS_RELEASE else 0)", 1, "Bool Env + NOT (!)"),
# --- Error Handling ---
("UNDEFINED_VAR + 1", "Error: Environment variable 'UNDEFINED_VAR' is not defined.", "Missing Var"),
("pow2(2, 4, 6)", "Error: Function 'pow2' expects 1 arguments (got 3)", "Param Mismatch"),
# --- Code Injection (Security Audit) ---
("__import__('os').system('ls')", "Error: Unsupported function call type: Attribute", "Injection: RCE"),
("os.popen('whoami')", "Error: Unsupported function call type: Attribute", "Injection: Attribute"),
("open('/etc/passwd')", "Error: Forbidden function call: open", "Injection: File Access"),
("().__class__.__mro__", "Error: Unsupported syntax: Attribute", "Injection: Jailbreak"),
("1; import os", "Error: invalid syntax", "Injection: Multi-stmt"),
]
print(f"{'Category':<20} | {'Expression':<55} | {'Result'}")
print("-" * 110)
passed = 0
for expr, expected, cat in test_cases:
actual = evaluator.run(expr)
# Logic: If 'expected' is an error string, check if it's contained in the 'actual' result
if isinstance(expected, str) and expected.startswith("Error"):
success = expected in str(actual)
else:
success = (actual == expected)
if success:
passed += 1
status = "\033[92mPASS\033[0m"
else:
status = f"\033[91mFAIL\033[0m (Got: {actual})"
print(f"{cat:<20} | {expr:<55} | {status}")
print("-" * 110)
print(f"Final Score: {passed}/{len(test_cases)}")
if __name__ == "__main__":
run_full_suite()This should eliminate the code injection risk and still be extremely powerful. |
|
Regardless of how it was done, that is a nice solution. I can see immediately there are other tools in Sming which would benefit from this (IFS fsbuild, Storage hwconfig, Graphics resource compiler). I propose adding this to Sming so it's generally available. There is the |
|
certainly, I wasn't sure what you think of using AI code so I didn't just want to sneak it in. |
It's fine, though a bit weird it's been written as a class. Nothing a bit of refactoring and tidying wouldn't sort! |
|
...although now I think about it, a class can be more easily extended for specialised use, such as resolving variables from other places. |
|
Maybe let's check for existing python packages which might do most of the work for us. For example https://pypi.org/project/expression-parser/ |
|
why not re-invent the wheel if we can ;-) but more seriously, would we stand to gain much from this? at a quick look, we'd probably need a bit of glue code (reading in the environment, setting the functions) to implement Also, if I understand |
|
I have created PR #2986 but will be happy any way this goes |
|
btw, thinking about other uses: |
This PR is a suggestion for how #75 might be approached. The idea is to interpret a
@prefix in an attribute name as a calculated value. The@prefix was chosen arbitrarily.An example is included in the test application:
During loading, the attribute value is evaluated in python and the result stored in
default.Note: "@default" and "default" (for example) are considered the same attribute and using both in the same property list will result in an error.
To test this, build and run as follows:
make clean SIMPLE_STRING="donkey2" make -j runThe test application now fails as the value has changed - "donkey" is expected.
integer expressions
JSON does not support extended number formats, such as
0x12, so this mechanism enables calculated values for numeric properties by adding the@prefix and using a string value: