Dynamic Code Generation and Debugging

SkyWalking OAP server uses four Domain-Specific Languages (DSLs) to define observability logic: OAL (traces/mesh metrics), MAL (meter metrics), LAL (log analysis), and Hierarchy (service matching rules). These DSL scripts are compiled into JVM bytecode when the OAP server starts. The generated classes run in-process — there are no intermediate source files.

When a runtime error occurs inside these generated classes, the stack trace references class names and source locations that map back to the original DSL configuration files. This document explains how to dump the generated bytecode to disk for inspection and how to read the error messages.

DSL Configuration Files

All paths are relative to the OAP distribution root directory.

Dumping Generated Classes

Set the environment variable SW_DYNAMIC_CLASS_ENGINE_DEBUG to any non-empty value before starting the OAP server. All four DSL compilers check this variable and dump .class files to disk when it is set.

# Binary distribution
export SW_DYNAMIC_CLASS_ENGINE_DEBUG=Y
bin/oapService.sh

# Docker
docker run -e SW_DYNAMIC_CLASS_ENGINE_DEBUG=Y ... apache/skywalking-oap-server

# Kubernetes (in container env section)
env:
  - name: SW_DYNAMIC_CLASS_ENGINE_DEBUG
    value: "Y"

Output Directory Structure

The generated .class files are written to sibling directories next to oap-libs/:

Binary distribution (apache-skywalking-apm-bin/):

apache-skywalking-apm-bin/
├── config/                   ← DSL source scripts (*.oal, *.yaml, *.yml)
├── oap-libs/                 ← OAP server jars
├── oal-rt/                   ← Generated OAL classes
│   ├── metrics/              ←   e.g., ServiceRespTimeMetrics.class
│   ├── metrics/builder/      ←   e.g., ServiceRespTimeMetricsBuilder.class
│   └── dispatcher/           ←   e.g., ServiceDispatcher.class
├── mal-rt/                   ← Generated MAL classes (e.g., otel_rules_vm_L25_cpu_total_percentage.class, meter_analyzer_config_vm_L20_filter.class)
├── lal-rt/                   ← Generated LAL classes (e.g., default_L3_default.class)
└── hierarchy-rt/             ← Generated Hierarchy classes (e.g., hierarchy_definition_L88_name.class)

Docker (/skywalking/):

/skywalking/
├── config/
├── oap-libs/
├── oal-rt/
├── mal-rt/
├── lal-rt/
└── hierarchy-rt/

The OAL output directories are cleaned on each restart. MAL, LAL, and Hierarchy directories are created on demand if they don’t exist.

Inspecting Generated Classes

Use javap to decompile a generated .class file:

javap -v -p oal-rt/metrics/ServiceRespTimeMetrics.class

The output includes:

• SourceFile attribute — shows the DSL source file and the generated class name.
• LineNumberTable — maps bytecode offsets to statement numbers, used by the JVM in stack traces.
• LocalVariableTable — shows named local variables for readability.

Reading Error Stack Traces

When a runtime error occurs inside a generated class, the JVM prints a stack trace that combines the SourceFile attribute and the LineNumberTable. The format is:

at <package>.<ClassName>.<method>(SourceFile:LineNumber)

The .java generated source file is written ON DEMAND only — when SW_DYNAMIC_CLASS_ENGINE_DEBUG is set. Javassist compiles from an in-memory string, so a default deployment produces classes with no generated source file on disk at all. SourceFile therefore names the thing an operator can open — the rule file — and appends the generated class file name for anyone who did dump classes:

(<dsl-file-path>:<rule-line>)<GeneratedClassName>.java

Examples, all four DSLs:

(otel-rules/activemq/activemq-broker.yaml:32)otel_rules_activemq_activemq_broker_L32_service_meter.java
(lal/execution-basic.yaml:110)execution_basic_L110_if_else_if_warn.java
(core.oal:20)ServiceRespTimeMetrics.java
(hierarchy-definition.yml:12)hierarchy_definition_L12_service_rule.java

The path is complete and paste-able. The class name cannot substitute for it: name sanitising maps /, - and . all to _, so otel_rules_activemq_activemq_broker could be any of several paths, and the extension is dropped entirely.

A class shared by several rules names the file without a line — an OAL dispatcher handles every metric of one scope, so no single rule line would be true for it:

(core.oal)ServiceRespTimeDispatcher.java

Line numbers after the final : address the generated .java, and how precise they are depends on the DSL:

Every method’s line is located by searching the assembled generated source file text for its declaration, not by counting from the method above it — the envelope varies per rule, so a per-method constant would drift the moment codegen changes. One fixed offset does remain, DslGeneratedFileWriter.SOURCE_FILE_PREAMBLE_LINES: the licence header the writer itself prepends is the same for every file, so it is a property of the writer rather than of any rule.

Example Stack Trace

java.lang.ArithmeticException: / by zero
    at ...metrics.generated.ServiceRespTimeMetrics.id0((core.oal:20)ServiceRespTimeMetrics.java:9)
    at ...worker.MetricsStreamProcessor.in(MetricsStreamProcessor.java:...)
    ...

Reading this:

• (core.oal:20) — the rule is on line 20 of core.oal, under config/. This is the part to open.
• ServiceRespTimeMetrics.java — the generated class for metric ServiceRespTime; it exists on disk only when SW_DYNAMIC_CLASS_ENGINE_DEBUG is set
• :9 — the id0 method’s signature line in that generated source. OAL carries one entry per method, so this locates the METHOD, not the statement inside it

Format Per DSL

Notes:

• SourceFile names the RULE, then the generated class file. The rule file is the part to open: the .java is written only when SW_DYNAMIC_CLASS_ENGINE_DEBUG is set, so in a default deployment it does not exist on disk.
• The class name’s stem is the rule file’s path with the catalog kept only where the generated class’s package does not already imply it. MAL’s catalogs share one package and two of them ship a vm.yaml, so MAL keeps it; LAL has a single catalog and its own package, so lal/ appears in SourceFile but not in the name.
• The class name carries the same line as _L{lineNo}_, but it cannot substitute for the path — name sanitising maps /, - and . all to _ and drops the extension, so otel_rules_activemq_activemq_broker matches several distinct paths.
• A dispatcher is shared by every metric of its scope, so it names the file without a line rather than borrow one metric’s.
• A line that was expected but could not be resolved renders as _Lunknown_ in the class name and is omitted from SourceFile, so the failure stays visible instead of silently degrading.
• When no source information is available at all, the class name falls back to MalExpr_<N> / LalExpr_<N> / HierarchyRule_<N>.
• All four DSLs share one implementation of this — org.apache.skywalking.oap.server.core.dsl.DslSourceRef — which owns the class-name segment, the SourceFile value, the generated source file, the signature-line lookup and the LineNumberTable.

Generating All DSL Classes Offline

You can compile every DSL script from the source tree without starting the OAP server. This is useful for verifying that all scripts are syntactically valid after editing, or for batch-inspecting the generated bytecode with javap or an IDE decompiler.

The DSLClassGeneratorTest in the server-starter module compiles all OAL, MAL, LAL, and Hierarchy scripts and dumps the .class files to target/generated-dsl-classes/.

# From the project root (requires a prior build: ./mvnw -Pbackend install -Dmaven.test.skip)
./mvnw test -pl oap-server/server-starter \
  -Dtest="DSLClassGeneratorTest#generateAllDSLClasses" \
  -Dcheckstyle.skip

Output

oap-server/server-starter/target/generated-dsl-classes/
├── oal/              ← Metrics, MetricsBuilder, Dispatcher classes
├── mal/              ← MAL expression and filter classes
├── lal/              ← LAL expression classes
└── hierarchy/        ← Hierarchy rule classes

The test fails if any script cannot be compiled and prints the list of failures.

What It Covers

Common Error Patterns

OAL Compilation Failure

OAL compilation errors are logged at ERROR level during OAP startup:

ERROR o.a.s.o.v.g.OALClassGeneratorV2 - Can't generate method id for ServiceRespTimeMetrics.

This indicates that the generated Java source for the id method failed to compile. Check the OAL script syntax at the reported metric name.

MAL/LAL Runtime Error

MAL and LAL errors during metric processing are caught and logged per-expression:

ERROR o.a.s.o.m.a.v.MetricConvert - Analyze Analyzer{...} error
java.lang.NullPointerException
    at ...otel_rules_vm_L25_cpu_total_percentage.run((otel-rules/vm.yaml:25)otel_rules_vm_L25_cpu_total_percentage.java:5)

This tells you: the error is in otel-rules/vm.yaml, line 25, metric cpu_total_percentage, at statement 5 of the generated run() method. The parenthesised part is the rule coordinate the JVM prints verbatim from SourceFile — it is what to open, and it resolves whether or not the .java was dumped. The processing continues for other metrics — a single expression failure does not crash the server.

Hierarchy Compilation Failure

Hierarchy rule compilation errors are thrown at startup:

IllegalStateException: Failed to compile hierarchy rule: lower-short-name-remove-namespace,
    expression: { (u, l) -> { if (...) { ... } } }

Check the rule expression syntax in config/hierarchy-definition.yml under auto-matching-rules.