Pipeline 2 jhyphen crashes

[PaulRambags]

Expected Behavior

I expect the Pipeline 2 with our Dedicon module to run without crashing.

Actual Behavior

The Pipeline 2 crashes. Sometimes there is a stack trace in daisy-pipeline.log. Also sometimes the JVM generates an exception report because of an EXCEPTION_ACCESS_VIOLATION which has a stack trace. Often the Pipeline just stops working without any stack trace. When I inspect the stack trace, the problem seems to be in jhyphen. Whether the pipeline crashes depends on various aspects: The version of the JVM that is used, the input (the DTBook), and how long the Pipeline has already been running / how many books it has processed since the last restart.

Steps to Reproduce

Provide a list of steps taken to reproduce the problem.

1. Use pipeline-assembly commit ed3ac2d (‘Release Notes of 1.14.14’) to create a version with our Dedicon module. To include this module, add the following in pom.xml at line 2519: <artifactItem><groupId>nl.dedicon.pipeline</groupId><artifactId>braille-module</artifactId><version>2.0.1</version></artifactItem>
2. Run make.exe zip-win in a command prompt. This will create a assembly-1.14.15-SNAPSHOT-win.zip file in the target folder.
3. Unzip that file. This will give a daisy-pipeline folder.
4. Run pipeline2.bat in the bin subfolder to start the Pipeline.
5. Let the Pipeline process a DTBook. I have several examples of books that result in crashes, but I can't publish those publicly. Processing can be done via the Web UI or by invoking dp2.exe dedicon:dtbook-to-pef --source {source.xml} --output {target-folder} in the cli subfolder.

Details

Crashes started to occur more frequently after I added the following in the style sheet:

/* --------------------------------------------------------------------------------
 * Use Dutch as language for interpolation
 *
 * If $braille-notation-code == "nl", capital signs etc. are formatted in Dutch.
 * Hyphenation rules remain unchanged, i.e. hyphenation follows the language of
 * the dtbook.
 */
@if $braille-notation-code == "nl" {
    @text-transform {
        system: braille;
        language: nl;
        type: literary;
    }
}

Environment

• Operating system: Windows Server 2022 Standard version 21H2, 64-bit
• DAISY Pipeline 2 version: 1.14.14
• Interface: command line interface, and web UI
• JVM: Java(TM) SE Runtime Environment (build 1.8.0_411-b09)

Logs

https://gist.github.com/PaulRambags/e9e0b00638a026735a48d101d93a76c2

[NPavie]

Hi @PaulRambags ,

If it is possible, in the pipeline home directory, can you look at the system/common/ directory and give us the version of the org.daisy.bindings.jhyphen jar file in it ?

[NPavie]

@bertfrees reported that you still encountered the issue with the fix we added in last august in org.daisy.bindings.jhyphen-1.0.3 (sorry about that)

Can you provide additionnal information / updates on the pipeline version tested, and if it is tested through the DAISY pipeline app or through the pipeline-assembly (and if it is the latter, which commit / tag is targeted for the pipeline-assembly build) ?

[PaulRambags]

Hi Nicolas, org.daisy.bindings.jhyphen-1.0.3.jar org.daisy.pipeline.modules.braille.libhyphen-utils-3.2.1-windows.jar org.daisy.pipeline.modules.braille.libhyphen-utils-3.4.0.jar Greetings, Paul Rambags

[bertfrees]

After a lot of debugging I have localised a bug in libhyphen that was causing crashes on Windows on my end. By adding a workaround in jhyphen, I managed to fix the issue.

The bug is on this line:

char * nhy = (char *) strstr(word, nh);

strstr expects a null-terminated string, but word is not null-terminated. There is also a strnstr version of this function, but because it is probably not portable, I opted for a workaround, namely to pass null-terminated strings from jhyphen.

[PaulRambags]

After a lot of debugging I have localised a bug in libhyphen that was causing crashes on Windows on my end. By adding a workaround in jhyphen, I managed to fix the issue.

The bug is on this line:

char * nhy = (char *) strstr(word, nh);
strstr expects a null-terminated string, but word is not null-terminated. There is also a strnstr version of this function, but because it is probably not portable, I opted for a workaround, namely to pass null-terminated strings from jhyphen.

@bertfrees sent us a snapshot version jhyphen-1.0.4-SNAPSHOT.jar with a fix for the crashes. We've tested this and it seems to work! 😀