PostgreSQL Seg Fault

[corbettht]

PostgreSQL version: PostgreSQL 9.5.3 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
Host OS: Ubuntu 14.04 LTS
CUDA: 7.5
GPU: NVIDIA Tesla K40c

Description: Postgres server crash, possibly pg_strom related.
Server back-end was killed by the kernel with signal 11 during data warehouse operations. 22 connections were active at the time of the crash.

Postgres log states the following:
2016-06-13 22:12:55 UTC LOG: server process (PID 18989) was terminated by signal 11$ Segmentation fault
2016-06-13 22:12:55 UTC DETAIL: Failed process was running: SELECT widget_property FROM widgets WHERE widget_id = 'camera_widget'
2016-06-13 22:12:55 UTC LOG: terminating any other active server processes
2016-06-13 22:12:55 UTC WARNING: terminating connection because of crash of another server process
2016-06-13 22:12:55 UTC DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2016-06-13 22:12:55 UTC HINT: In a moment you should be able to reconnect to the dat abase and repeat your command.
2016-06-13 22:12:55 UTC WARNING: terminating connection because of crash of another server process
...
2016-06-13 22:12:55 UTC LOG: all server processes terminated; reinitializing
2016-06-13 22:12:56 UTC LOG: database system was interrupted; last known up at 2016- 06-13 22:01:37 UTC
2016-06-13 22:13:00 UTC LOG: database system was not properly shut down; automatic r ecovery in progress
2016-06-13 22:13:00 UTC LOG: redo starts at E0/696B91E0
2016-06-13 22:13:10 UTC FATAL: the database system is in recovery mode
2016-06-13 22:13:34 UTC LOG: invalid record length at E2/F12E3F58
2016-06-13 22:13:34 UTC LOG: redo done at E2/F12E3F30
2016-06-13 22:13:34 UTC LOG: last completed transaction was at log time 2016-06-13 2 2:12:55.335794+00
2016-06-13 22:14:38 UTC LOG: MultiXact member wraparound protections are now enabled
2016-06-13 22:14:38 UTC LOG: database system is ready to acceptconnections``

System log contains the following lines:
[698383.006205] postgres[23372]: segfault at 10 ip 00007f98d72fe91f sp 00007ffd04d60cb0 error 4 in pg_strom.so[7f98d72d0000+b8000]
[961079.403648] traps: postgres[18989] general protection ip:7f98dd44c8c7 sp:7ffd04d60b50 error:0 in postgres[7f98dd371000+5f2000]
Thanks for your time - I appreciate any guidance you might have.
Please let me know if there's any more information I can provide that would be helpful.

Thanks,

[kaigai]

Thanks for your report.

Is it the problematic query when you saw the system crash?
SELECT widget_property FROM widgets WHERE widget_id = 'camera_widget'

Unfortunately, the log message didn't talk enough to identify the problem.
Could you collect a back trace from the core dump?

If "ulimit -c" is configured correctly, you can see a core dump file at $PGDATA directory.
gdb -c <path to core> -f /usr/local/pgsql/bin/postgres -f /usr/local/pgsql/lib/pg_strom.so
then, bt shows the call trace.

Also, is this problem reproducible on no concurrent task execution?
The current version of PG-Strom is not mature for resource handling when multiple sessions work concurrently.

Thanks,