Skip to content

DB2 - XA resource prepare for XID - the XA resource has become unavailable raised -7 #240

Description

@hci2

Describe the bug
It seems to be quite similar to this issue: #38 and StackOverflow issue: https://stackoverflow.com/questions/49392568/xa-resource-has-become-unavailable-xid-raised-7. So after 9 - 48 hours after starting the application and a long-running distributed transaction, the following exception appears:

2025-07-07 22:34:12.228 DEBUG 26930 --- [nio-8080-exec-1] o.s.t.jta.JtaTransactionManager          : Initiating transaction commit
2025-07-08 06:54:28.012 ERROR 26930 --- [nio-8080-exec-8] c.a.datasource.xa.XAResourceTransaction  : XA resource 'lfvXaDataSource': prepare for XID '6C66722D32363933305F746D313735313931373339393136353030303037:6C66722D32363933305F746D39' raised -7: the XA resource has become unavailable
com.ibm.db2.jcc.am.XaException: [jcc][t4][10401][12066][4.32.28] XA exception: XAER_RMFAIL : [jcc][t4][2030][11211][4.32.28] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: Reply.fill() - insufficient data (-1).  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001 ERRORCODE=-4228, SQLSTATE=null
	at com.ibm.db2.jcc.am.b7.c(b7.java:561)
	at com.ibm.db2.jcc.am.b7.c(b7.java:590)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:2118)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:1028)
	at com.ibm.db2.jcc.t4.a4.b(a4.java:954)
	at com.ibm.db2.jcc.t4.a4.b(a4.java:949)
	at com.ibm.db2.jcc.t4.a4.prepare(a4.java:938)
	at com.atomikos.datasource.xa.XAResourceTransaction.prepare(XAResourceTransaction.java:384)
	at com.atomikos.icatch.imp.PrepareMessage.send(PrepareMessage.java:40)
	at com.atomikos.icatch.imp.PrepareMessage.send(PrepareMessage.java:19)
	at com.atomikos.icatch.imp.PropagationMessage.submit(PropagationMessage.java:67)
	at com.atomikos.icatch.imp.Propagator$PropagatorThread.run(Propagator.java:63)
	at com.atomikos.icatch.imp.Propagator.submitPropagationMessage(Propagator.java:42)
	at com.atomikos.icatch.imp.ActiveStateHandler.prepare(ActiveStateHandler.java:171)
	at com.atomikos.icatch.imp.CoordinatorImp.prepare(CoordinatorImp.java:523)
	at com.atomikos.icatch.imp.CoordinatorImp.terminate(CoordinatorImp.java:687)
	at com.atomikos.icatch.imp.CompositeTransactionImp.commit(CompositeTransactionImp.java:282)
	at com.atomikos.icatch.jta.TransactionImp.commit(TransactionImp.java:172)
	at com.atomikos.icatch.jta.TransactionManagerImp.commit(TransactionManagerImp.java:414)
	at com.atomikos.icatch.jta.UserTransactionManager.commit(UserTransactionManager.java:159)
	at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1035)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at at.statistik.lfr.web.extern.AmaResource$$EnhancerBySpringCGLIB$$8a41f5bf.einspielung(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at at.statistik.entis.angular.filter.CspFilter.handle(CspFilter.java:42)
	at at.statistik.entis.angular.filter.CspFilter.doFilter(CspFilter.java:34)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at at.statistik.lfr.config.CorsConfig.doFilter(CorsConfig.java:34)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:355)
	at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:54)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
	at psiprobe.Tomcat90AgentValve.invoke(Tomcat90AgentValve.java:35)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:388)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:936)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.lang.Thread.run(Thread.java:750)
Caused by: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][4.32.28] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: Reply.fill() - insufficient data (-1).  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
	at com.ibm.db2.jcc.am.b7.a(b7.java:338)
	at com.ibm.db2.jcc.t4.a.a(a.java:577)
	at com.ibm.db2.jcc.t4.a.a(a.java:561)
	at com.ibm.db2.jcc.t4.a.a(a.java:556)
	at com.ibm.db2.jcc.t4.y.b(y.java:315)
	at com.ibm.db2.jcc.t4.y.c(y.java:342)
	at com.ibm.db2.jcc.t4.y.c(y.java:455)
	at com.ibm.db2.jcc.t4.y.v(y.java:1230)
	at com.ibm.db2.jcc.t4.ad.c(ad.java:201)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:2527)
	at com.ibm.db2.jcc.t4.a4.a(a4.java:1012)
	... 111 common frames omitted

It can only be resolved after restarting the whole application. After the first occurrence, it seems to persist. The underlying error of DB2 -4499 is described in IBM Troubleshooting and looks like the database connection was not available. But I checked the database with a db explorer tool and can access it. So I expect that the Atomikos database pooling should do a reconnect like HikariCP in Spring does.

To Reproduce
Steps to reproduce the behavior:

  1. Restart application
  2. Start a long running distributed transaction
  3. Wait and see the error log

Additional context

General setup (used multiple times for different datasources):

@Slf4j
@EnableTransactionManagement
@Configuration
@EnableConfigurationProperties({AtomikosProperties.class, JtaProperties.class})
public class JtaTransactionManagerConfig {

    @Bean(initMethod = "init", destroyMethod = "shutdownWait")
    UserTransactionServiceImp userTransactionService(
            AtomikosProperties atomikosProperties, JtaProperties jtaProperties) {
        Properties properties = new Properties(atomikosProperties.asProperties());
        // This allows you to override spring.jta.atomikos.properties.*
        if (StringUtils.hasText(jtaProperties.getTransactionManagerId())) {
            properties.setProperty("com.atomikos.icatch.tm_unique_name", jtaProperties.getTransactionManagerId());
        }
        if (StringUtils.hasText(jtaProperties.getLogDir())) {
            properties.setProperty("com.atomikos.icatch.log_base_dir", jtaProperties.getLogDir());
        }
        return new UserTransactionServiceImp(properties);
    }

    @Bean(initMethod = "init", destroyMethod = "close")
    @DependsOn("userTransactionService")
    public UserTransactionManager userTransactionManager() {
        UserTransactionManager userTransactionManager = new UserTransactionManager();
        userTransactionManager.setStartupTransactionService(false);
        userTransactionManager.setForceShutdown(true);
        return userTransactionManager;
    }

    @Bean("jtaTransactionManager")
    public JtaTransactionManager jtaTransactionManager(final UserTransactionManager userTransactionManager) {
        JtaTransactionManager jtaTransactionManager = new JtaTransactionManager();
        jtaTransactionManager.setUserTransaction(userTransactionManager); //Bean UserTransaction is not used
        jtaTransactionManager.setTransactionManager(userTransactionManager);
        jtaTransactionManager.setAllowCustomIsolationLevels(true);
        return jtaTransactionManager;
    }

    @Bean
    @Primary
    PlatformTransactionManager transactionManager(final UserTransactionManager userTransactionManager) {
        //Bean UserTransaction is not used
        return new JtaTransactionManager(userTransactionManager, userTransactionManager);
    }

Failing datasource - XA resource:

@Bean
	@Primary
	LocalContainerEntityManagerFactoryBean lfvEntityManagerFactory() {
		HibernateJpaVendorAdapter jpaVendorAdapter = new HibernateJpaVendorAdapter();
		jpaVendorAdapter.setGenerateDdl(false);

		Properties props = new Properties();
		props.put(AvailableSettings.DIALECT, "org.hibernate.dialect.DB2Dialect");
		props.put(AvailableSettings.FORMAT_SQL, config.getPropertiesHibernate().getOrDefault("format_sql", "false"));
		props.put(AvailableSettings.JPA_TRANSACTION_TYPE, "jta");

		LocalContainerEntityManagerFactoryBean factoryBean = new LocalContainerEntityManagerFactoryBean();
		factoryBean.setDataSource(lfvDataSource());
		factoryBean.setJpaVendorAdapter(jpaVendorAdapter);
		factoryBean.setPackagesToScan("...");
		factoryBean.setJpaProperties(props);
		return factoryBean;
	}

	@Bean(name= "dsLfv")
	@DependsOn({
			"lfrConfiguration",
			"jtaTransactionManager",
			"userTransactionManager"
	})
	@Primary
	public DataSource lfvDataSource() {
		log.debug("lfvDataSource()");
		final URI jdbcUri = URI.create(config.getJdbcUrl().trim().substring(5)); // Substring to skip prefix "jdbc:"

		DB2XADataSource xaDataSource = new DB2XADataSource();
		xaDataSource.setDatabaseName(jdbcUri.getPath().substring(1, 6));
		xaDataSource.setCurrentSchema(config.getSchemaLfrv().trim());
		xaDataSource.setUser(config.getJdbcUsername().trim());
		xaDataSource.setPassword(config.getJdbcPassword().trim());
		xaDataSource.setSslConnection(true);
//		This leads to InvalidAlgorithmParameterException: DH key size must be multiple of 64, and can only range from 512 to 8192 (inclusive). The specific key size 256 is not supported
//		xaDataSource.setSecurityMechanism(DB2BaseDataSource.ENCRYPTED_USER_AND_PASSWORD_SECURITY);
		xaDataSource.setServerName(jdbcUri.getHost());
		xaDataSource.setPortNumber(jdbcUri.getPort());
		xaDataSource.setDriverType(4); //Thin driver type used for JDBC, see https://www.atomikos.com/Documentation/ConfiguringDB2

		AtomikosDataSourceBean atomikosDataSource = new AtomikosDataSourceBean();
		atomikosDataSource.setXaDataSource(xaDataSource);
		atomikosDataSource.setUniqueResourceName("...");
		atomikosDataSource.setPoolSize(3);
		atomikosDataSource.setMaxPoolSize(20);
		atomikosDataSource.setMaxIdleTime(600000);
		atomikosDataSource.setMaxLifetime(1800000);
		atomikosDataSource.setBorrowConnectionTimeout(30000);
		atomikosDataSource.setTestQuery("SELECT current date FROM sysibm.sysdummy1");
		return atomikosDataSource;
	}

	@Bean(name = "jdbcLfv")
	@Autowired
	public JdbcTemplate lfvJdbcTemplate(@Qualifier("dsLfv") DataSource dsLfv) {
		return new JdbcTemplate(dsLfv);
	}

Tooling:

  • Spring Boot: 2.7.18
  • Java: 8
  • Maven
  • spring-boot-starter-jta-atomikos: 2.7.18
  • atomikos: 4.0.6
  • App Server: Tomcat
  • OS: Linux

I understand that this project is not intended for support - because bug reports may or may not be considered for inclusion some day (in a future release). If this is issue is important to me then I can go to https://www.atomikos.com/Main/SupportOverview and arrange a paid support subscription.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions