Troubleshooting on Apache Dubbo

Application Start Failure

Mon, 01 Jan 0001 00:00:00 +0000

During the development and production deployment stages, applications may fail to start due to various unexpected changes. For Dubbo, startup failures usually present error messages similar to the following.

Caused by: java.lang.IllegalStateException: Dubbo Module[1.1.1] is stopping or stopped, can not start again

[DUBBO] Dubbo Application[1.1](first-dubbo-consumer) start failure

Summary

Correctly configure log output, go back to the first error location and handle it.

Troubleshooting Methods

1 Configure Log Output

Dubbo currently supports multiple log frameworks. If multiple log frameworks (such as log4j and logback) are supported in the environment, Dubbo will output log frameworks in the order of (log4j > slf4j > log4j2 > jcl).

Request Timing Sampling

Mon, 01 Jan 0001 00:00:00 +0000

The performance sampling feature can detect the time spent at various points in the Dubbo processing chain, logging the call duration when a timeout occurs ( usageTime / timeout > profilerWarnPercent * 100 ).

This feature has two modes: simple profiler and detail profiler, where the simple profiler mode is enabled by default and the detail profiler mode is disabled by default. The detail profiler collects more data, such as the processing time of each filter and specific protocol timing. If long processing times are detected in the Dubbo framework while in simple profiler mode, you can enable detail profiler mode to better troubleshoot issues.

Address not found exception

Mon, 01 Jan 0001 00:00:00 +0000

During the development and production deployment process, since Dubbo is a framework that requires service discovery for calls, it is easy to encounter No Provider exceptions due to various objective reasons. This article aims to provide a systematic troubleshooting approach to help you quickly locate and resolve the issue when exceptions occur.

java.lang.IllegalStateException: Failed to check the status of the service org.apache.dubbo.samples.api.GreetingsService. No provider available for the service org.apache.dubbo.samples.api.GreetingsService from the url consumer://*** to the consumer 30.221.146.226 use dubbo version 3.2.0-beta.4

Exception in thread "main" org.apache.dubbo.rpc.RpcException: No provider available from registry 127.0.0.1:2181 for service org.apache.dubbo.samples.api.GreetingsService on consumer 30.221.146.226 use dubbo version 3.2.0-beta.4, please check status of providers(disabled, not registered or in blacklist).

Summary in one sentence

When the service cannot be found, first check whether the service is fully developed and deployed, then corroborate whether it is registered in the registry, check the service side’s published status if registered, and consumer side’s subscription status if not registered. Any step along this way encountering issues will lead to exceptions.

Low Request Success Rate

Mon, 01 Jan 0001 00:00:00 +0000

In a production environment, the request success rate and latency are the most critical metrics. This article will introduce troubleshooting approaches when the request success rate decreases.

Summary in One Sentence

Analyze bottlenecks through end-to-end metrics (consumer end, network, server end, external dependencies, etc.)

Troubleshooting Approaches

1 Is the Consumer End Constructing Requests Normally?

1.1 Check if Class Objects Are Serializable

When using Dubbo for RPC remote calls, due to cross-process calls, Dubbo checks whether all data objects implement the Serializable interface to prevent unexpected data requests over the network.