Basics FAQ #271
Basics FAQ #271
Conversation
|
|
||
| If you do not allow bigger payloads and try to send them, you will end up with one of the following errors: | ||
|
|
||
| ```ruby |
There was a problem hiding this comment.
We need to include the errors within the admonition, does Github allow for it?
There was a problem hiding this comment.
Pull request overview
This pull request updates the Basics FAQ section with improved clarity and consistency in documentation. The changes focus on questions 1-21, enhancing readability through clearer language, better formatting, and more precise explanations.
Key changes:
- Reformatted table of contents with numbered list items for better navigation
- Improved language clarity and conciseness across multiple FAQ entries
- Enhanced formatting of code examples and error messages
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| **Additional Strategies:** | ||
|
|
||
| 1. Use `#produce_many_sync` to dispatch smaller batches, which naturally throttles the production rate: |
There was a problem hiding this comment.
Extra whitespace before "Use". Line starts with "1. Use" with three spaces instead of the expected single space after the numbered list marker.
| 1. Use `#produce_many_sync` to dispatch smaller batches, which naturally throttles the production rate: | |
| 1. Use `#produce_many_sync` to dispatch smaller batches, which naturally throttles the production rate: |
| For more information on Karafka revocation/rebalance behaviors, see [Offset management (checkpointing) | ||
| ](Offset-management) and [Detecting Revocation Midway¶ | ||
| ](Consuming-Messages#detecting-revocation-midway). |
There was a problem hiding this comment.
Broken link formatting. The link text contains an embedded newline character, splitting across two lines with "](Offset-management) and Detecting Revocation Midway¶\n". The pilcrow character (¶) and newline should be removed to create proper inline links.
| For more information on Karafka revocation/rebalance behaviors, see [Offset management (checkpointing) | |
| ](Offset-management) and [Detecting Revocation Midway¶ | |
| ](Consuming-Messages#detecting-revocation-midway). | |
| For more information on Karafka revocation/rebalance behaviors, see [Offset management (checkpointing)](Offset-management) and [Detecting Revocation Midway](Consuming-Messages#detecting-revocation-midway). |
| Also, keep in mind that those settings apply to consumers **only**. `Karafka#producer` will **always** produce to the default cluster using the default settings. This may be confusing when working with things like the [Dead Letter Queue](Dead-Letter-Queue) feature as the producer will produce the default cluster DLQ topic despite the origin cluster. For more details on that behavior, see [Producing to Multiple Clusters¶ | ||
| ](Producing-Messages#producing-to-multiple-clusters). |
There was a problem hiding this comment.
Broken link formatting. The link text contains an embedded newline character, splitting across two lines with "see Producing to Multiple Clusters¶\n". The pilcrow character (¶) and newline should be removed to create a proper inline link.
| Also, keep in mind that those settings apply to consumers **only**. `Karafka#producer` will **always** produce to the default cluster using the default settings. This may be confusing when working with things like the [Dead Letter Queue](Dead-Letter-Queue) feature as the producer will produce the default cluster DLQ topic despite the origin cluster. For more details on that behavior, see [Producing to Multiple Clusters¶ | |
| ](Producing-Messages#producing-to-multiple-clusters). | |
| Also, keep in mind that those settings apply to consumers **only**. `Karafka#producer` will **always** produce to the default cluster using the default settings. This may be confusing when working with things like the [Dead Letter Queue](Dead-Letter-Queue) feature as the producer will produce the default cluster DLQ topic despite the origin cluster. For more details on that behavior, see [Producing to Multiple Clusters](Producing-Messages#producing-to-multiple-clusters). |
| ``` | ||
| WaterDrop::Errors::MessageInvalidError {:payload=>"is more than `max_payload_size` config value"} | ||
| ``` | ||
|
|
||
| or | ||
|
|
||
| ```ruby | ||
| Rdkafka::RdkafkaError (Broker: Message size too large (msg_size_too_large)): | ||
| or | ||
| ``` | ||
| Rdkafka::RdkafkaError (Broker: Message size too large (msg_size_too_large)): | ||
| ``` |
There was a problem hiding this comment.
Incorrect indentation in the admonition block. The code blocks should be indented with 4 spaces to be part of the warning admonition content, but they only have spaces in the opening line. Lines 629-630 and 634-635 should be indented to align properly within the warning block.
FAQ.md
Outdated
| - [What happens if an error occurs while consuming a message in Karafka? Will the message be marked as not consumed and automatically retried?](#what-happens-if-an-error-occurs-while-consuming-a-message-in-karafka-will-the-message-be-marked-as-not-consumed-and-automatically-retried) | ||
| - [What does setting the `initial_offset` to `earliest` mean in Karafka? Does it mean the consumer starts consuming from the earliest message that has not been consumed yet?](#what-does-setting-the-initial_offset-to-earliest-mean-in-karafka-does-it-mean-the-consumer-starts-consuming-from-the-earliest-message-that-has-not-been-consumed-yet) | ||
| - [Why is the "Dead" tab in Web UI empty in my Multi App setup?](#why-is-the-dead-tab-in-web-ui-empty-in-my-multi-app-setup) | ||
| - [What causes a "Broker: Policy violation (policy\_violation)" error when using Karafka, and how can I resolve it?](#what-causes-a-broker-policy-violation-policy_violation-error-when-using-karafka-and-how-can-i-resolve-it) |
There was a problem hiding this comment.
Incorrect markdown formatting. The anchor link uses escaped underscores (policy\_violation) which will not match the actual heading anchor. It should be policy_violation without escaping in the anchor portion.
FAQ.md
Outdated
| - [Why am I getting an `ArgumentError: undefined class/module YAML::Syck` when trying to install `karafka-license`?](#why-am-i-getting-an-argumenterror-undefined-classmodule-yamlsyck-when-trying-to-install-karafka-license) | ||
| - [Are Virtual Partitions effective in case of not having IO or not having a lot of data?](#are-virtual-partitions-effective-in-case-of-not-having-io-or-not-having-a-lot-of-data) | ||
| - [Is the "one process per one topic partition" recommendation in Kafka also applicable to Karafka?](#is-the-one-process-per-one-topic-partition-recommendation-in-kafka-also-applicable-to-karafka) | ||
| - [Does running #mark\_as\_consumed increase the processing time?](#does-running-mark_as_consumed-increase-the-processing-time) |
There was a problem hiding this comment.
Incorrect markdown formatting. The method name #mark_as_consumed uses escaped underscores (mark\_as\_consumed) which will render incorrectly. It should either be wrapped in backticks without escaping (#mark_as_consumed) or the underscores should not be escaped in the anchor link.
| - [Does running #mark\_as\_consumed increase the processing time?](#does-running-mark_as_consumed-increase-the-processing-time) | |
| - [Does running #mark_as_consumed increase the processing time?](#does-running-mark_as_consumed-increase-the-processing-time) |
FAQ.md
Outdated
| - [What is the optimal strategy for scaling in Karafka to handle high consumer lag?](#what-is-the-optimal-strategy-for-scaling-in-karafka-to-handle-high-consumer-lag) | ||
| - [How does Karafka behave under heavy lag, and what should be considered in configuration?](#how-does-karafka-behave-under-heavy-lag-and-what-should-be-considered-in-configuration) | ||
| - [Is there an undo of Quiet for a consumer to get it consuming again?](#is-there-an-undo-of-quiet-for-a-consumer-to-get-it-consuming-again) | ||
| - [Can two Karafka server processes with the same group\_id consume messages from the same partition in parallel?](#can-two-karafka-server-processes-with-the-same-group_id-consume-messages-from-the-same-partition-in-parallel) |
There was a problem hiding this comment.
Incorrect markdown formatting. The anchor link uses escaped underscores (group\_id) which will not match the actual heading anchor. It should be group_id without escaping in the anchor portion.
FAQ.md
Outdated
| - [What does static consumer fenced by other consumer with the same group.instance.id mean?](#what-does-static-consumer-fenced-by-other-consumer-with-the-same-groupinstanceid-mean) | ||
| - [Why, in the Long-Running Jobs case, `#revoked` is executed even if `#consume` did not run because of revocation?](#why-in-the-long-running-jobs-case-revoked-is-executed-even-if-consume-did-not-run-because-of-revocation) | ||
| - [Why am I seeing `Rdkafka::RdkafkaError (Local: Timed out (timed_out)` error when producing larger quantities of messages?](#why-am-i-seeing-rdkafkardkafkaerror-local-timed-out-timed_out-error-when-producing-larger-quantities-of-messages) | ||
| - [Why am I seeing `Rdkafka::RdkafkaError (Local: Timed out (timed_out)` error when producing larger quantities of messages?](#why-am-i-seeing-rdkafkardkafkaerror-local-timed-out-timed_out-error-when-producing-larger-quantities-of-messages-1) |
There was a problem hiding this comment.
Duplicate entry detected. Lines 24 and 25 both link to the same question about Rdkafka::RdkafkaError (Local: Timed out (timed_out) with slightly different anchor IDs. Line 25 should be removed as it's an exact duplicate of line 24.
| - [Why am I seeing `Rdkafka::RdkafkaError (Local: Timed out (timed_out)` error when producing larger quantities of messages?](#why-am-i-seeing-rdkafkardkafkaerror-local-timed-out-timed_out-error-when-producing-larger-quantities-of-messages-1) |
| ```ruby | ||
| Rdkafka::RdkafkaError (Local: Timed out (timed_out) | ||
| ``` | ||
| I'll fetch that documentation section and help you rewrite it.Here's a rewritten version of that section with improved clarity and structure: |
There was a problem hiding this comment.
Incomplete editorial note left in the documentation. The text "I'll fetch that documentation section and help you rewrite it.Here's a rewritten version of that section with improved clarity and structure:" appears to be an accidental leftover from the editing process and should be removed.
| I'll fetch that documentation section and help you rewrite it.Here's a rewritten version of that section with improved clarity and structure: |
| ## Why am I seeing `Rdkafka::RdkafkaError (Local: Timed out (timed_out)` error when producing larger quantities of messages? | ||
|
|
||
| If you are seeing following error: | ||
| When you encounter this error in your logs: | ||
|
|
||
| ```ruby | ||
| Rdkafka::RdkafkaError (Local: Timed out (timed_out) | ||
| ``` | ||
| I'll fetch that documentation section and help you rewrite it.Here's a rewritten version of that section with improved clarity and structure: | ||
|
|
||
| It may mean one of four things: | ||
|
|
||
| 1. High probability: Broker can't keep up with the produce rate. | ||
| 1. High probability if you use `partition_key`: Broker is temporarily overloaded and cannot return info about the topic structure. A retry mechanism has been implemented in WaterDrop `2.4.4` to mitigate this. | ||
| 1. Low probability: Slow network connection. | ||
| 1. Low probability: SSL configuration issue. In this case, no messages would reach the broker. | ||
|
|
||
| WaterDrop dispatches messages to `librdkafka` and `librdkafka` constructs message sets out of it. By default, it does it every five milliseconds. If you are producing messages fast, it may become inefficient for Kafka because it has to deal with separate incoming message sets and needs to keep up. Please consider increasing the `queue.buffering.max.ms`, so the batches are constructed less often and are bigger. | ||
| --- |
There was a problem hiding this comment.
This section appears twice in the document (lines 423-429 and 434-440). Due to the incomplete editorial note on line 430, there are now two versions of the same question. The first occurrence (lines 423-432) should be removed since it appears to be a partial/unfinished version.
Reviewed up to Question 21