Kafka 消费者按 group、分区和偏移位置读取记录。auto.offset.reset 只在没有有效已提交偏移等条件下提供重置策略;不能将默认 latest 简化成“每次启动都只看最新消息”。
单节点测试条件
新增实验固定 Kafka broker/client 3.9.1、JDK 21,使用本轮新建的单节点 KRaft 容器。客户端与 broker 共享隔离网络命名空间,所以 127.0.0.1:9092 指向测试 broker,没有开放宿主端口或连接历史文中地址。完整容器准备命令和环境变量在仓库 test-phase6-examples.py 中记录。
复用 SelectorProvider 文章的共享 POM。下面测试创建随机 topic 和 group,先发送三条消息,再用 earliest 和关闭自动提交的消费者拉取,处理后同步提交,核对提交的是下一条偏移。
package site;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.admin.*;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.*;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class KafkaConsumerTest {
@Test void consumesAndCommitsAnExplicitGroup() throws Exception {
String topic = "phase6-" + UUID.randomUUID(), group = "group-" + UUID.randomUUID();
var adminProperties = new Properties();
adminProperties.put("bootstrap.servers", "127.0.0.1:9092");
try (var admin = Admin.create(adminProperties)) {
admin.createTopics(List.of(new NewTopic(topic,1,(short)1))).all().get(20,TimeUnit.SECONDS);
try {
var producerProperties = new Properties();
producerProperties.putAll(adminProperties);
producerProperties.put("key.serializer", StringSerializer.class.getName());
producerProperties.put("value.serializer", StringSerializer.class.getName());
try (var producer = new KafkaProducer<String,String>(producerProperties)) {
for (int i=0;i<3;i++) producer.send(new ProducerRecord<>(topic,"key","value-"+i)).get(10,TimeUnit.SECONDS);
}
var properties = new Properties();
properties.putAll(adminProperties);
properties.put("group.id",group);
properties.put("enable.auto.commit","false");
properties.put("auto.offset.reset","earliest");
properties.put("key.deserializer",StringDeserializer.class.getName());
properties.put("value.deserializer",StringDeserializer.class.getName());
try (var consumer = new KafkaConsumer<String,String>(properties)) {
consumer.subscribe(List.of(topic));
var values = new ArrayList<String>();
long deadline = System.nanoTime()+TimeUnit.SECONDS.toNanos(30);
while (values.size()<3 && System.nanoTime()<deadline) {
for (var record:consumer.poll(Duration.ofMillis(500))) values.add(record.value());
}
assertEquals(List.of("value-0","value-1","value-2"),values);
consumer.commitSync();
long committed = consumer.committed(Set.of(new TopicPartition(topic,0))).values().iterator().next().offset();
assertEquals(3,committed);
System.out.println("received=3; values=0,1,2; committed-next-offset=3");
}
} finally { admin.deleteTopics(List.of(topic)).all().get(20,TimeUnit.SECONDS); }
}
}
}
mvn -Dtest=KafkaConsumerTest test
实际结果与处理边界
本次输出 received=3; values=0,1,2; committed-next-offset=3。单分区测试便于核对顺序;跨分区不存在这里展示的全局顺序保证。try-with-resources 关闭客户端,finally 删除本轮 topic,测试不留下无限 poll 循环。
提交偏移与完成外部业务副作用是两个动作。先提交可能丢失未处理工作,后提交失败重试可能重复处理,应按业务设计幂等或事务边界。这个例子没有验证再均衡、多 broker、故障重试、权限认证或恰好一次处理,也不能把 commitSync 成功当作所有外部写入已提交。
下一步阅读
原资料片段(未重建原环境)
下方保留原资料的上下文、示例与截图;本轮验证只对应上方新增文件,未将旧运行结果记为本轮结果。
消费者消费消息的基本实现
测试代码
package com.example.kafka;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;
public class MySimpleConsumer {
private final static String TOPIC_NAME = "test";
private final static String CONSUMER_GROUP_NAME = "testGroup";
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
"127.0.0.1:9092" /* 改为测试环境地址;本轮未运行此无限循环片段 */);
// 消费分组名
props.put(ConsumerConfig.GROUP_ID_CONFIG, CONSUMER_GROUP_NAME);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
//1.创建⼀个消费者的客户端
KafkaConsumer< String, String > consumer = new KafkaConsumer < String,
String > (props);
//2. 消费者订阅主题列表
consumer.subscribe(Arrays.asList(TOPIC_NAME));
while (true) {
/*
* 3.poll() API 是拉取消息的⻓轮询
*/
ConsumerRecords< String, String > records =
consumer.poll(Duration.ofMillis(10000));
for (ConsumerRecord< String, String > record: records) {
//4.打印消息
System.out.printf("收到消息:partition = %d,offset = %d, key = %s, value = %s%n ", record.partition(),
record.offset(), record.key(), record.value());
}
}
}
}
配置日志打印
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<logger name="org.apache.kafka.clients" level="info" />
</configuration>

生产者发送消息

历史 latest 配置示意(需要有效偏移条件)

版权声明:本文不是「本站」原创文章,版权归原作者所有 | 原文地址: