本文共 2247 字，大约阅读时间需要 7 分钟。

Flume容错（故障转移）配置实例

1. 概念

Flume容错机制用于实现故障转移，即在一台机器出现故障时，另一台机器代替其工作。这种设计通常用于解决单点故障问题，通过为关键部件设置备份，确保数据不会丢失或处理中断。

容错机制的工作原理是：当主机器出现故障时，容错入口会自动切换到备用机器，确保流数据继续正常传输。备份的数量越多，容错能力越强，但同时也会带来资源的浪费。

2. 脚本编写

本节将介绍如何配置一个简单的Flume容错场景。

2.1. 创建配置文件

2.1.1. hadoop01（执行端）

# agent1配置
agent1.sources = r1
agent1.channels = c1
agent1.sinks = k1 k2
agent1.sinkgroups = g1
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1000
agent1.channels.c1.transactionCapacity = 100
agent1.sources.r1.channels = c1
agent1.sources.r1.type = exec
agent1.sources.r1.command = tail -F /home/xiaokang/logs/456.log
#basic sono配置
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = avro
agent1.sinks.k1.hostname = hadoop02
agent1.sinks.k1.port = 52020
agent1.sinks.k2.channel = c1
agent1.sinks.k2.type = avro
agent1.sinks.k2.hostname = hadoop03
agent1.sinks.k2.port = 52020
# failover配置
agent1.sinkgroups.g1 Processor.type = failover
agent1.sinkgroups.g1.sinks = k1 k2
agent1.sinkgroups.g1.processor.priority.k1 = 10
agent1.sinkgroups.g1.processor.priority.k2 = 1
agent1.sinkgroups.g1.processor.maxPenalty = 10000

2.1.2. hadoop02（数据源）

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop02
a1.sources.r1.port = 52020
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

2.1.3. hadoop03（备用端）

a1.sources = r1
a1.sinks = k1
a1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.bind = hadoop02
a1.sources.r1.port = 52020
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1

2.2. 启动Flume Agent

# hadoop01启动
$ flume-ng agent -n agent1 -c /opt/software/flume-1.9.0/conf/ -f flume_scripts/exec-avro-failover.properties -Dflume.root.logger=INFO,console
# hadoop02启动
$ flume-ng agent -n a1 -c /opt/software/flume-1.9.0/conf/ -f flume_scripts/avro-logger-failover.properties -Dflume.root.logger=INFO,console
# hadoop03启动
$ flume-ng agent -n a1 -c /opt/software/flume-1.9.0/conf/ -f flume_scripts/avro-logger-failover.properties -Dflume.root.logger=INFO,console

3. 验证

3.1. 模拟数据生成

在hadoop01执行以下命令，生成模拟日志数据：

$ while true; do date >> /home/xiaokang/logs/456.log; done

3.2. 测试容错机制

启动Flume Agent后，监控hadoop02的日志接收情况。通过登录hadoop02查看Flume运行日志，确认数据是否 正常流向hadoop03（备用端）。

3.3. 故障模拟与测试

• 偶意列中断hadoop02的网络连接
• 查看hadoop01的日志输出，确认数据接收情况
• 观察是否hadoop03接收到数据，并正常处理

转载地址：http://ijpaz.baihongyu.com/