/tags/kafka/Recent content in Kafka on Mi&Bee BlogHugo -- gohugo.iozh-CN蓝宝石的傻话Sat, 25 Nov 2023 00:00:00 +0000/posts/telemetry/deepflow-agent-proto-dev/Sat, 25 Nov 2023 00:00:00 +0000/posts/telemetry/deepflow-agent-proto-dev/<ul> <li><a href="#%E6%A6%82%E8%BF%B0">概述：</a></li> <li><a href="#%E5%A6%82%E4%BD%95%E5%88%86%E6%9E%90%E4%B8%80%E4%B8%AA%E5%8D%8F%E8%AE%AEmongodb">如何分析一个协议(MongoDB)</a> <ul> <li><a href="#%E5%8D%8F%E8%AE%AE%E6%96%87%E6%A1%A3%E7%9A%84%E5%88%86%E6%9E%90%E6%80%9D%E8%B7%AF">协议文档的分析思路</a></li> <li><a href="#mongodb%E5%8D%8F%E8%AE%AE%E6%93%8D%E4%BD%9C%E7%A0%81%E8%AF%B4%E6%98%8E%E8%A1%A8">MongoDB协议操作码说明表</a></li> <li><a href="#%E5%AF%B9%E6%9C%80%E5%B8%B8%E8%A7%81%E7%9A%84%E6%93%8D%E4%BD%9C%E7%A0%81op_msg%E5%88%86%E6%9E%90">对最常见的操作码<code>OP_MSG</code>分析</a></li> </ul> </li> <li><a href="#%E5%9C%A8deepflow-agent%E6%89%A9%E5%B1%95%E4%B8%80%E4%B8%AA%E5%8D%8F%E8%AE%AE%E8%A7%A3%E6%9E%90%E9%87%87%E9%9B%86">在<code>DeepFlow Agent</code>扩展一个协议解析采集</a> <ul> <li><a href="#deepflow-agent%E7%9A%84%E5%BC%80%E5%8F%91%E6%96%87%E6%A1%A3%E6%A6%82%E8%A6%81"><code>DeepFlow Agent</code>的开发文档概要</a></li> <li><a href="#%E4%BB%A3%E7%A0%81%E6%8C%87%E5%BC%95">代码指引</a> <ul> <li><a href="#%E5%AE%9A%E4%B9%89%E4%B8%80%E4%B8%AA%E5%8D%8F%E8%AE%AE%E5%B9%B6%E7%94%A8%E4%B8%80%E4%B8%AA%E5%B8%B8%E9%87%8F%E6%A0%87%E8%AF%86">定义一个协议，并用一个常量标识。</a></li> <li><a href="#%E4%B8%BA%E6%96%B0%E5%8D%8F%E8%AE%AE%E5%87%86%E5%A4%87%E8%A7%A3%E6%9E%90%E9%80%BB%E8%BE%91">为新协议准备解析逻辑</a> <ul> <li><a href="#%E5%AE%9A%E4%B9%89%E7%BB%93%E6%9E%84%E4%BD%93">定义结构体</a></li> <li><a href="#%E5%AE%9E%E7%8E%B0l7protocolparserinterface">实现 <code>L7ProtocolParserInterface</code></a></li> </ul> </li> </ul> </li> </ul> </li> <li><a href="#%E5%88%A9%E7%94%A8wasm%E6%8F%92%E4%BB%B6%E6%89%A9%E5%B1%95deepflow%E7%9A%84%E5%8D%8F%E8%AE%AE%E9%87%87%E9%9B%86">利用<code>Wasm</code>插件扩展<code>DeepFlow</code>的协议采集</a> <ul> <li><a href="#kafka%E5%8D%8F%E8%AE%AE%E5%88%86%E6%9E%90"><code>Kafka</code>协议分析</a> <ul> <li><a href="#kafka%E7%9A%84header%E5%92%8Cdata%E6%A6%82%E8%A7%88"><code>Kafka</code>的<code>Header</code>和<code>Data</code>概览</a></li> <li><a href="#kafka%E7%9A%84fetch-api">Kafka的Fetch API</a></li> <li><a href="#kafka%E7%9A%84produce-api">Kafka的Produce API</a></li> </ul> </li> <li><a href="#kafka%E5%8D%8F%E8%AE%AEdeepflow-agent%E5%8E%9F%E7%94%9F%E8%A7%A3%E7%A0%81">Kafka协议DeepFlow Agent原生解码</a></li> <li><a href="#deepflow-agent%E7%9A%84-wasm-%E6%8F%92%E4%BB%B6"><code>DeepFlow Agent</code>的 <code>Wasm</code> 插件</a> <ul> <li><a href="#wasm-go-sdk-%E7%9A%84%E6%A1%86%E6%9E%B6">Wasm Go SDK 的框架</a></li> <li><a href="#%E6%8F%92%E4%BB%B6%E4%BB%A3%E7%A0%81%E6%8C%87%E5%BC%95">插件代码指引</a></li> </ul> </li> </ul> </li> <li><a href="#%E7%BB%93%E8%AF%AD">结语</a> <ul> <li><a href="#%E5%8E%9F%E7%94%9Frust%E6%89%A9%E5%B1%95">原生Rust扩展</a></li> <li><a href="#wasm%E6%8F%92%E4%BB%B6%E6%89%A9%E5%B1%95">Wasm插件扩展</a></li> </ul> </li> <li><a href="#%E9%99%84%E5%BD%95">附录</a></li> </ul> <div style="position:relative; padding-bottom:75%; width:100%; height:0"> <iframe src="//player.bilibili.com/player.html?isOutside=true&aid=921401645&bvid=BV1Nu4y1A7ZC&cid=1345829549&p=1&autoplay=false" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true" style="position:absolute; height: 100%; width: 100%;"></iframe> </div> <h2 id="概述">概述：</h2> <p><code>MongoDB</code> 目前使用广泛，但是缺乏有效的可观测能力。 <code>DeepFlow</code> 在可观测能力上是很优秀的解决方案，但是却缺少了对 <code>MongoDB</code> 协议的支持。 该文是为 <code>DeepFlow</code> 扩展了 <code>MongoDB</code> 协议解析，增强 <code>MongoDB</code> 生态的可观测能力，简要描述了从协议文档分析到在 <code>DeepFlow</code> 内实现代码解析的过程拆解。</p>