unity3d：GameFramework+xLua+Protobuf+lua-protobuf，与服务器交互收发协议

概述

1.cs收发协议，通过protobuf序列化
2.lua收发协议，通过lua-protobuf序列化

一条协议字节流组成

C#协议基类

CSPacketBase，SCPacketBaseC#用协议基类

proto生成的CS类，基于这两个基类。分别为CSPacketBase是客户端发送至服务器，SCPacketBase是服务器发送至客户端
Q：为什么要区分这2个
A：反射注册所有SCPacketBase类，为C#接收协议反序列化候选
一个类示例

  [global::System.Serializable, global::ProtoBuf.ProtoContract(Name=@"CSLogin")]
  public partial class CSLogin : CSPacketBase
  {
    public CSLogin() {}
    

    private string _account = "";
    [global::ProtoBuf.ProtoMember(1, IsRequired = false, Name=@"account", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue("")]
    public string account
    {
      get { return _account; }
      set { _account = value; }
    }

    private string _password = "";
    [global::ProtoBuf.ProtoMember(2, IsRequired = false, Name=@"password", DataFormat = global::ProtoBuf.DataFormat.Default)]
    [global::System.ComponentModel.DefaultValue("")]
    public string password
    {
      get { return _password; }
      set { _password = value; }
    } 
	//网络协议Id
	public override int Id 
	{ 
		get {return (int)Network.NetMsgID.CSLogin;} 
	} 

	//回到引用池，变量设置初始化。如果是引用型成员变量也要回到引用池
	public override void Clear()
	{
		_account = "" ;
		_password = "" ;

	}
     	
  }

SCPacketLua

C#中接收包，传递给Lua处理
其中m_id为协议id，m_len为字节数组长度

public class SCPacketLua : PacketBase
{
    public int m_id;
    public int m_len;
    public PacketBuffer m_bytes; 

在网络事件使用完毕GameFramework.EventPool.HandleEvent 中

ReferencePool.Release(e);

触发回收

    public override void Clear()
    {
        ReferencePool.Release(m_bytes);
        m_bytes = null; //要去引用，不然引用池那释放不了 

CSPacketLua

C#中用于接收从lua传递过来的字节流，发送给服务器

    public  class CSPacketBaseLua : PacketBase
    {
        public PacketBuffer m_bytes; //发送字节流
        public int m_id = 0; //协议id
        public ushort Len //字节流长度

C#中发送协议

CSLogin csLogin = ReferencePool.Acquire<CSLogin>();
csLogin.account = "123";
csLogin.password = "456";
GameEntry.Network.Send(csLogin);

主线程遍历发送队列

每有一个发送，把packet放入到发送队列中，unity主线程中遍历发送队列，把当前帧的全部待发送包，序列化到一个流中

GameFramework.Network.NetworkManager.NetworkChannelBase.ProcessSend

if (m_SendState.Stream.Length > 0 || m_SendPacketPool.Count <= 0)
{
    //发送流中还有未发送完的或者没有待发送的包
    return false;
}
//所有在发送队列中，都序列化到流中
while (m_SendPacketPool.Count > 0)
{
    Packet packet = null;
    lock (m_SendPacketPool)
    {
        //每次从发送队列中取一个
        packet = m_SendPacketPool.Dequeue();
    }

    bool serializeResult = false;
    try
    {
        serializeResult = m_NetworkChannelHelper.Serialize(packet, m_SendState.Stream);
    }
}
//流的操作，写完，要回到起点
m_SendState.Stream.Position = 0L;    

序列化消息包

StarForce.NetworkChannelHelper.Serialize

/// 

/// 序列化消息包。
/// 
/// 消息包类型。
/// 要序列化的消息包。
/// 要序列化的目标流。
/// 是否序列化成功。
public bool Serialize<T>(T packet, Stream destination) where T : Packet
{
    //todo：频繁is as 会有性能消耗
    if (packet is CSPacketBase)
    {
        PacketBase packetImpl = packet as CSPacketBase;
        //先写入body
        m_CachedStream.SetLength(1024*8); // 此行防止 Array.Copy 的数据无法写入
        m_CachedStream.Position = PacketHeaderLength;
        Serializer.Serialize(m_CachedStream, packet);

        //获得body长度,再写入消息头，id，body长度
        ushort bodyLen = (ushort)((int)m_CachedStream.Position - PacketHeaderLength);

        arrID = BitConverter.GetBytes(packetImpl.Id);
        arrBodyLen = BitConverter.GetBytes(bodyLen);
        m_CachedStream.Position = 0;
        m_CachedStream.Write(arrID, 0, 4);
        m_CachedStream.Write(arrBodyLen, 0, 2);


        m_CachedStream.SetLength(PacketHeaderLength + bodyLen);
        byte[] arrBytes = m_CachedStream.ToArray();
        Log.Info("序列化字节流：{0}", BitConverter.ToString(arrBytes));
        m_CachedStream.WriteTo(destination);
        ReferencePool.Release(packet);
        return true;
    }
    //else if (packetImpl.PacketType == PacketType.ClientToServerLua)
    else if (packet is CSPacketLua)
    {
        m_CachedStream.SetLength(1024 * 8); // 此行防止 Array.Copy 的数据无法写入
        CSPacketLua packetLua = packet as CSPacketLua;  
        arrID = BitConverter.GetBytes(packetLua.Id);
        arrBodyLen = BitConverter.GetBytes(packetLua.Len);
        m_CachedStream.Position = 0; //每次开始写流，流位置要设置为0，代表起始位置。每次写byte，pos会自动增加
        m_CachedStream.Write(arrID, 0, 4);
        m_CachedStream.Write(arrBodyLen, 0, 2);
        m_CachedStream.Write(packetLua.m_bytes.Buffer, 0, packetLua.Len);
        m_CachedStream.SetLength(PacketHeaderLength + packetLua.Len);//写完流，要设置下流的真实长度。因为流是复用的，不然不会截断
        byte[] arrBytes = m_CachedStream.ToArray();
        Log.Info("序列化字节流Lua：{0}", BitConverter.ToString(arrBytes));
        m_CachedStream.WriteTo(destination);//缓存流写入到发送流中
        ReferencePool.Release(packet);
        return true;
    }

    Log.Warning("Send packet invalid.");
    return false;
}

对于CSPacketBase类型
1.m_CachedStream是每个packet序列化的流，每次使用前需要设置position
2.先设置m_CachedStream.Position = PacketHeaderLength; 先跳过id，bodyLen位置，先写入body
3.protobuf序列化Serializer.Serialize(m_CachedStream, packet);后得到ushort bodyLen = (ushort)((int)m_CachedStream.Position - PacketHeaderLength);即为bodyLen长度
4.再设置m_CachedStream.Position = 0;，写入id字节流，bodyLen字节流
5.m_CachedStream.SetLength(PacketHeaderLength + bodyLen);写完后要设置长度截断，因为m_CachedStream是复用的，可能上次使用后面还有字节数据
6. m_CachedStream.WriteTo(destination);即为发送流，每次会添加到发送流的末尾
对于CSPacketLua类型
1.由于byte是在lua中序列化好的传递到C#的，只需要按照顺序写入到m_CachedStream中，其他流程与CSPacketBase一致

发送流

GameFramework.Network.NetworkManager.TcpWithSyncReceiveNetworkChannel.SendAsync

private void SendAsync()
{
    try
    {
        m_Socket.BeginSend(m_SendState.Stream.GetBuffer(), (int)m_SendState.Stream.Position, (int)(m_SendState.Stream.Length - m_SendState.Stream.Position), SocketFlags.None, m_SendCallback, m_Socket);
    }

每次从流中取position到length部分。每次发送一个完整流，从0开始。进入到发送回调
GameFramework.Network.NetworkManager.TcpWithSyncReceiveNetworkChannel.SendCallback

private void SendCallback(IAsyncResult ar)
{
    Socket socket = (Socket)ar.AsyncState;
   int bytesSent = 0;
    try
    {
        bytesSent = socket.EndSend(ar);
    }

    m_SendState.Stream.Position += bytesSent;
    if (m_SendState.Stream.Position < m_SendState.Stream.Length)
    {
        SendAsync();
        return;
    }

发送了一段，设置流位置
如果位置

C#中接收协议

初始化时反射注册协议id对应type，协议id对应处理Handle

StarForce.NetworkChannelHelper.Initialize

// 反射注册包和包处理函数。
Type packetBaseType = typeof(SCPacketBase);
Type packetHandlerBaseType = typeof(PacketHandlerBase);
Assembly assembly = Assembly.GetExecutingAssembly();
Type[] types = assembly.GetTypes();
for (int i = 0; i < types.Length; i++)
{
    if (!types[i].IsClass || types[i].IsAbstract)
    {
        continue;
    }

    if (types[i].BaseType == packetBaseType)
    {
        //确定msgID反序列化的类结构
        PacketBase packetBase = (PacketBase)Activator.CreateInstance(types[i]);
        Type packetType = GetServerToClientPacketType(packetBase.Id);
        //防止监听的协议id重复
        if (packetType != null)
        {
            Log.Warning("Already exist packet type '{0}', check '{1}' or '{2}'?.", packetBase.Id.ToString(), packetType.Name, packetBase.GetType().Name);
            continue;
        }

        m_ServerToClientPacketTypes.Add(packetBase.Id, types[i]);
    }
    else if (types[i].BaseType == packetHandlerBaseType)
    {
        //网络事件handle处理
        IPacketHandler packetHandler = (IPacketHandler)Activator.CreateInstance(types[i]);
        m_NetworkChannel.RegisterHandler(packetHandler);
    }
}

异步接收流

连接成功，开始异步接收流
GameFramework.Network.NetworkManager.TcpNetworkChannel.ReceiveAsync

//每次获取到完整头/body。流会pos = 0，len为头长，或者body长
//每次拆包都是读取pos累积，剩余要读的为len-pos
m_Socket.BeginReceive(m_ReceiveState.Stream.GetBuffer(), (int)m_ReceiveState.Stream.Position, (int)(m_ReceiveState.Stream.Length - m_ReceiveState.Stream.Position), SocketFlags.None, m_ReceiveCallback, m_Socket);

有服务器下发协议，进入到接收回调中
GameFramework.Network.NetworkManager.TcpNetworkChannel.ReceiveCallback

private void ReceiveCallback(IAsyncResult ar)
{
    Socket socket = (Socket)ar.AsyncState;
  
    int bytesReceived = 0;
    try
    {
        bytesReceived = socket.EndReceive(ar);
    }
    //每次读取pos会累加
    m_ReceiveState.Stream.Position += bytesReceived;
    if (m_ReceiveState.Stream.Position < m_ReceiveState.Stream.Length)
    {
        //未读满上次设定长度，接着读。上次设定长度为HeadLen，BodyLen
        ReceiveAsync();
        return;
    }
    //开始解析，流要置为0位置开始解析
    m_ReceiveState.Stream.Position = 0L;

    bool processSuccess = false;
    if (m_ReceiveState.PacketHeader != null)
    {
        //上次获取到头，这次反序列处理body
        processSuccess = ProcessPacket();
        m_ReceivedPacketCount++;
    }
    else
    {
        //未获取到头，反序列化头
        processSuccess = ProcessPacketHeader();
    }

    if (processSuccess)
    {
        //处理成功，接着接收
        ReceiveAsync();
        return;
    }
}

1.初始化时，设置第一次流接收Length为6（协议id int+ bodyLen ushort）。即待接收包头
2.如果接收满了Length，进入到处理包头，解析出协议id，bodyLen
3.设置下一次接收为Length为bodyLen。即待接收包体
4.如果接收满Length，此时进入到处理包体，解析出body对应的对象。设置下次接收为包头Length6，循环到第一步
注意事项
如果有拆包黏包，在接收回调中处理，并且接满一个模式，再解析

    m_ReceiveState.Stream.Position += bytesReceived;
    if (m_ReceiveState.Stream.Position < m_ReceiveState.Stream.Length)
    {
        //未读满上次设定长度，接着读。上次设定长度为HeadLen，BodyLen
        ReceiveAsync();
        return;
    }

每次开始解析前，需要流postion = 0开始，因为随着接收，position到了末尾，无法解析

    //开始解析，流要置为0位置开始解析
    m_ReceiveState.Stream.Position = 0L;

解析包头

StarForce.NetworkChannelHelper.DeserializePacketHeader

source.Position = 0;
SCPacketHeader scHead = ReferencePool.Acquire<SCPacketHeader>();
source.Read(arrID, 0, 4);
scHead.Id = BitConverter.ToInt32(arrID,0);
source.Read(arrBodyLen, 0, 2);
scHead.PacketLength = BitConverter.ToUInt16(arrBodyLen, 0);

得到协议id，bodyLen

解析包体

StarForce.NetworkChannelHelper.DeserializePacket

Packet packet = null;
if (scPacketHeader.IsValid)
{
    Type packetType = GetServerToClientPacketType(scPacketHeader.Id);
    if (packetType != null)
    {
        //source 为接收流，每次接收一整条消息前，设置了Len
        packet = (Packet)ReferencePool.Acquire(packetType);
        packet = (Packet)RuntimeTypeModel.Default.Deserialize(source, packet, packetType);
    }
    else
    {
        //如果id找不到字节流传递给Lua中反序列化为table
        SCPacketLua luaPacket = ReferencePool.Acquire<SCPacketLua>();//引用池中使用，后续会在使用事件通知后，回到引用池
        luaPacket.m_id = scPacketHeader.Id;
        luaPacket.m_len = scPacketHeader.PacketLength;
        luaPacket.m_bytes = PacketBuffer.GetBuffer(scPacketHeader.PacketLength);
        source.Read(luaPacket.m_bytes.Buffer, 0, scPacketHeader.PacketLength);
        packet = luaPacket;

1.根据包体id（协议id），找到初始化反射注册的协议id，type
2.如果有，说明是C#用协议，protobuf反序列化为对象，加入到事件队列中，等待分发，这样做事为了从其他线程中转回主线程处理
3.如果不存在type，说明是Lua用协议，把字节流保存到SCPacketLua，传递到Lua处理字节流转为Lua中table

网络消息分发

GameFramework.EventPool.HandleEvent

if (m_EventHandlers.TryGetValue(e.Id, out range))
{
    LinkedListNode<EventHandler<T>> current = range.First;
    while (current != null && current != range.Terminal)
    {
        m_CachedNodes[e] = current.Next != range.Terminal ? current.Next : null;

        if (m_NoSenderDict.ContainsKey(e.Id) && m_NoSenderDict[e.Id] == current.Value)
        {
            current.Value(null, e);
        }
        else
        {
            //这里会传递网络handle
            current.Value(sender, e);
        }

        current = m_CachedNodes[e];
    }

    m_CachedNodes.Remove(e);
}
else if (m_DefaultHandler != null)
{
    m_DefaultHandler(sender, e);
}
else if ((m_EventPoolMode & EventPoolMode.AllowNoHandler) == 0)
{
    noHandlerException = true;
}

ReferencePool.Release(e);

1.C#中初始化时反射注册协议id对应handle，例如handle中内容

public class SCLoginHandler : PacketHandlerBase
{

    public override int Id
    {
        get
        {
            return (int)Network.NetMsgID.SCLogin;
        }
    }

    public override void Handle(object sender, Packet packet)
    {
        SCLogin packetImpl = (SCLogin)packet;
        Log.Info("SCLoginHandler name:{0}-passeword{1}", packetImpl.account, packetImpl.password);
    }
}

会在current.Value(sender, e);中调用到 Handle，这里可以把协议转换好的对象，进一步处理
2.未找到协议id对应handle，执行m_DefaultHandler(sender, e);，这里可以在初始化设置委托在lua中执行，把SCPacketLua传递到Lua进一步处理

Lua中发送协议

lua中

function TestSendPlayerInfo()
    --每次协议都是全手写table，未看到可生成协议格式.lua文件，这样不可复用，每次都需要看proto描述，写全部字段。
    --需要在业务module中手写一遍全部协议send函数，确定发送的参数，组装成一个table再发送。这样相当于手动写了一遍
    local data = {
        name = "789",
        level = 123
    }
    LuaEntry.NetworkModule:Send(NetMsgID.CSPlayerInfo,data)
end

function NetworkModule:Send(msgID,data)
    local sProto = MsgID2Proto[msgID]
    if sProto == nil then
        Log.Info("消息ID：{0}找不到需要序列化的proto对象",msgID)
        return
    end
    local bytes = assert(pb.encode(sProto, data))--返回值虽然为string，但是这是字节数组在lua中表达，可以直接传递给C#的byte[]
    local sHex = pb.tohex(bytes)
    Log.Info("Send Hex：{0}",sHex)
    GameEntry.Network:SendByLua(msgID,bytes) --有时候调用不到，生成一次wrap，再清除掉
end

1.lua中声明一个table为packet，里面字段为proto中描述
2.assert(pb.encode(sProto, data)，table序列化二进制数组，返回值虽然为string，但这是字节数组在lua中的表达，可以直接传递到c#的byte[]中https://www.jianshu.com/p/63987134c1ba
C#处理

public static void SendByLua(this NetworkComponent networkComponent, int msgID, byte[] bytes)
{
    CSPacketLua packet = ReferencePool.Acquire<CSPacketLua>();
    packet.m_id = msgID;
    packet.m_bytes = PacketBuffer.GetBufferAndCopyBytes(bytes,bytes.Length);
    packet.Len = (ushort)bytes.Length;
    networkComponent.Send<CSPacketLua>(packet);
}

Lua中接收协议

C#中注册网络委托

public static Action<SCPacketLua> CreateFunc = null;
m_NetworkChannel.SetDefaultHandler(LuaPacketHandler);

在网络消息分发时，未找打id对应handle，再进入到网络委托中处理
GameFramework.EventPool.HandleEvent

 m_DefaultHandler(sender, e);

Lua中执行网络委托内容

SF.NetworkChannelHelper.CreateFunc = handler(self,self.CreatePacket)

function NetworkModule:CreatePacket(packet)
    local luaPacket = packet
    local sProto = MsgID2Proto[luaPacket.Id]
    if sProto == nil then
        Log.Info("消息ID：{0}找不到需要序列化的proto对象",luaPacket.Id)
        return
    end
    local data = assert(pb.decode(sProto, luaPacket.m_bytes.Buffer))
    PrintTable(data,false,true,"CreatePacket")
end

把C#中传递过来的SCPacketLua中的字节流使用lua-protobuf反序列化为table

流程图

GFxLuaProto发送协议流程图

GFxLuaProto接收协议流程图

遇到错误

字节流长度不对

ProtoBuf.ProtoException: Invalid wire-type; this usually means you have over-written a file without truncating or setting the length
使用字节流反序列化错误，检查长度之类
复用流，每次使用完要进行截断SetLength，否则会带入上次长度
流每次写入，都会改变position位置

lua-protobuf中反序列化，默认值问题

如果protobuf的成员值为默认值，序列化后会缺省这部分字节流。
lua中反序列化不出这个member。需要设置lua-protobuf中使用默认值

pb.option "use_default_values" --将默认值表复制到解码目标表中来

安卓测试

从C#发送，C#接收处打印

从Lua发送，Lua接收处打印