含可选值的Avro Schema序列化JS对象失败，如何解决？

问题描述 #

尝试用Avro Schema序列化JavaScript对象时，遇到联合类型字段的异常：当字段值为null时可正常序列化，但传入数字时会抛出invalid ["null","float"]错误。相关代码及错误信息如下：

关联Schema #

{
  "type": "record",
  "name": "incoming_telemetry",
  "doc": "Telemetry message from herby device",
  "fields": [
    {"name": "deviceId", "type": "string"},
    {"name": "timestamp", "type": "string"},
    {"name": "waterTableRange", "type": ["null", "float"]},
    {"name": "batteryCapacity", "type": ["null", "float"]},
    {"name": "batteryCurrent", "type": ["null", "float"]},
    {"name": "solarVoltage", "type": ["null", "float"]},
    {"name": "batteryVoltage", "type": ["null", "float"]},
    {"name": "temperature", "type": ["null", "float"]},
    {"name": "wifiStrength", "type": ["null", "float"]},
    {"name": "flowRate", "type": ["null", "float"]},
    {"name": "dummy", "type": ["null", "float"]}
  ]
}

序列化脚本 #

const avro = require('avro-js');
const schema = avro.parse('publishToAnalytics/incoming_telemetry_v1.0.avsc')
const data = {
    deviceId: 'test',
    timestamp: new Date().toDateString(),
    waterTableRange: 1.1,
    batteryCapacity: 2.2,
    batteryCurrent: 3.3,
    solarVoltage: 4.4,
    batteryVoltage: 5.5,
    temperature: 6.6,
    wifiStrength: 7.7,
    flowRate: 8.8,
};
const dataBuffer = schema.toBuffer(data) 

错误信息 #

Uncaught Error Error: invalid ["null","float"]: 1.1
    at throwInvalidError (/home/marc/coding/boum/iot/node_modules/avro-js/lib/schemas.js:2204:9)
    at UnionType._write (/home/marc/coding/boum/iot/node_modules/avro-js/lib/schemas.js:803:7)
    at writePet (<eval>/VM46947624:5:82)
    at Type.toBuffer (/home/marc/coding/boum/iot/node_modules/avro-js/lib/schemas.js:259:8)
    at <anonymous> (/home/marc/coding/boum/iot/publishToAnalytics/publishToAnalytics.js:16:18)
    at Module._compile (internal/modules/cjs/loader:1101:14)
    at Module._extensions..js (internal/modules/cjs/loader:1153:10)
    at Module.load (internal/modules/cjs/loader:981:32)
    at Module._load (internal/modules/cjs/loader:822:12)
    at executeUserEntryPoint (internal/main/run_main:81:12)
    at <anonymous> (internal/main/run_main_module:17:47)

问题原因 #

JavaScript中所有数字默认都是双精度浮点数（对应Avro的double类型），而avro-js库对Avro的float类型校验严格，直接传入JS数字会被识别为double，不符合["null", "float"]的类型要求，因此抛出错误。

解决方案 #

方案1：修改Schema，将float替换为double #

这是最贴合JS类型系统的方案，因为JS本身没有单精度浮点数，修改后的字段类型示例如下：

{"name": "waterTableRange", "type": ["null", "double"]},
{"name": "batteryCapacity", "type": ["null", "double"]},
// 其余同类型字段同理替换

修改后无需改动序列化脚本，直接传入JS数字即可正常完成序列化。

方案2：保留float类型，用avro-js类型包装器包装数字 #

如果必须使用Avro的float类型，需要将JS数字用avro-js提供的Float类型包装，修改后的脚本如下：

const avro = require('avro-js');
const schema = avro.parse('publishToAnalytics/incoming_telemetry_v1.0.avsc')
// 引入Float类型包装器
const FloatType = avro.types.Float;

const data = {
    deviceId: 'test',
    timestamp: new Date().toDateString(),
    waterTableRange: new FloatType(1.1),
    batteryCapacity: new FloatType(2.2),
    batteryCurrent: new FloatType(3.3),
    solarVoltage: new FloatType(4.4),
    batteryVoltage: new FloatType(5.5),
    temperature: new FloatType(6.6),
    wifiStrength: new FloatType(7.7),
    flowRate: new FloatType(8.8),
};
const dataBuffer = schema.toBuffer(data) 

包装后的数字会被avro-js识别为合法的float类型，顺利通过校验。

内容的提问来源于stack exchange，提问作者Mr P