如何在ruamel.yaml中实现tuple与np.array的隐式解析器及健壮表示器?
问题:优化ruamel.yaml对tuple、numpy数组及复数的隐式解析与往返序列化
我有一个项目,要求用户手动编写YAML文件,其中部分条目需要以tuple或numpy数组格式填写。Python内部会区分tuple和list来提供便捷接口,比如(1, 2, 3)和[1, 2, 3]是不同的。为提升用户体验,希望用户能直接用括号输入tuple(如name: (1,2,3)),用np.array([1,2,3])格式输入numpy数组(已接受精度损失的妥协)。我使用ruamel.yaml是因为它能保留注释。
目前已实现可用功能,但解析部分不够规范:未使用合适的隐式解析器,而是用了不够健壮的eval。以下是最小可运行示例,注释标注了实现不够健壮或不规范的地方:
import sys import numpy as np import ruamel.yaml def _tupleRepresenter(dumper, data): # TODO: Make this more robust return dumper.represent_scalar(u'tag:yaml.org,2002:str', str(data)) def _numpyRepresenter(dumper, data): # TODO: Make this more robust as_string = 'np.array(' + np.array2string(data, max_line_width=np.inf, precision=16, prefix='np.array(', separator=', ', suffix=')') + ')' return dumper.represent_scalar(u'tag:yaml.org,2002:str', as_string) def load_yaml(file): # TODO: Resolve tuples and arrays properly when loading yaml = ruamel.yaml.YAML() yaml.Representer.add_representer(tuple, _tupleRepresenter) yaml.Representer.add_representer(np.ndarray, _numpyRepresenter) return yaml.load(file) def dump_yaml(data, file): yaml = ruamel.yaml.YAML() yaml.Representer.add_representer(tuple, _tupleRepresenter) yaml.Representer.add_representer(np.ndarray, _numpyRepresenter) return yaml.dump(data, file) yaml_file = """ test_tuple: (1, 2, 3) test_array: np.array([4,5,6]) """ data = load_yaml(yaml_file) data['test_tuple'] = eval(data['test_tuple']) # This feels dirty data['test_array'] = eval(data['test_array']) # This feels dirty dump_yaml(data, sys.stdout) # test_tuple: (1, 2, 3) # test_array: np.array([4, 5, 6])
我希望能优化该实现,使用规范的隐式解析器、健壮的表示器,更符合ruamel.yaml的设计意图。
更新
在帮助下,我实现了几乎完整的功能(暂时忽略用正规解析替代eval的需求),但仍存一个问题:新标签被导出为字符串,重新加载时会变为字符串,无法实现多次往返解析。如何解决该问题?以下是最小可运行示例:
import sys import numpy as np import ruamel.yaml # TODO: Replace evals by actual parsing # TODO: Represent custom types without the string quotes _tuple_re = "^(?:\\((?:.|\\n|\\r)*,(?:.|\\n|\\r)*\\){1}(?: |\\n|\\r)*$)" _array_re = "^(?:(np\\.|)array\\(\\[(?:.|\\n|\\r)*,(?:.|\\n|\\r)*\\]\\){1}(?: |\\n|\\r)*$)" _complex_re = "^(?:(?:\\d+(?:(?:\\.\\d+)?(?:e[+\\-]\\d+)?)?)?(?: *[+\\-] *))?(?:\\d+(?:(?:\\.\\d+)?(?:e[+\\-]\\d+)?)?)?[jJ]$" def _tuple_constructor(self, node): return eval(self.construct_scalar(node)) def _array_constructor(self, node): value = node.value if not value.startswith('np.'): value = 'np.' + value return eval(value) def _complex_constructor(self, node): return eval(node.value) def _tuple_representer(dumper, data): return dumper.represent_scalar(u'tag:yaml.org,2002:str', str(data)) def _array_representer(dumper, data): as_string = 'np.array(' + np.array2string(data, max_line_width=np.inf, precision=16, prefix='np.array(', separator=', ', suffix=')') + ')' as_string = as_string.replace(' ', '').replace(',', ', ') return dumper.represent_scalar(u'tag:yaml.org,2002:str', as_string) def _complex_representer(dumper, data): repr = str(data).replace('(', '').replace(')', '') return dumper.represent_scalar(u'tag:yaml.org,2002:str', repr) custom_types = { '!tuple': {'re':_tuple_re, 'constructor': _tuple_constructor, 'representer':_tuple_representer, 'type': tuple, 'first':list('(') }, '!nparray': {'re':_array_re, 'constructor': _array_constructor, 'representer':_array_representer, 'type': np.ndarray, 'first':list('an') }, '!complex': {'re':_complex_re, 'constructor': _complex_constructor, 'representer':_complex_representer, 'type': complex, 'first':list('0123456789+-jJ')}, } def load_yaml(file): yaml = ruamel.yaml.YAML() for tag,ct in custom_types.items(): yaml.Constructor.add_constructor(tag, ct['constructor']) yaml.Resolver.add_implicit_resolver(tag, ruamel.yaml.util.RegExp(ct['re']), ct['first']) yaml.Representer.add_representer(ct['type'], ct['representer']) return yaml.load(file) def dump_yaml(data, file): yaml = ruamel.yaml.YAML() for tag,ct in custom_types.items(): yaml.Constructor.add_constructor(tag, ct['constructor']) yaml.Resolver.add_implicit_resolver(tag, ruamel.yaml.util.RegExp(ct['re']), ct['first']) yaml.Representer.add_representer(ct['type'], ct['representer']) return yaml.dump(data, file) yaml_file = """ test_tuple: (1, 2, 3) test_array: array([4.0,5+0j,6.0j]) test_complex: 3 + 2j """ data = load_yaml(yaml_file) dump_yaml(data, sys.stdout) # test_tuple: '(1, 2, 3)' # test_array: 'np.array([4.+0.j, 5.+0.j, 0.+6.j])' # test_complex: '3+2j'
感谢您的帮助!
内容的提问来源于stack exchange,提问作者JeanOlivier
相关产品推荐
相关产品推荐

