ABAP环境下高性能解析符合RFC4180规范的未知结构CSV文件咨询

核心需求 #

需在ABAP中实现最高性能的大型CSV文件解析能力，满足以下要求：

• 提前未知CSV结构，解析结果需为string_table组成的内表或同类结构
• 解析逻辑完全符合RFC4180规范
• 不可使用特定场景专属调用

现有方案缺陷梳理 #

社区博客方案 #

对公开的CSV处理相关博客内容进行验证后，发现所有方案均存在不足：

• 自行编码实现：给出的代码示例不完善，无法直接使用
• 调用KCD_CSV_FILE_TO_INTERN_CONVERT读取文件：属于特定方案专属调用，并非全环境可用，字段过大时会触发dump
• 结合RTTI动态编程与函数模块RSDS_CONVERT_CSV：需要提前知晓CSV结构，不符合需求
• 使用CL_RSDA_CSV_CONVERTER类：同样要求提前知晓CSV结构，不符合需求

开源方案 #

对GitHub上首个可用的ABAP CSV解析开源方案ZwdCSV进行验证后，发现不符合要求：

• 代码中包含宏，不可接受
• 要求提前知晓CSV结构，不符合需求

其他自研尝试 #

• 正则表达式实现：大文件处理时性能过差，无法满足大型CSV解析需求
• 移植Java CSV解析代码到ABAP：ABAP遍历字符串的效率远低于Java，性能不达标

当前自研实现 #

后续采用拆分/计数的实现思路，目前该方案性能表现最佳，代码如下：

REPORT z_csv_test.

CLASS lcl_csv_parser DEFINITION CREATE PRIVATE.

  PUBLIC SECTION.
    TYPES:
             tt_string_matrix TYPE STANDARD TABLE OF string_table WITH EMPTY KEY.
    CLASS-METHODS:
      create
        IMPORTING
          !iv_delimiter      TYPE string DEFAULT  '"'
          !iv_separator      TYPE string DEFAULT  ','
          !iv_line_separator TYPE abap_cr_lf DEFAULT cl_abap_char_utilities=>cr_lf
        RETURNING
          VALUE(r_result)    TYPE REF TO lcl_csv_parser.
    METHODS:
      parse
        IMPORTING
          iv_string       TYPE string
        RETURNING
          VALUE(r_result) TYPE tt_string_matrix,
      constructor
        IMPORTING
          !iv_delimiter      TYPE string
          !iv_separator      TYPE string
          !iv_line_separator TYPE string.
  PROTECTED SECTION.
  PRIVATE SECTION.
    DATA:
      gv_delimiter         TYPE string,
      gv_separator         TYPE string,
      gv_line_separator    TYPE string,
      gv_escaped_delimiter TYPE string.
    METHODS parse_line_to_string_table
      IMPORTING
        iv_line         TYPE string
      RETURNING
        VALUE(r_result) TYPE string_table.
ENDCLASS.

CLASS lcl_csv_parser IMPLEMENTATION.

  METHOD create.
    r_result = NEW #(
      iv_delimiter      = iv_delimiter
      iv_line_separator = CONV #( iv_line_separator )
      iv_separator      = iv_separator  ).
  ENDMETHOD.

  METHOD constructor.
    me->gv_delimiter = iv_delimiter.
    me->gv_separator = iv_separator.
    me->gv_line_separator = iv_line_separator.
    me->gv_escaped_delimiter = |{ iv_delimiter }{ iv_delimiter }|.
  ENDMETHOD.

  METHOD parse.
    "get the lines
    SPLIT iv_string AT me->gv_line_separator INTO TABLE DATA(lt_lines).
    DATA lx_open_line TYPE abap_bool VALUE abap_false.
    DATA lv_current_line TYPE string.

    LOOP AT lt_lines ASSIGNING FIELD-SYMBOL(<ls_line>).

      FIND ALL OCCURRENCES OF me->gv_delimiter IN <ls_line> IN CHARACTER MODE MATCH COUNT DATA(lv_count).
      IF ( lv_count MOD 2 )  = 1.
        IF lx_open_line = abap_true.
          lv_current_line = |{ lv_current_line }{ me->gv_line_separator }{ <ls_line> }|.
          lx_open_line = abap_false.
          APPEND parse_line_to_string_table( lv_current_line ) TO r_result.
        ELSE.
          lv_current_line = <ls_line>.
          lx_open_line = abap_true.
        ENDIF.
      ELSE.
        IF lx_open_line = abap_true.
          lv_current_line = |{ lv_current_line }{ me->gv_line_separator }{ <ls_line> }|.
        ELSE.
          APPEND parse_line_to_string_table( <ls_line> ) TO r_result.
        ENDIF.

      ENDIF.
    ENDLOOP.

  ENDMETHOD.

  METHOD parse_line_to_string_table.
    SPLIT iv_line AT me->gv_separator INTO TABLE DATA(lt_line).
    DATA lx_open_field TYPE abap_bool VALUE abap_false.
    DATA lv_current_field TYPE string.
    LOOP AT lt_line ASSIGNING FIELD-SYMBOL(<ls_field>).
      FIND ALL OCCURRENCES OF me->gv_delimiter IN <ls_field> IN CHARACTER MODE MATCH COUNT DATA(lv_count).
      IF ( lv_count MOD 2 ) = 1.
        IF lx_open_field = abap_true.
          lv_current_field = |{ lv_current_field }{ me->gv_separator }{ <ls_field> }|.
          lx_open_field = abap_false.
          APPEND lv_current_field TO r_result.
        ELSE.
          lv_current_field = <ls_field>.
          lx_open_field = abap_true.
        ENDIF.
      ELSE.
        IF lx_open_field = abap_true.
          lv_current_field = |{ lv_current_field }{ me->gv_separator }{ <ls_field> }|.
        ELSE.
          APPEND <ls_field> TO r_result.
        ENDIF.
      ENDIF.

    ENDLOOP.

    REPLACE ALL OCCURRENCES OF me->gv_escaped_delimiter IN TABLE r_result WITH me->gv_delimiter.

  ENDMETHOD.

ENDCLASS.
CLASS lcl_test_csv_parser DEFINITION
  FINAL
  CREATE PUBLIC .

  PUBLIC SECTION.
    CLASS-METHODS run.
    CLASS-METHODS get_file
      RETURNING VALUE(r_result) TYPE string.

  PROTECTED SECTION.
  PRIVATE SECTION.

ENDCLASS.

CLASS lcl_test_csv_parser IMPLEMENTATION.

  METHOD get_file.
    DATA lv_file_line TYPE string.
    DO 10 TIMES.
      lv_file_line = |"1234,{ cl_abap_char_utilities=>cr_lf }567890",{ lv_file_line }|.
    ENDDO.
    lv_file_line = lv_file_line && cl_abap_char_utilities=>cr_lf.

    DATA(lt_file_as_table) = VALUE string_table(
        FOR i = 1 THEN  i + 1 UNTIL  i = 1000000
            ( lv_file_line ) ).

    CONCATENATE LINES OF lt_file_as_table INTO r_result.
  ENDMETHOD.

  METHOD run.
    DATA lv_prepare_start TYPE timestampl.
    GET TIME STAMP FIELD lv_prepare_start.

    DATA(lv_file) = get_file(  ).

    DATA lv_prepare_end TYPE timestampl.
    GET TIME STAMP FIELD lv_prepare_end.

    WRITE |Preparation took { cl_abap_tstmp=>subtract( tstmp1 = lv_prepare_end tstmp2 = lv_prepare_start ) }|.

    DATA lv_parse_start TYPE timestampl.
    GET TIME STAMP FIELD lv_parse_start.
    DATA(lo_parser) = lcl_csv_parser=>create(  ).
    DATA(lt_file) = lo_parser->parse( lv_file  ).
    DATA lv_parse_end TYPE timestampl.
    GET TIME STAMP FIELD lv_parse_end.

    WRITE |Parse took { cl_abap_tstmp=>subtract( tstmp1 = lv_parse_end tstmp2 = lv_parse_start ) }|.

  ENDMETHOD.

ENDCLASS.

START-OF-SELECTION.
  lcl_test_csv_parser=>run( ).

请问是否有更优的实现方式？

内容的提问来源于stack exchange，提问作者Dmitry Yudin