Python新手求教：未封装函数/类的脚本能否用unittest做单元测试？

Absolutely! You can absolutely write unit tests for Python scripts that aren’t wrapped in functions or classes—including AWS Glue scripts like the join/relationalize example you mentioned. The key is to make small, non-breaking changes to the script to enable testability, while keeping it fully compatible with AWS Glue’s requirement to run the entire script as the entry point. Here’s how to do it step by step:

1. Refactor the Script (Gently) for Testability #

You don’t have to rewrite the entire script—just extract the core logic into reusable functions. This way, you can test individual pieces without running the full Glue job, while the original script still works exactly as it did before in Glue.

For example, take the original script’s top-level code:

# Original top-level code
glueContext = GlueContext(SparkContext.getOrCreate())
datasource0 = glueContext.create_dynamic_frame.from_catalog(database="db", table_name="table")
datasource1 = glueContext.create_dynamic_frame.from_catalog(database="db", table_name="another_table")
join1 = datasource0.join(keys1=["id"], keys2=["foreign_id"], frame2=datasource1)
relationalized_df = join1.relationalize("root", "s3://output/path")
glueContext.write_dynamic_frame.from_options(frame=relationalized_df, connection_type="s3", ...)

Refactor it into functions, then keep the original execution flow intact at the bottom:

def init_glue_context():
    return GlueContext(SparkContext.getOrCreate())

def load_data(glue_context, db_name, table_name):
    return glue_context.create_dynamic_frame.from_catalog(database=db_name, table_name=table_name)

def join_dynamic_frames(frame1, frame2, key1, key2):
    return frame1.join(keys1=[key1], keys2=[key2], frame2=frame2)

def relationalize_data(frame, root_name, output_path):
    return frame.relationalize(root_name, output_path)

def write_output(glue_context, frame, connection_params):
    glue_context.write_dynamic_frame.from_options(frame=frame, **connection_params)

# Keep the original execution flow for Glue
if __name__ == "__main__":
    ctx = init_glue_context()
    data0 = load_data(ctx, "db", "table")
    data1 = load_data(ctx, "db", "another_table")
    joined_data = join_dynamic_frames(data0, data1, "id", "foreign_id")
    relationalized = relationalize_data(joined_data, "root", "s3://output/path")
    write_output(ctx, relationalized, {"connection_type": "s3", ...})

This change doesn’t affect how Glue runs the script (it still executes the full flow when run as the entry point), but now you can test each function independently.

2. Mock AWS Glue Dependencies for Local Testing #

Since you can’t easily spin up a real Glue context locally, use Python’s built-in unittest.mock library to simulate Glue-specific objects like GlueContext and DynamicFrame. This lets you test your logic without needing an AWS account or Glue environment.

For example, here’s a test case for the join_dynamic_frames function:

import unittest
from unittest.mock import Mock
import your_script_name  # Replace with your script's filename

class TestGlueScript(unittest.TestCase):
    def test_join_dynamic_frames(self):
        # Create mock DynamicFrames
        mock_frame1 = Mock()
        mock_frame2 = Mock()
        mock_joined_frame = Mock()
        
        # Set up the mock join method to return our mock result
        mock_frame1.join.return_value = mock_joined_frame
        
        # Call the function under test
        result = your_script_name.join_dynamic_frames(mock_frame1, mock_frame2, "id", "foreign_id")
        
        # Verify the join was called with the correct parameters
        mock_frame1.join.assert_called_once_with(keys1=["id"], keys2=["foreign_id"], frame2=mock_frame2)
        # Verify we got the expected result
        self.assertEqual(result, mock_joined_frame)

if __name__ == '__main__':
    unittest.main()

For more realistic testing, you can also use the aws-glue-libs package (available via Conda) to create a local Glue context and work with real DynamicFrame objects—this is great for testing data transformation logic that relies on Glue’s specific methods.

3. Run Your Tests Locally #

Once you’ve written your test cases, run them using Python’s unittest module just like any other test suite:

python -m unittest test_your_script.py -v

Key Reassurance for Your Glue Concern #

You mentioned that Glue can only run the entire script as the entry point—this approach doesn’t break that. When Glue executes your script, it runs the full flow defined in the if __name__ == "__main__": block, just like it did before. The refactored functions are just there to make testing easier; Glue doesn’t need to run them individually.

内容的提问来源于stack exchange，提问作者Michael Z