如何使用Oozie自动化Shell脚本及定时执行Pig脚本？

Got it, let's tackle your problem head-on—automating those repetitive Shell scripts for Flume data, plus scheduling Pig scripts to run at fixed intervals using Oozie. I'll break this down into two clear sections so you can follow along easily.

Oozie's Shell action makes it straightforward to run your custom Shell scripts. Here's the step-by-step process:

1. Prep Your Shell Script #

First, make sure your Flume data processing script (let's call it process_flume_data.sh) is ready. Upload it to HDFS and grant execution permissions:

hdfs dfs -put process_flume_data.sh /user/your_username/scripts/
hdfs dfs -chmod +x /user/your_username/scripts/process_flume_data.sh

2. Create the Workflow XML #

Make a workflow.xml file—this defines the Oozie workflow that runs your Shell script. Here's a basic template:

<workflow-app name="shell-flume-workflow" xmlns="uri:oozie:workflow:0.5">
    <start to="shell-action"/>
    <action name="shell-action">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>process_flume_data.sh</exec>
            <file>/user/your_username/scripts/process_flume_data.sh#process_flume_data.sh</file>
            <!-- Add arguments if your script needs them -->
            <!-- <argument>arg1</argument> -->
        </shell>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

• The <file> tag pulls your script from HDFS and makes it available to the action.
• Uncomment <argument> lines if your script requires input parameters.

3. Write the Job Properties File #

Create a job.properties file with all the configuration Oozie needs:

nameNode=hdfs://your-nn-host:8020
jobTracker=your-jt-host:8032
oozie.wf.application.path=${nameNode}/user/your_username/oozie/shell_workflow/
# Optional: Add custom parameters here
# script_arg=flume_data_dir

4. Upload & Submit the Job #

Upload the workflow.xml and job.properties to HDFS (keep them in the same directory as your script or a dedicated workflow folder):

hdfs dfs -put workflow.xml job.properties /user/your_username/oozie/shell_workflow/

Then submit the Oozie job:

oozie job -config job.properties -run

Check the job status with:

oozie job -info <your-job-id>

For recurring Pig jobs, you'll use Oozie's Coordinator—it handles the scheduling logic. Here's how to set it up:

1. Prep Your Pig Script #

First, get your Pig script (e.g., flume_data_analysis.pig) ready, upload it to HDFS:

hdfs dfs -put flume_data_analysis.pig /user/your_username/pig_scripts/

2. Create a Pig Workflow XML #

Make a pig_workflow.xml that defines the Pig action:

<workflow-app name="pig-flume-workflow" xmlns="uri:oozie:workflow:0.5">
    <start to="pig-action"/>
    <action name="pig-action">
        <pig xmlns="uri:oozie:pig-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <script>/user/your_username/pig_scripts/flume_data_analysis.pig</script>
            <!-- Add parameters if needed -->
            <!-- <param>INPUT_DIR=/user/flume/data</param> -->
        </pig>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Pig workflow failed: ${wf:errorMessage(wf:lastErrorNode())}</message>
    </kill>
    <end name="end"/>
</workflow-app>

3. Write the Coordinator XML #

Create a coordinator.xml to define the schedule. Let's say you want to run the Pig script every hour:

<coordinator-app name="hourly-pig-coordinator" frequency="${coord:hours(1)}" start="2024-05-20T00:00Z" end="2024-12-31T23:59Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.4">
    <action>
        <workflow>
            <app-path>${nameNode}/user/your_username/oozie/pig_coordinator/workflow/</app-path>
            <!-- Pass parameters to the workflow -->
            <configuration>
                <property>
                    <name>jobTracker</name>
                    <value>${jobTracker}</value>
                </property>
                <property>
                    <name>nameNode</name>
                    <value>${nameNode}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>

• Adjust frequency to your needs (e.g., ${coord:minutes(30)} for every 30 minutes).
• The start and end times use ISO 8601 format (UTC by default; change timezone if needed).

4. Coordinator Job Properties #

Create coord_job.properties:

nameNode=hdfs://your-nn-host:8020
jobTracker=your-jt-host:8032
oozie.coord.application.path=${nameNode}/user/your_username/oozie/pig_coordinator/
# Optional: Set timezone if not using UTC
# oozie.timezone=America/New_York

5. Upload & Submit the Coordinator Job #

Upload all files to HDFS:

hdfs dfs -put pig_workflow.xml /user/your_username/oozie/pig_coordinator/workflow/
hdfs dfs -put coordinator.xml coord_job.properties /user/your_username/oozie/pig_coordinator/

Submit the coordinator job:

oozie job -config coord_job.properties -run

Check its status with:

oozie job -info <your-coord-job-id>

• Permissions: Make sure your user has read/write access to all HDFS paths and YARN submission rights.
• Testing: Always run the workflow once (without the coordinator) to confirm it works before scheduling.
• Notifications: Add an email action to your workflow to get alerts on success/failure—super helpful for monitoring.
• Time Zones: Oozie defaults to UTC, so adjust the timezone parameter in the coordinator if your schedule uses local time.

内容的提问来源于stack exchange，提问作者Kshitiz Katiyar