Module 19: Distributed Computing for Soil Process Simulation

Parallelize computationally intensive soil models using MPI and distributed frameworks. Handle load balancing for heterogeneous workloads across HPC clusters.

The course objective is to parallelize and scale computationally intensive soil process models for execution on High-Performance Computing (HPC) clusters. Students will master both high-level distributed frameworks like Dask for data parallelism and the low-level Message Passing Interface (MPI) for tightly-coupled model parallelism. A key focus will be on designing and implementing load-balancing strategies to handle the heterogeneous workloads characteristic of real-world soil simulations.

This module provides the computational horsepower for the physics-based modeling aspects of the curriculum. While Module 14 focused on cloud-native infrastructure for data-driven ML, this module tackles the different but equally critical challenge of large-scale scientific simulation. The ability to run complex models of water flow, nutrient cycling, and carbon dynamics in parallel is essential for creating the synthetic data for Physics-Informed Neural Networks (Module 53) and for running the large-scale "what-if" scenarios needed for Policy Decision Support Tools (Module 88).

Hour 1-2: The Computational Wall: Why and When to Go Parallel 🧱

Learning Objectives:

• Differentiate between data parallelism and model parallelism.
• Understand the architectural differences between a cloud-native K8s cluster and a traditional HPC cluster.
• Analyze a soil simulation problem and determine the appropriate parallelization strategy.

Content:

• The Simulation Bottleneck: Many critical soil models (e.g., HYDRUS for water flow, DNDC for biogeochemistry) are too slow or memory-intensive to run for large areas or long time periods on a single computer.
• Two Flavors of Parallelism:
  • Data Parallelism (Pleasingly Parallel): Running the same model thousands of times with different inputs (e.g., different climate scenarios, different soil types). This is like having thousands of researchers working independently.
  • Model Parallelism (Tightly Coupled): Splitting a single, large simulation across many computers that must constantly communicate. This is like a large team of researchers that needs to have meetings every five minutes.
• HPC vs. Cloud: A comparison of the two dominant paradigms for large-scale computing.
  • HPC (Slurm/PBS): Optimized for long-running, tightly-coupled jobs with high-speed interconnects.
  • Cloud (Kubernetes): Optimized for services, elasticity, and fault tolerance.

Conceptual Design Lab:

• You are given two tasks:
  1. A Monte Carlo analysis that requires running a soil carbon model 10,000 times with different randomized parameters.
  2. A high-resolution 3D simulation of water infiltrating a single, large field plot.
• For each task, you must design a parallelization strategy, choose the appropriate paradigm (data vs. model parallelism), and justify whether an HPC cluster or a Kubernetes cluster would be a better fit.

Hour 3-4: Easy Wins: Data Parallelism with Dask 🚀

Learning Objectives:

• Understand the core concepts of Dask: lazy evaluation and task graphs.
• Use dask.delayed to parallelize existing, single-threaded Python code with minimal changes.
• Visualize a parallel computation using the Dask dashboard.

Content:

• Dask: Parallel Python in Python: A native Python library for distributed computing that integrates seamlessly with libraries like NumPy, pandas, and scikit-learn.
• The Power of Laziness: Dask builds a graph of all the computations you want to do, and only executes it when you explicitly ask for the result. This allows it to optimize the entire workflow.
• The @dask.delayed Decorator: The magic wand for custom functions. By adding this single line of code to your existing soil simulation function, you can instantly turn it into a building block for a parallel Dask graph, without needing to rewrite the function's internal logic.

Hands-on Lab:

• Take a simple (but artificially slow) Python function that simulates one year of soil organic matter decomposition.
• Write a standard for loop to run this simulation for 100 different soil plots, and time it.
• Now, using dask.delayed and a Dask LocalCluster, rewrite the loop to build a list of delayed tasks.
• Execute the tasks in parallel with dask.compute() and time the result.
• Use the Dask dashboard (available at localhost:8787) to watch the tasks being executed across all your CPU cores.

Hour 5-6: The Hard Core: Introduction to the Message Passing Interface (MPI) 💬

Learning Objectives:

• Understand the MPI programming model of communicating sequential processes.
• Write a basic mpi4py application that uses rank and size.
• Implement fundamental point-to-point communication with send and recv.

Content:

• When You Need Full Control: For model parallelism, where different parts of a simulation must precisely exchange information, high-level tools like Dask are not enough. We need direct control over the network messages.
• MPI: The Lingua Franca of HPC: A standardized API for passing messages between processes running on different nodes of a cluster.
• Core MPI Concepts:
  • World / Communicator: The group of all processes working on a job.
  • Rank: The unique ID of a single process, from 0 to N-1.
  • Size: The total number of processes (N).
• Point-to-Point Communication:
  • comm.send(data, dest=rank): Send a Python object to a specific destination process.
  • data = comm.recv(source=rank): Block and wait to receive an object from a specific source process.
• Running MPI Code: mpiexec -n 8 python my_script.py

MPI "Hello, World!" Lab:

• Write a Python script using mpi4py.
• The script will have each process get its rank and the world size.
• Each process will print a message like "Hello from rank 3 of 8!".
• Then, implement a simple exchange: rank 0 will create a dictionary and send it to rank 1. Rank 1 will receive it and print its contents.

Hour 7-8: Model Parallelism: Domain Decomposition & Halo Exchange ⚃

Learning Objectives:

• Implement the domain decomposition strategy to split a spatial problem across MPI processes.
• Understand the concept of "ghost cells" or "halo regions."
• Implement a halo exchange to communicate boundary conditions between neighboring processes.

Content:

• Splitting the World: The most common pattern for parallelizing spatial simulations. If you have a 100x100 grid, you can give a 25x100 strip to each of 4 MPI processes.
• The Boundary Problem: To calculate the next time step for a cell at the edge of its strip, a process needs to know the value of the cell in the neighboring strip (which is owned by another process).
• The Halo Exchange: Each process allocates extra memory cells around its local domain—the "ghost cells" or "halo." Before each time step, processes engage in a highly choreographed send and recv dance to populate these halos with the data from their neighbors.

Hands-on Lab:

• Implement a 1D domain decomposition for a simple heat diffusion model using mpi4py.
• Each MPI process will manage a sub-section of a 1D array representing a metal rod.
• The core of the lab is to write the halo exchange logic: each process i (except the ends) will send its leftmost cell to process i-1 and its rightmost cell to process i+1, while simultaneously receiving data from them to populate its own halo.

Hour 9-10: The Unbalanced World: Handling Heterogeneous Workloads ⚖️

Learning Objectives:

• Identify the causes and consequences of load imbalance in parallel simulations.
• Differentiate between static and dynamic load-balancing strategies.
• Implement a dynamic task-based approach to naturally balance workloads.

Content:

• The Straggler Problem: If one process is given a much harder piece of work (e.g., simulating a complex clay soil vs. a simple sand), all other processes will finish their work and sit idle, waiting for the one "straggler." This kills parallel efficiency.
• Static Balancing: If you know the cost distribution beforehand, you can do a smarter domain decomposition, giving smaller regions to the processes that will simulate complex areas. This is difficult to get right.
• Dynamic Balancing (The Manager/Worker Pattern): A more robust approach. Break the problem into many small tasks. A "manager" process hands out tasks to "worker" processes. When a worker finishes, it requests a new task. This ensures that fast workers simply do more tasks, and no one sits idle. High-level frameworks like Dask and Ray have this built-in.

Dynamic Load Balancing Lab:

• Create a Dask application where the work is a list of 1000 tasks.
• The runtime of each task will be drawn from a skewed distribution (e.g., a log-normal distribution), so some tasks are 10x longer than others.
• Use the Dask dashboard to visualize the execution. You will see that as soon as a worker core finishes a short task, the scheduler immediately gives it another one, ensuring all cores stay busy and the total job finishes as quickly as possible.

Hour 11-12: Running on a Cluster: The Slurm Scheduler 🖥️

Learning Objectives:

• Understand the role of a workload manager like Slurm in an HPC environment.
• Write a Slurm batch script to request resources and launch a parallel job.
• Use tools like dask-jobqueue to programmatically create Dask clusters on an HPC system.

Content:

• The Gatekeeper of the HPC: You don't just run code on an HPC cluster; you submit a "job" to a scheduler like Slurm, which decides when and where to run it.
• The Slurm Batch Script: A shell script containing #SBATCH directives that request resources:
  • --nodes=4: "I need 4 machines."
  • --ntasks-per-node=32: "I want to run 32 processes on each of those machines."
  • --time=01:30:00: "My job will run for at most 1 hour and 30 minutes."
• Launching Jobs: sbatch my_script.sh to submit, squeue to check status, scancel to kill.
• Dynamic Clusters with dask-jobqueue: A powerful library that lets your Python script act as a client that submits jobs to Slurm to start Dask workers, creating an elastic cluster tailored to your computation.

Slurm Lab:

• Write a simple Slurm batch script (#SBATCH ...) that uses mpiexec to launch the MPI "Hello, World!" script from Hour 6.
• Then, write a Python script that uses dask-jobqueue to create a SLURMCluster object. The script will then connect a client to this cluster, run a simple Dask computation, and scale the cluster down.

Hour 13-14: Measuring Performance: Scaling and Profiling ⏱️

Learning Objectives:

• Define and measure strong and weak scaling for a parallel application.
• Understand Amdahl's Law and the limits of parallel speedup.
• Use profiling tools to identify performance bottlenecks in parallel code.

Content:

• Is It Worth It? We need to rigorously measure if our parallelization effort was successful.
• Scaling Analysis:
  • Strong Scaling: "I keep the problem size fixed and add more processors. How much faster does it get?"
  • Weak Scaling: "I increase the problem size and the number of processors together. Can I solve a 10x bigger problem with 10x the cores in the same amount of time?"
• Amdahl's Law: The fundamental theorem of parallel computing. The speedup of a program is ultimately limited by the fraction of the code that must be run serially.
• Profiling: Identifying the slowest parts of your code. For MPI, this means identifying if the bottleneck is computation on the nodes or communication between them.

Performance Analysis Lab:

• Take the 1D MPI heat diffusion code from the halo exchange lab.
• Run it on 1, 2, 4, 8, and 16 processes for a fixed problem size.
• For each run, record the total execution time.
• Plot the speedup (Time(1) / Time(N)) and efficiency (Speedup / N) as a function of the number of processes.
• Analyze the plot: Does it scale linearly? When does the efficiency start to drop off, and why?

Hour 15: Capstone: Parallelizing a Heterogeneous Watershed Simulation 🏆

Final Challenge: You are given a single-threaded Python model that simulates nutrient transport across a 2D landscape. The landscape is defined by a grid, where each cell has a different soil type. The computational cost of the simulation is highly dependent on the soil type, with clay soils being 10 times slower to simulate than sandy soils. The model is too slow to run at the desired resolution.

Your Mission:

1. Analyze and Strategize: Examine the model's code. Is communication between adjacent grid cells required at every time step? Based on this, choose a parallelization strategy: a high-level, dynamic task-based approach (Dask) or a low-level, tightly-coupled domain decomposition (MPI). Write a clear justification for your choice.
2. Implement the Parallelization:
   • If Dask: Decompose the landscape into many small, independent patches. Use dask.delayed to create a task graph. Dask's scheduler will handle the load balancing automatically.
   • If MPI: Implement a 2D domain decomposition and halo exchange. You must also implement a simple static load balancing scheme by giving smaller grid regions to the MPI ranks that will be handling the slow, clay-heavy areas.
3. Deploy on a Cluster: Write a launch script (e.g., a Slurm batch script or a Python script using dask-jobqueue) to run your parallel simulation on a multi-node cluster environment.
4. Benchmark and Analyze: Perform a scaling analysis. Run the simulation on an increasing number of cores and measure the speedup. Create a plot to visualize the performance and efficiency of your parallel implementation.

Deliverables:

• All documented Python code for the parallelized model.
• The launch script(s).
• A final report in a Jupyter Notebook or markdown format that includes:
  • Your justification for the chosen parallelization strategy.
  • The scaling plot and a detailed analysis of its performance, including a discussion of any bottlenecks.
  • A critical comparison of how your implementation specifically addressed the load-balancing challenge posed by the heterogeneous soil types.

Assessment Criteria:

• The correctness and quality of the parallel implementation.
• The strategic justification for the chosen parallelization approach.
• The rigor and insight of the performance and scaling analysis.
• The effectiveness of the solution in handling the specified load-balancing problem.