Welcome to today’s comprehensive guide on Handling Dynamic Web Tables with Pagination in Selenium with Java. Building upon our previous session where we explored handling static tables, this tutorial delves into managing dynamic tables that incorporate pagination. Dynamic tables are prevalent in modern web applications, presenting challenges due to their changing nature and the presence of multiple pages of data. Mastering the interaction with such tables is crucial for developing robust and efficient automation scripts.

Table of Contents

1. Introduction

Dynamic web tables with pagination are a staple in many web applications, presenting data across multiple pages with varying numbers of rows. Automating interactions with such tables involves not only navigating through different pages but also accurately extracting and processing data from each page. This guide provides a step-by-step approach to effectively handle these tables using Selenium WebDriver in Java.

2. Understanding Dynamic Tables with Pagination

2.1. Characteristics of Dynamic Tables

Dynamic Web Tables differ from static tables in that their content and structure can change based on user interactions or backend processes. Key characteristics include:

  • Variable Number of Rows: Rows can be added or removed dynamically.
  • Pagination: Data is divided across multiple pages, each displaying a subset of the total data.
  • Dynamic Content: Content within cells can change without a full page reload.

2.2. Challenges with Pagination

Handling pagination introduces several challenges:

  • Dynamic Page Counts: The total number of pages may change based on data updates.
  • Navigation Controls: Locating and interacting with pagination controls (e.g., Next, Previous, Page Numbers) can be complex.
  • Synchronization: Ensuring that data loads completely before interacting with it to avoid flaky tests.

3. Approach to Handling Dynamic Tables with Pagination

To effectively automate interactions with dynamic tables with pagination, follow these structured steps:

3.1. Determining the Total Number of Pages

Objective: Identify the total number of pages in the table to iterate through them.

Approach:

  1. Locate the Pagination Element: Find the web element that displays pagination information (e.g., “Showing 1 to 10 of 1909 pages”).
  2. Extract the Total Page Count: Parse the extracted text to obtain the total number of pages.

Sample Scenario:

Suppose the pagination text is: “Showing 1 to 10 of 1909 pages”. We need to extract the number 1909 as the total number of pages.

Implementation Steps:

1. Capture the Pagination Text:

WebElement paginationElement = driver.findElement(By.xpath("//div[@class='pagination-info']"));
String paginationText = paginationElement.getText(); // e.g., "Showing 1 to 10 of 1909 pages"

2. Extract the Total Pages Using String Methods:

// Example paginationText: "Showing 1 to 10 of 1909 pages"
String[] parts = paginationText.split(" ");
String totalPagesStr = parts[5]; // "1909"
int totalPages = Integer.parseInt(totalPagesStr);
System.out.println("Total number of pages: " + totalPages);

Dynamic Extraction Using substring and indexOf:

For more robust extraction, especially when the format may vary slightly:

String paginationText = "Showing 1 to 10 of 1909 pages";

// Extract the starting index of "of " and ending index before " pages"
int startIndex = paginationText.indexOf("of ") + 3; // Start after "of "
int endIndex = paginationText.indexOf(" pages");

// Extract the substring containing the total number of pages
String totalPagesStr = paginationText.substring(startIndex, endIndex);
int totalPages = Integer.parseInt(totalPagesStr);
System.out.println("Total number of pages: " + totalPages);

Objective: Iterate through each page of the table based on the total page count.

Approach:

  1. Loop Through Page Numbers: Use a for loop from page 1 to totalPages.
  2. Click on the Page Number Link: Dynamically construct the XPath to locate the page number and perform a click action.
  3. Handle Conditional Clicks: Since the first page is typically displayed by default, avoid clicking it again.

Implementation Steps:

1. Set Up the Loop:

for (int p = 1; p <= totalPages; p++) { if (p > 1) { // Skip clicking if p is 1, as it's already displayed
        // Construct the XPath for the page number
        String pageXPath = "//ul[@class='pagination']/li/a[text()='" + p + "']";
        WebElement pageLink = driver.findElement(By.xpath(pageXPath));
        pageLink.click();
        
        // Optional: Add explicit wait to ensure page loads
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
        wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//table[@class='data-table']")));
    }
    
    // Proceed to extract data from the current page
    extractDataFromCurrentPage(driver);
}

2. Define the Data Extraction Method:

public static void extractDataFromCurrentPage(WebDriver driver) {
    // Locate the table
    WebElement table = driver.findElement(By.xpath("//table[@class='data-table']/tbody"));
    
    // Find all rows
    List allRows = table.findElements(By.tagName("tr"));
    int numberOfRows = allRows.size();
    
    // Iterate through each row
    for (int r = 1; r <= numberOfRows; r++) {
        // Extract Customer Name (Assuming it's in the 2nd column)
        String customerNameXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[2]";
        String customerName = driver.findElement(By.xpath(customerNameXPath)).getText();
        
        // Extract Email Address (Assuming it's in the 3rd column)
        String emailXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[3]";
        String email = driver.findElement(By.xpath(emailXPath)).getText();
        
        // Extract Status (Assuming it's in the 5th column)
        String statusXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[5]";
        String status = driver.findElement(By.xpath(statusXPath)).getText();
        
        // Print the extracted data
        System.out.println("Customer Name: " + customerName + "\tEmail: " + email + "\tStatus: " + status);
        
        // Optionally, interact with checkboxes if needed
        // String checkboxXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[1]/input[@type='checkbox']";
        // WebElement checkbox = driver.findElement(By.xpath(checkboxXPath));
        // checkbox.click();
    }
}

Notes:

  • Explicit Waits: Utilize explicit waits (WebDriverWait) instead of Thread.sleep() for better synchronization.
  • Dynamic XPath Construction: By embedding the loop variable p into the XPath, the script dynamically interacts with each page number.

3.3. Extracting Data from Each Page

Objective: Retrieve relevant data (e.g., Customer Name, Email Address, Status) from each page of the table.

Approach:

  1. Locate the Table Body: Identify the <tbody> element containing the table rows.
  2. Iterate Through Each Row: Loop through all <tr> elements within the <tbody>.
  3. Extract Desired Columns: For each row, extract data from specific <td> elements based on their column positions.

Implementation Steps:

As illustrated in the extractDataFromCurrentPage method above, iterate through each row and extract data from the desired columns using dynamic XPath expressions.

Handling Checkboxes:

If your automation requires interacting with checkboxes (e.g., selecting all customers), include steps to locate and click the checkbox within each row.

// Example: Selecting the checkbox in the first column
String checkboxXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[1]/input[@type='checkbox']";
WebElement checkbox = driver.findElement(By.xpath(checkboxXPath));
checkbox.click();

3.4. Handling Dynamic Elements (e.g., Bootstrap Tables)

Modern web applications often use frameworks like Bootstrap to enhance the UI, resulting in tables constructed with <div> tags instead of traditional <table>, <tr>, and <td> tags. Handling such tables requires a slightly different approach.

Characteristics of Bootstrap Tables:

  • Div-Based Structure: Uses nested <div> elements with specific classes and roles.
  • Role Attributes: Elements may have role="row" and role="cell" attributes to denote rows and cells.

Approach:

  1. Inspect the Table Structure: Understand how rows and cells are defined within the <div> hierarchy.
  2. Use Appropriate Locators: Utilize class names and role attributes to accurately locate rows and cells.
  3. Iterate and Extract Data: Similar to traditional tables but adjusted for the div-based structure.

Sample HTML Structure:

<div role="table" class="bootstrap-table">
    <div role="rowgroup">
        <div role="row">
            <div role="columnheader">ID</div>
            <div role="columnheader">Name</div>
            <div role="columnheader">Email</div>
            <div role="columnheader">Status</div>
        </div>
        <div role="row">
            <div role="cell">1</div>
            <div role="cell">John Doe</div>
            <div role="cell">john.doe@example.com</div>
            <div role="cell">Active</div>
        </div>
        <!-- More rows -->
    </div>
</div>

Implementation Steps:

1. Locate the Table Body:

WebElement tableBody = driver.findElement(By.xpath("//div[@role='table']/div[@role='rowgroup']"));

2. Find All Rows:

List allRows = tableBody.findElements(By.xpath(".//div[@role='row']"));
int numberOfRows = allRows.size();

3. Iterate Through Each Row and Extract Data:

for (int r = 2; r <= numberOfRows; r++) { // Starting from 2 to skip header
    String customerNameXPath = "//div[@role='table']/div[@role='rowgroup']/div[@role='row'][" + r + "]/div[@role='cell'][2]";
    String customerName = driver.findElement(By.xpath(customerNameXPath)).getText();
    
    String emailXPath = "//div[@role='table']/div[@role='rowgroup']/div[@role='row'][" + r + "]/div[@role='cell'][3]";
    String email = driver.findElement(By.xpath(emailXPath)).getText();
    
    String statusXPath = "//div[@role='table']/div[@role='rowgroup']/div[@role='row'][" + r + "]/div[@role='cell'][4]";
    String status = driver.findElement(By.xpath(statusXPath)).getText();
    
    System.out.println("Customer Name: " + customerName + "\tEmail: " + email + "\tStatus: " + status);
    
    // Optionally, interact with checkboxes if present
    // String checkboxXPath = "//div[@role='table']/div[@role='rowgroup']/div[@role='row'][" + r + "]/div[@role='cell'][1]/input[@type='checkbox']";
    // WebElement checkbox = driver.findElement(By.xpath(checkboxXPath));
    // checkbox.click();
}

Notes:

  • Role-Based Locators: Utilize @role='row' and @role='cell' to accurately target rows and cells within the div-based table.
  • Dynamic Indexing: Similar to traditional tables, dynamically construct XPath expressions using loop variables to navigate through rows and cells.

4. Sample Code Implementation

Below is a complete sample script that demonstrates handling a dynamic table with pagination. This script performs the following tasks:

  1. Logs into the application.
  2. Navigates to the Customers Sub-Menu.
  3. Determines the total number of pages dynamically.
  4. Iterates through each page, extracting Customer Name, Email Address, and Status.
  5. Handles potential pop-up windows.

Prerequisites:

  • Selenium WebDriver: Ensure that Selenium WebDriver is set up in your project.
  • ChromeDriver: Update the path to chromedriver.exe as per your system configuration.

Sample Code:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.NoSuchElementException;
import org.openqa.selenium.TimeoutException;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.openqa.selenium.support.ui.ExpectedConditions;
import java.util.List;
import java.util.concurrent.TimeUnit;

public class DynamicPaginationTableHandler {
    public static void main(String[] args) {
        // 1. Setup WebDriver
        System.setProperty("webdriver.chrome.driver", "path_to_chromedriver"); // Update path
        WebDriver driver = new ChromeDriver();

        // 2. Configure WebDriver
        driver.manage().window().maximize();
        driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS); // Implicit Wait

        try {
            // 3. Navigate to the Login Page
            driver.get("https://example-ecommerce.com/login"); // Replace with actual URL

            // 4. Perform Login
            WebElement usernameField = driver.findElement(By.xpath("//input[@name='username']"));
            WebElement passwordField = driver.findElement(By.xpath("//input[@name='password']"));
            WebElement loginButton = driver.findElement(By.xpath("//button[@type='submit']"));

            usernameField.clear();
            usernameField.sendKeys("demo"); // Replace with actual username
            passwordField.clear();
            passwordField.sendKeys("demo"); // Replace with actual password
            loginButton.click();

            // 5. Handle Potential Pop-Up Window
            try {
                WebDriverWait wait = new WebDriverWait(driver, 5);
                WebElement closeButton = wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//button[@class='close-popup']")));
                if (closeButton.isDisplayed()) {
                    closeButton.click();
                }
            } catch (TimeoutException e) {
                // Pop-up did not appear; proceed
            }

            // 6. Navigate to Customers Sub-Menu
            WebElement customersMenu = driver.findElement(By.xpath("//ul[@id='main-menu']/li/a[text()='Customers']"));
            customersMenu.click();

            WebElement customersSubMenu = driver.findElement(By.xpath("//ul[@id='main-menu']/li/a[text()='Customers']/following-sibling::ul/li/a[text()='Sub Menu']"));
            customersSubMenu.click();

            // 7. Determine Total Number of Pages
            WebElement paginationInfo = driver.findElement(By.xpath("//div[@class='pagination-info']"));
            String paginationText = paginationInfo.getText(); // e.g., "Showing 1 to 10 of 1909 pages"

            // Extract total pages
            int totalPages = 1; // Default to 1
            try {
                String[] parts = paginationText.split(" ");
                String totalPagesStr = parts[5]; // "1909"
                totalPages = Integer.parseInt(totalPagesStr);
            } catch (Exception e) {
                System.out.println("Failed to extract total pages. Defaulting to 1.");
            }
            System.out.println("Total number of pages: " + totalPages);

            // 8. Iterate Through Each Page
            for (int p = 1; p <= totalPages; p++) { if (p > 1) {
                    // Construct XPath for the page number
                    String pageXPath = "//ul[@class='pagination']/li/a[text()='" + p + "']";
                    try {
                        WebElement pageLink = driver.findElement(By.xpath(pageXPath));
                        pageLink.click();

                        // Optional: Wait for table to load
                        WebDriverWait wait = new WebDriverWait(driver, 10);
                        wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//table[@class='data-table']")));
                    } catch (NoSuchElementException e) {
                        System.out.println("Page number " + p + " not found. Continuing to next page.");
                        continue;
                    }
                }

                // 9. Extract Data from Current Page
                extractDataFromCurrentPage(driver);
            }

        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            // 10. Close the Browser
            driver.quit();
        }
    }

    public static void extractDataFromCurrentPage(WebDriver driver) {
        try {
            WebElement tableBody = driver.findElement(By.xpath("//table[@class='data-table']/tbody"));
            List allRows = tableBody.findElements(By.tagName("tr"));
            int numberOfRows = allRows.size();
            System.out.println("Number of rows on this page: " + numberOfRows);

            for (int r = 1; r <= numberOfRows; r++) {
                // Extract Customer Name (Assuming it's in the 2nd column)
                String customerNameXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[2]";
                String customerName = driver.findElement(By.xpath(customerNameXPath)).getText();

                // Extract Email Address (Assuming it's in the 3rd column)
                String emailXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[3]";
                String email = driver.findElement(By.xpath(emailXPath)).getText();

                // Extract Status (Assuming it's in the 5th column)
                String statusXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[5]";
                String status = driver.findElement(By.xpath(statusXPath)).getText();

                // Print the extracted data
                System.out.println("Customer Name: " + customerName + "\tEmail: " + email + "\tStatus: " + status);

                // Optionally, interact with checkboxes
                /*
                String checkboxXPath = "//table[@class='data-table']/tbody/tr[" + r + "]/td[1]/input[@type='checkbox']";
                WebElement checkbox = driver.findElement(By.xpath(checkboxXPath));
                if (!checkbox.isSelected()) {
                    checkbox.click();
                }
                */
            }
        } catch (NoSuchElementException e) {
            System.out.println("Failed to locate table or its elements on this page.");
        }
    }
}

Explanation:

  1. Setup and Configuration:
    • Initializes the Chrome WebDriver.
    • Maximizes the browser window and sets an implicit wait of 10 seconds for element loading.
  2. Login Process:
    • Navigates to the login page.
    • Locates and interacts with the username and password fields.
    • Clicks the login button.
    • Handles a potential pop-up window using an explicit wait. If the pop-up appears, it closes it; otherwise, it proceeds.
  3. Navigating to the Customers Sub-Menu:
    • Clicks on the main “Customers” menu.
    • Clicks on the “Sub Menu” within the Customers menu.
  4. Determining Total Number of Pages:
    • Locates the pagination information element.
    • Extracts and parses the total number of pages from the pagination text.
  5. Iterating Through Each Page:
    • Uses a for loop to navigate through each page based on the total page count.
    • Constructs a dynamic XPath for each page number and clicks the corresponding link.
    • Waits for the table to load after each navigation.
  6. Extracting Data from Each Page:
    • Defines the extractDataFromCurrentPage method to locate the table body and iterate through each row.
    • Extracts Customer Name, Email Address, and Status from each row.
    • Optionally interacts with checkboxes if required.
  7. Exception Handling:
    • Handles NoSuchElementException to manage scenarios where elements might not be found, ensuring the script doesn’t abruptly fail.
  8. Closing the Browser:
    • Ensures that the browser closes regardless of whether the script executes successfully or encounters exceptions.

Notes:

  • Dynamic XPath Construction: Embedding loop variables within XPath expressions allows the script to interact with varying elements based on the current iteration.
  • Synchronization: Using explicit waits (WebDriverWait) ensures that elements are loaded and interactable before actions are performed, enhancing script reliability.
  • Exception Handling: Incorporating try-catch blocks prevents the script from terminating unexpectedly due to unforeseen issues, such as missing elements.

5. Assignments

Assignment 1: Handling Dynamic Tables with Pagination

Objective: Automate the process of navigating through a dynamic table with pagination, extracting specific data from each page.

Scenario Overview:

  1. Navigate to the Application:
  2. Log In to the Application.
  3. Navigate to the Customers Sub-Menu:
    • Click on the “Customers” main menu.
    • Click on the “Sub Menu” within Customers.
  4. Determine the Total Number of Pages:
    • Locate the pagination information.
    • Extract and parse the total number of pages dynamically.
  5. Iterate Through Each Page:
    • Loop from page 1 to totalPages.
    • Click on each page number dynamically.
    • Extract and print Customer Name, Email Address, and Status from each row.
  6. Handle Potential Pop-Up Windows:
    • Detect and close pop-ups if they appear.

Submission: Provide the complete Selenium script with appropriate comments explaining each step.

Sample Skeleton: Refer to the Sample Code Implementation section above for a comprehensive example.

Assignment 2: Handling Bootstrap Tables

Objective: Automate interactions with a Bootstrap-based table that does not use traditional <table>, <tr>, and <td> tags.

Scenario Overview:

  1. Navigate to the Application:
  2. Log In to the Application.
  3. Navigate to the Admin Tab:
    • Click on the “Admin” tab to access the user management table.
  4. Extract Data from the Bootstrap Table:
    • Identify the structure of the Bootstrap table.
    • Extract Username and User Role from each row.
    • Handle scenarios where the number of rows varies.

Implementation Steps:

  1. Locate the Bootstrap Table: 
    WebElement tableBody = driver.findElement(By.xpath("//div[@role='table']/div[@role='rowgroup']"));
  2. Find All Rows: 
    List allRows = tableBody.findElements(By.xpath(".//div[@role='row']"));
int numberOfRows = allRows.size();
  3. Iterate Through Each Row and Extract Data: 
    for (int r = 2; r <= numberOfRows; r++) { // Starting from 2 to skip header
    String usernameXPath = "//div[@role='table']/div[@role='rowgroup']/div[@role='row'][" + r + "]/div[@role='cell'][2]";
    String username = driver.findElement(By.xpath(usernameXPath)).getText();
    
    String roleXPath = "//div[@role='table']/div[@role='rowgroup']/div[@role='row'][" + r + "]/div[@role='cell'][3]";
    String role = driver.findElement(By.xpath(roleXPath)).getText();
    
    System.out.println("Username: " + username + "\tRole: " + role);
}

Notes:

  • Role Attributes: Utilize @role='row' and @role='cell' to navigate through the div-based table structure.
  • Dynamic Row Handling: Adjust the loop to accommodate varying numbers of rows.

Submission: Provide the complete Selenium script with appropriate comments explaining each step.

6. Best Practices

  1. Use Explicit Waits Over Thread.sleep():
    • Why?: Enhances synchronization and reduces test execution time.
    • How?: 
      WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
wait.until(ExpectedConditions.elementToBeClickable(By.id("elementID")));
  2. Prefer Reliable Locators:
    • Why?: Ensures consistency and reduces maintenance.
    • How?:
      • Use By.id or By.name when available.
      • Use descriptive XPath or CSS selectors.
      • Avoid brittle locators that depend on dynamic attributes.
  3. Handle Exceptions Gracefully:
    • Why?: Prevents abrupt test failures and allows for better debugging.
    • How?: 
      try {
    // Interaction code
} catch (NoSuchElementException e) {
    System.out.println("Element not found: " + e.getMessage());
}
  4. Avoid Hardcoding Values:
    • Why?: Enhances flexibility and reusability.
    • How?:
      • Use variables or external data sources for input values.
      • Parameterize test scripts.
  5. Maintain Clean and Readable Code:
    • Why?: Facilitates easier maintenance and collaboration.
    • How?:
      • Use descriptive variable and method names.
      • Comment complex logic or XPath expressions.
      • Organize code into reusable methods or Page Object Model (POM) structures.
  6. Use Page Object Model (POM):
    • Why?: Promotes reusability and separation of concerns.
    • How?:
      • Create separate classes for each page.
      • Encapsulate element locators and interaction methods within page classes.
  7. Regularly Update WebDriver and Browsers:
    • Why?: Ensures compatibility and leverages the latest features and fixes.
  8. Securely Handle Credentials:
    • Why?: Prevents exposure of sensitive information.
    • How?: Use environment variables or encrypted storage solutions.
  9. Validate Actions:
    • Why?: Ensures that the intended actions (like selecting a checkbox) have been performed successfully.
    • How?: 
      assert checkbox.isSelected() : "Checkbox was not selected.";
  10. Avoid Overcomplicating Locators:
    • Why?: Simplifies maintenance and reduces errors.
    • How?: Use the simplest locator that uniquely identifies an element.

7. Conclusion

Handling dynamic web tables with pagination is a critical skill in Selenium WebDriver automation. By systematically determining the number of pages, navigating through each page, and extracting relevant data, you can ensure comprehensive coverage of dynamic data scenarios. Additionally, understanding and adapting to different table structures, such as those built with Bootstrap, further enhances the robustness of your automation scripts.

Key Takeaways:

  • Dynamic Page Counts: Always extract the total number of pages dynamically to accommodate data changes.
  • Robust Locators: Use flexible and reliable locators to interact with varying elements.
  • Synchronization: Implement explicit waits to handle dynamic content loading effectively.
  • Exception Handling: Incorporate try-catch blocks to manage unexpected scenarios gracefully.

Next Steps:

  • Practice Assignments: Implement the provided assignments to reinforce your understanding of dynamic table handling.
  • Explore Advanced Topics: Delve into handling more complex tables, integrating Selenium with testing frameworks like TestNG or JUnit, and leveraging the Page Object Model (POM) for better code organization.
  • Continuous Learning: Stay updated with Selenium WebDriver’s latest features and best practices to build efficient and robust automation scripts.

Happy Automating!


Last Post


Next Post