Introduction
Note: entire development also things described herein have been taking place on YouTube LIVE ( YouTube now stores over 180TB of our live programming sessions since the early 2017).
Hello everyone!
We are excited to share some significant improvements we’ve made to our blockchain search functionality, particularly in how we handle pagination when searching for transactions, blocks, and domains. Our previous approach had some limitations that could impact performance and user experience, especially as our blockchain continues to grow. We’ve now implemented a new solution based on iterators, which brings several benefits, especially for our upcoming Blockchain Explorer UI dApp.
In this update, we’ll explain the differences between our prior approach and the current one, introduce the concept of iterators in a gentle way, and highlight the benefits these changes bring to our platform and community.
Why Pagination Matters
Before diving into the technical details, let’s briefly discuss why pagination is important.
Pagination is the process of dividing a large set of results into smaller, manageable chunks or “pages.” When you search for something on a platform with a vast amount of data—like our blockchain—you don’t want to load all the results at once. Instead, you retrieve a subset of results per page, which improves performance and user experience.
The Challenge with Our Previous Approach
In our earlier implementation, when a user performed a search across transactions, blocks, and domains, we faced a dilemma:
-
Independent Pagination: Each search category (transactions, blocks, domains) applied pagination separately. This meant that if you requested 10 results per page, you might get up to 10 transactions, 10 blocks, and 10 domains, resulting in a total of up to 30 results instead of the expected 10.
-
Inconsistent Total Counts: The total number of results was calculated separately for each category and then summed up. This made it difficult to provide accurate pagination across all categories combined.
-
Performance Issues: To combine and paginate results from all categories accurately, we would need to fetch all matching results from each category before applying pagination. This could be computationally intensive and inefficient, especially as our blockchain grows.
Introducing Iterators
To address these challenges, we’ve adopted an approach based on iterators.
What Are Iterators?
An iterator is a programming construct that allows you to traverse through a collection of items one at a time, without needing to load the entire collection into memory. Think of it like flipping through the pages of a book one by one, rather than trying to read the whole book at once.
In our context, iterators enable us to:
- Fetch Results Incrementally: Retrieve search results one at a time or in small batches, rather than all at once.
- Combine Results Efficiently: Merge results from different categories (transactions, blocks, domains) as we retrieve them.
- Maintain Performance: Avoid loading large datasets into memory, thus keeping our application responsive.
How the Iterator-Based Approach Works
1. Creating Iterators for Each Category
We have implemented separate iterators for transactions, blocks, and domains:
- TransactionSearchIterator
- BlockSearchIterator
- DomainSearchIterator
Each iterator knows how to fetch and return the next relevant result in its category, considering the search query and any filters applied.
2. Merging Results from Different Categories
We use a technique called incremental merging:
- Initialization: We initialize each iterator and fetch the first result from each.
- Priority Queue (Heap): We use a data structure called a priority queue (or heap) to keep track of the next result to return across all categories, based on a sorting criterion like timestamp or relevance.
- Fetching Results: As we return a result to the user, we fetch the next result from the corresponding iterator and insert it into the priority queue.
- Pagination: We continue this process until we’ve collected enough results to fill the requested page.
3. Applying Pagination
By incrementally merging results and only fetching what we need, we can:
- Provide Accurate Pagination: Ensure the user gets exactly the number of results they requested per page, across all categories combined.
- Handle Large Data Sets Efficiently: Avoid the performance issues associated with fetching all results upfront.
Benefits for the Blockchain Explorer UI dApp
Our upcoming Blockchain Explorer UI dApp will greatly benefit from this iterator-based approach:
Improved Performance
- Faster Load Times: By fetching only the necessary data, pages will load more quickly.
- Responsive Searches: Users can perform searches without experiencing delays, even as the blockchain grows.
Accurate and Consistent Results
- Unified Pagination: Users will see a consistent number of results per page, making navigation intuitive.
- Relevant Results: Results are sorted and merged based on relevance or recency, providing users with the most pertinent information first.
Scalability
- Handles Growth: As more transactions, blocks, and domains are added to the blockchain, the iterator approach ensures that our search functionality remains efficient.
- Resource Management: The application uses memory and computational resources more effectively, preventing potential slowdowns or crashes.
Enhanced User Experience
- Seamless Navigation: Users can easily navigate through pages of results without unexpected jumps or inconsistencies.
- Comprehensive Search: By combining results from all categories, users get a holistic view of the blockchain data related to their query.
Understanding the Technical Improvements
Prior Approach vs. Iterator Approach
Prior Approach
- Fetch All Data: Needed to retrieve all matching results from each category to provide accurate pagination.
- Performance Bottleneck: Fetching large amounts of data could slow down the application.
- Inaccurate Pagination: Combining results after independent pagination led to inconsistent page sizes and total counts.
Iterator Approach
- Fetch Incrementally: Retrieves results as needed, one at a time or in small batches.
- Efficient Merging: Combines results from different categories on the fly, using a priority queue.
- Accurate Pagination: Provides consistent and precise pagination across all categories.
Key Programming Concepts Introduced
- Iterators: Allow sequential access to elements in a collection without exposing the underlying representation.
- Priority Queue (Heap): A data structure that allows us to efficiently retrieve the next most relevant result based on a sorting criterion.
- Incremental Fetching: The process of retrieving data in small amounts as needed, rather than all at once.
Example Scenario
Let’s consider a user searching for “blockchain” with 5 results per page.
Using the Prior Approach:
- Fetches up to 5 transactions, 5 blocks, and 5 domains separately.
- Combines them into a single list, potentially giving up to 15 results on a single page.
- Pagination becomes inconsistent, and performance may suffer due to fetching unnecessary data.
Using the Iterator Approach:
- Initializes iterators for each category.
- Merges results from all categories into a single, sorted list.
- Returns exactly 5 results per page, maintaining the specified page size.
- Fetches only the data needed for the current page, improving performance.
Conclusion
By adopting the iterator-based approach for our blockchain search functionality, we’ve significantly improved performance, scalability, and user experience. Our Blockchain Explorer UI dApp will be able to provide fast, accurate, and consistent search results, making it easier for users to explore and interact with the blockchain.
We believe these changes will greatly benefit our community by providing a more robust and user-friendly platform. We’re excited for you to experience these improvements firsthand and welcome any feedback you may have.
Thank you for your continued support!