Skip to content

Challenges during dev

Jump to bottom
Al Amin edited this page Jul 25, 2025 · 1 revision

🧠 Project Challenges Faced (WPForms Advanced Entries Manager)

🔧 1. Handling Dynamic Form Structures

  • Challenge: Each WPForms form can have different fields. Storing them in a traditional column-based table (1 field = 1 column) would break scalability and flexibility.

  • Solution: Decided to serialize entry data in a single entry column while storing critical fields like name and email in separate dedicated columns for fast lookup and indexing.

🧵 2. Optimizing Storage for Millions of Entries

  • Challenge: Seeding and storing over 2 million+ entries raised concerns around database bloat and long-term performance.

  • Solution: Implemented batch insertions via a WP-CLI-based seeder and ensured entries are indexed efficiently (e.g., composite index (form_id, id)).

🔍 3. Query Performance Bottlenecks

  • Challenge: Querying entries with form_id and pagination (page, per_page) resulted in 5--6 seconds latency due to inefficient indexes.

  • Solution: Added composite indexes like (form_id, id) and ensured all frequent filters (status, email, name, created_at) had proper indexes.

  • Result: Reduced latency from ~6s to under 300ms, even with 800k+ rows.

🔄 4. Sync with Google Sheets

  • Challenge: Automatic sync during wpforms_process_complete could accidentally send test or seeded data to live Google Sheets.

  • Solution: Separated sync logic to rely on data from our custom DB table, giving us full control and the ability to prevent auto sync for seeded/tested data.

🔐 5. Safe and Efficient Seeder

  • Challenge: While inserting millions of rows, had to avoid memory leaks, SQL size limits, and accidental syncing.

  • Solution:

    • Used efficient batch insert logic (1000 rows per batch).

    • Avoided serializing sensitive fields like name and email.

    • Ran seeder via wp eval-file to keep it isolated.

🧪 6. Testing with Realistic Data

  • Challenge: Needed to simulate real-world usage (UK/US names, emails, comments).

  • Solution: Built a faker-style seeder with unique +suffix emails and professional names to simulate production data.

🧩 7. Flexible Search + Pagination

  • Challenge: Search needed to work by name, email, and even partial matches with pagination.

  • Solution:

    • Enhanced query to detect if search is email.

    • Used LIKE for name wildcard searches.

    • Indexed both columns for blazing-fast queries.

📉 8. Initial Query Slowness

  • Challenge: First page loads were still slow due to full table scans or missing covering indexes.

  • Solution: Used EXPLAIN, reviewed query plans, and restructured indexes accordingly to minimize scanned rows.

✅ Summary

This project required deep thought into database design, optimization, indexing, and plugin architecture to ensure a smooth UX for managing huge datasets inside WordPress. The biggest win was turning a 6s load time into sub-second performance with a clean and scalable backend.

Clone this wiki locally