dlsite-rs-next

A high-performance Rust client library for the DLsite platform, providing access to product information, search, circle data, and more.

Stability: This library is in early development (0.x). The public API may change between releases. Features marked as "stub" or "requires feature flag" are not yet production-ready.

Installation

Add this to your Cargo.toml:

[dependencies]
dlsite-rs-next = "0.2"

Feature Flags

Feature	Description
(default)	unknown-field-log + reqwest-rustls-tls
search-html	HTML scraping-based search and circle functionality (adds scraper, rayon)
cookie-store	Session/auth support (adds reqwest/cookies)
reqwest-native-tls	Use native TLS instead of rustls

Important: The search-html feature flag controls API visibility:

• Without search-html: DlsiteClient::search() and DlsiteClient::circle() methods are not available
• Types like SearchProductQuery, SearchResult, CircleClient, etc. are not accessible
• This ensures the public API surface matches the actual feature requirements

To use search or circle functionality:

[dependencies]
dlsite-rs-next = { version = "0.2", features = ["search-html"] }

TLS Backend: By default, this crate uses rustls (pure Rust TLS). To use native TLS:

[dependencies]
dlsite-rs-next = { version = "0.2", default-features = false, features = ["reqwest-native-tls", "unknown-field-log"] }

Performance

The library includes several performance optimizations that provide significant speedups:

Optimization	Speedup	Use Case
Parallel Parsing	3-4x	Large result sets (50+ items)
Result Caching	10-100x	Repeated queries
Batch Queries	2-3x	Multi-page queries
Streaming API	-50% memory	Large result processing
Selector Caching	5-10%	All queries
Combined	10-100x	Typical usage

See docs/dlsite_endpoint_inventory.md for the full endpoint coverage matrix.

Implemented features

• Get product information by scraping html and using ajax api for web.
  • Basic information
  • Additional information
  • Multi-language support via Language enum (Jp, En, Ko, ZhCn, ZhTw) (HTML scraping still uses Japanese page; locale applies to review and API calls)
• Get product review (with locale support via get_review_with_locale)
• Get product information using api (with locale support via get_with_locale)
• Get product thumbnail (ProductApiClient::get_product_thumbnail)
• Get product screenshots (ProductApiClient::list_product_screenshots)
• Search product (all documented parameters now implemented)
• Site abstraction via Site enum (Maniax, Books, Soft, Pro, Appx, Comic, Home, Custom)
• Get circle info
  • Get circle product list
  • Get circle profile metadata (get_circle_profile)
  • Get circle games only (CircleClient::list_circle_games)
  • Resolve circle name to maker_id (CircleClient::resolve_circle_name)
  • Get circle sale list (needs network capture)
• Work type helpers (WorkType::is_game())
• Login and user related feature (stubs behind cookie-store feature)
• Get ranking (stub — endpoint URL needs network capture verification)

Features

Performance Optimizations

• Parallel Parsing: 3-4x faster search result parsing using rayon
• Result Caching: 10-100x faster repeated queries with LRU cache
• Batch Queries: 2-3x faster multi-page queries with concurrent requests
• Streaming API: 50% less memory usage for large result sets
• Selector Caching: 5-10% faster parsing with pre-compiled CSS selectors

Reliability Features

• Rate Limiting: Automatic 2 requests/second to prevent IP bans
• Retry Logic: Automatic retry with exponential backoff for transient failures
• Connection Pooling: Configurable connection pool for better resource usage

Maintainability

• Unified Selector Architecture: All CSS selectors are centralized in a single module with caching for consistent parsing behavior
• Comprehensive Test Coverage: Fixture-based parser tests and wiremock-based integration tests ensure reliability without network dependencies

Code Quality

• Centralized Selectors: All CSS selectors are defined in a single module with caching for consistency
• Comprehensive Testing: HTML fixture-based parser tests with snapshot testing for regression detection
• Mock Integration Tests: Network-free tests using wiremock for reliable CI/CD

Example

Basic Usage

• Get product by api

  use dlsite_rs_next::DlsiteClient;
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::default();
      let product = client.product_api().get("RJ01014447").await.unwrap();
      assert_eq!(product.creators.unwrap().voice_by.unwrap()[0].name, "佐倉綾音");
  }

• Get product thumbnail and screenshots

  use dlsite_rs_next::DlsiteClient;
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::default();
  
      // Get thumbnail URL
      let thumbnail = client.product_api().get_product_thumbnail("RJ01014447").await.unwrap();
      println!("Thumbnail: {}", thumbnail);
  
      // Get screenshot URLs
      let screenshots = client.product_api().list_product_screenshots("RJ01014447").await.unwrap();
      for url in screenshots {
          println!("Screenshot: {}", url);
      }
  }

• Search products (with automatic parallel parsing and caching) Note: Requires search-html feature flag

  use dlsite_rs_next::{DlsiteClient, client::search::SearchProductQuery, interface::query::SexCategory};
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::default();
      let query = SearchProductQuery {
          sex_category: Some(vec![SexCategory::Male]),
          keyword: Some("ASMR".to_string()),
          ..Default::default()
      };
      let results = client
          .search()
          .search_product(&query)
          .await
          .expect("Failed to search");
      println!("Found {} products", results.products.len());
  }

Advanced Usage

• Batch query multiple pages concurrently Note: Requires search-html feature flag

  use dlsite_rs_next::{DlsiteClient, client::search::SearchProductQuery, interface::query::SexCategory};
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::default();
      let queries = vec![
          SearchProductQuery {
              sex_category: Some(vec![SexCategory::Male]),
              page: Some(1),
              ..Default::default()
          },
          SearchProductQuery {
              sex_category: Some(vec![SexCategory::Male]),
              page: Some(2),
              ..Default::default()
          },
      ];
  
      let results = client
          .search()
          .search_products_batch(&queries)
          .await
          .expect("Failed to search");
  
      for (i, result) in results.iter().enumerate() {
          println!("Page {}: {} products", i + 1, result.products.len());
      }
  }

• Stream large result sets with callback Note: Requires search-html feature flag

  use dlsite_rs_next::{DlsiteClient, client::search::SearchProductQuery, interface::query::SexCategory};
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::default();
      let query = SearchProductQuery {
          sex_category: Some(vec![SexCategory::Male]),
          ..Default::default()
      };
  
      let total = client
          .search()
          .search_product_stream(&query, |item| {
              println!("Processing: {} ({})", item.title, item.id);
          })
          .await
          .expect("Failed to search");
  
      println!("Total items: {}", total);
  }

• Get circle games and resolve circle name Note: Requires search-html feature flag

  use dlsite_rs_next::DlsiteClient;
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::default();
  
      // List all games from a circle (filters out non-game works)
      let games = client.circle().list_circle_games("RG24350").await.unwrap();
      for game in games {
          println!("{}: {}", game.id, game.title);
      }
  
      // Resolve circle name to maker_id
      if let Some(maker_id) = client.circle().resolve_circle_name("Circle Name").await.unwrap() {
          println!("Found maker ID: {}", maker_id);
      }
  }

• Custom client configuration

  use dlsite_rs_next::{DlsiteClient, RetryConfig};
  use std::time::Duration;
  
  #[tokio::main]
  async fn main() {
      let client = DlsiteClient::builder("https://www.dlsite.com/maniax")
          .pool_max_idle_per_host(20)  // Increase connection pool
          .timeout(Duration::from_secs(60))  // Increase timeout
          .cache(200, Duration::from_secs(7200))  // Larger cache, 2 hour TTL
          .retry_config(RetryConfig::new(
              5,  // Max 5 retries
              Duration::from_millis(200),  // Initial delay 200ms
              Duration::from_secs(30),  // Max delay 30s
          ))
          .build()
          .expect("Failed to build client");
  
      // Use the custom client
      let product = client.product_api().get("RJ01014447").await.unwrap();
      println!("Product: {}", product.work_name);
  }

Development

Before submitting a PR or publishing, run the verification checks:

# Individual checks
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps
cargo publish --dry-run

# Or use the convenience script
./scripts/check.sh

License

Licensed under the MIT License. See [LICENSE](LICENSE) for details.