JOIL (JetOctopus Internal Linker) is an AI-powered internal linking tool that helps you pass link equity from your strong pages to pages that need more visibility.
It works by analyzing two types of pages:
- Donors: Powerful pages with traffic, authority, and crawl budget that give links
- Acceptors: Pages that need a boost and receive links
Using AI embeddings, JOIL understands the semantic meaning of your content and matches donors to acceptors based on topical relevance. This ensures every recommended link makes contextual sense and helps your target pages rank better.
JOIL integrates seamlessly with Screaming Frog, taking your CSV exports and generating a ready-to-implement list of link recommendations with anchor text.
- Analyzes your website pages to find linking opportunities
- Uses AI to match pages based on content similarity
- Generates anchor text recommendations
- Respects your linking rules and limits
- Outputs a CSV file ready for implementation
- Building internal link structure for new websites
- Improving internal linking on existing sites
- Scaling internal linking efforts across large websites
- Finding contextually relevant linking opportunities
- Automating link recommendations based on content similarity
JOIL works with two types of pages:
Donors are pages that give links. These are typically "powerful pages" with:
- Crawl budget
- Organic traffic
- Search impressions
- Existing authority
Common donor page types:
- Blog posts with traffic
- Popular resource pages
- High-ranking content pages
- Category pages with visibility
- Long-form guides with authority
Acceptors are pages that receive links. These are typically "weak pages" that:
- May lack crawl budget
- Have low or no organic traffic
- Need a boost in internal linking value
- You want to help rank better
Common acceptor page types:
- New product pages
- Underperforming service pages
- Deep-level category pages
- Pages you want to promote
- Content that needs more visibility
The opposite approach also works: Acceptors can be high-traffic pages while donors are a vast number of low-performance pages. This helps distribute link equity across many pages that need attention.
You can use the same CSV file for both acceptors and donors if you want any page to potentially link to any other page.
JOIL uses AI embeddings to understand the semantic meaning of your pages. Here's how it works:
AI embeddings are mathematical representations of your page content. Each page is converted into a list of numbers (typically 1,536 dimensions for OpenAI embeddings) that capture its meaning. Pages about similar topics have similar embeddings.
AI distance measures how similar two pages are:
- Lower distance = more similar content
- Higher distance = less related content
JOIL uses AI distance in two ways:
- Filtering: Only creates links between pages that are sufficiently similar (above the minimum AI distance threshold)
- Sorting: Ranks potential links by similarity, giving you the best matches first
The default minimum AI distance is 0.6, which creates links between moderately to highly related pages.
Matching fields are data columns that JOIL uses to determine which pages should link together. They act as filters before AI similarity is calculated.
Examples of matching fields:
- Category
- Topic
- Location (city, state, region)
- Product type
- Content type
How matching fields work:
If you set matching fields to "category,location":
- A donor about "Real Estate in Miami" can link to an acceptor about "Real Estate in Miami"
- But not to an acceptor about "Real Estate in Tampa" (location doesn't match)
- And not to an acceptor about "Restaurants in Miami" (category doesn't match)
Matching fields help ensure your links are contextually appropriate beyond just semantic similarity.
For JOIL to work effectively with Screaming Frog, your pages need to include JSON-LD breadcrumb schema markup. This structured data helps JOIL understand your site hierarchy and page relationships.
If your site doesn't have JSON-LD breadcrumbs, you'll need to set up custom extraction in Screaming Frog to capture the matching fields from pages, such as categories, brand names, types, and so on.
JOIL offers two algorithms for creating link recommendations:
Greedy algorithm (default):
- For each acceptor page, finds all matching donors
- Calculates AI distance to each potential donor
- Filters donors by minimum AI distance
- Sorts donors by AI similarity (best matches first)
- Creates links up to your acceptor limit (default: 5 links)
- Moves to the next acceptor
This algorithm prioritizes quality matches. Popular acceptors may receive their full allocation quickly, while less common topics might get fewer links.
Equal algorithm:
- Finds all potential matches for all acceptors
- Sorts matches by AI similarity
- Distributes links evenly across all acceptors in rounds
- Each round gives one link to each acceptor
- Continues until acceptors reach their limits or donors are exhausted
This algorithm ensures balanced distribution. All acceptors get similar numbers of links, preventing concentration on popular pages.
This is the recommended workflow for using JOIL with Screaming Frog.
Before starting, you need:
- Screaming Frog SEO Spider (desktop application)
- OpenAI API key
- Website to analyze
- JOIL binary on your computer
You need to configure Screaming Frog to collect the right data for JOIL.
- Open Screaming Frog
- Go to Configuration > API Access > OpenAI
- Enter your OpenAI API key
- Click OK to save
Next, configure the embeddings:
- Go to Configuration > Custom > Extraction
- Click Add to create a new extraction
- In the Prompt Configuration tab:
- Name: "Embeddings for Page Text"
- Select the embedding model (e.g., text-embedding-3-small)
- Click Test to verify it works
- Click OK to save
- Go to Configuration > Extraction > Structured Data > JSON-LD
- Check the checkbox to enable JSON-LD extraction
- Click OK
Extraction of the structured data:
- Go to Configuration > Custom > Extraction
- Click Add to create a new extraction
- Configure:
- Name: "JSON-LD"
- Extractor: "CSSPath"
- Expression:
script[type="application/ld+json"] - Extract: "Inner HTML"
- Click OK to save
- Enter your website URL in Screaming Frog
- Click Start to begin crawling
- Wait for the crawl to complete
- The embeddings will be generated automatically for each page
You need to export three CSV files from Screaming Frog:
- Acceptors URLs - Pages that will receive links
- Donors URLs - Pages that will give links
- Internal links - Existing links on your site (optional but recommended)
These exports follow the same process:
- In the main Screaming Frog window, go to the Internal tab
- In the dropdown menu, select HTML
- Use the filter to select the pages you want:
- For acceptors: Filter to important pages (products, services, categories)
- For donors: Filter to content pages (blog posts, guides, resources)
- For both: You can use the same export if all pages can be both acceptors and donors
- Click the Export button
- Select Export As > CSV
- Save with a descriptive filename:
acceptors.csvfor acceptor pagesdonors.csvfor donor pagesall_pages.csvif using the same file for both
Exporting existing links helps JOIL avoid recommending duplicate links:
- In the Screaming Frog menu, go to Bulk Export > Links > All Inlinks
- Save as CSV with filename like
existing_links.csv
This export is optional but recommended to prevent duplicate link suggestions.
Before running JOIL, verify your CSV files have the required columns:
Acceptors/donors CSV must include:
Addressorurl- The page URLExtract embeddings from page content- AI embeddings (if using OpenAI)- Any custom fields you want to use for matching (category, location, etc.)
Internal links CSV should include:
url- The source page URLtarget_url- The destination page URL
The simplest way to run JOIL with Screaming Frog exports:
./joil --acceptors-file acceptors.csv --donors-file donors.csvJOIL automatically detects the Screaming Frog format and creates an output file called interlinking.csv in the same directory as your acceptors file.
If you want any page to potentially link to any other page:
./joil --acceptors-file all_pages.csv --donors-file all_pages.csvJOIL automatically prevents pages from linking to themselves.
To save the results to a specific location:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--output-file /path/to/recommendations.csvTo use matching fields for more targeted linking:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--match-fields "category,location"This ensures links are only created between pages with matching category AND location values.
To control how many links each page gives or receives:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--max-acceptor-links 10 \
--max-donor-links 5--max-acceptor-links- Maximum links each acceptor receives (default: 5)--max-donor-links- Maximum links each donor gives (default: 5)
To make linking more or less strict:
# Stricter - only very similar pages link
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--min-ai-distance 0.4
# Looser - more loosely related pages can link
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--min-ai-distance 0.7Lower values = stricter matching, higher quality links Higher values = more permissive matching, more link opportunities
To use the equal distribution algorithm:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--algorithm equalUse greedy (default) for quality-focused linking or equal for balanced distribution.
A comprehensive command with multiple options:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--output-file link_recommendations.csv \
--match-fields "category,city" \
--max-acceptor-links 8 \
--max-donor-links 5 \
--min-ai-distance 0.6 \
--algorithm greedy \
--anchor-field "title"JOIL creates a CSV file with three columns:
donor_url,anchor_text,acceptor_urlExample output:
donor_url,anchor_text,acceptor_url
https://example.com/blog/miami-guide,Miami Real Estate,https://example.com/miami-homes
https://example.com/blog/tampa-tips,Tampa Properties,https://example.com/tampa-homes
https://example.com/resources/buying-guide,First Time Home Buyer,https://example.com/first-time-buyersdonor_url - Where to place the link
- This is the page that should contain the new link
- Typically a content page (blog post, article, guide)
anchor_text - The link text
- The clickable text for the link
- Generated from the acceptor page's title or custom anchors
- Should be descriptive and relevant
acceptor_url - Link destination
- This is where the link points to
- Typically an important page (product, service, category)
-
Review the output:
- Open the CSV in Excel or Google Sheets
- Check a sample of recommendations for quality
- Verify the anchor text makes sense
-
Filter if needed:
- Remove any links that don't fit editorially
- Adjust anchor text if necessary
- Sort by donor URL to see all links for each page
-
Import to your CMS:
- Most CMS platforms can import CSV files
- Map the columns to your internal link fields
- Process in batches if you have many recommendations
-
Manual implementation:
- Give the CSV to your content team
- Add links to the specified donor pages
- Use the exact anchor text provided
-
Automated implementation:
- Use the JOIL API if available
- Build custom scripts to process the CSV
- Integrate with your deployment pipeline
You can provide your own anchor text instead of using page titles.
Create a CSV with these columns:
url,query,importance
https://example.com/product-page,buy product name,10000
https://example.com/product-page,product name reviews,5000
https://example.com/product-page,best product name,3000Columns explained:
url- The acceptor page URLquery- The anchor text to useimportance- Weight for this anchor (higher = used more often)
If a page receives 10 links and has these anchors:
- "buy product" with importance 10,000 (62.5%)
- "product reviews" with importance 5,000 (31.25%)
- "best product" with importance 1,000 (6.25%)
JOIL will distribute:
- ~6 links with "buy product"
- ~3 links with "product reviews"
- ~1 link with "best product"
This helps create natural anchor text variation.
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--custom-anchors-file custom_anchors.csvTo use both custom anchors and page titles:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--custom-anchors-file custom_anchors.csv \
--standard-anchor-percent 0.3This creates:
- 70% of links using custom anchors (weighted by importance)
- 30% of links using standard anchors (from page title field)
Remove unwanted text from anchor text using regular expressions:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--anchor-clean-regexp '\[.*?\]' \
--anchor-clean-regexp '®|™'This removes:
- Text in square brackets:
[Updated 2024] - Trademark symbols:
®,™
You can add multiple --anchor-clean-regexp flags to apply several cleaning rules.
To prevent JOIL from recommending links that already exist:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--existing-links-file existing_links.csvThe existing links CSV should have url and target_url columns (exported from Screaming Frog).
Control how many matching fields are required:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--match-fields "category,city,state" \
--min-fields-required 2This allows links when at least 2 out of 3 fields match, creating more linking opportunities while maintaining relevance.
To skip field matching and rely entirely on AI semantic similarity:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--match-fields "" \
--enable-ai-sortingThis is useful when:
- You don't have good matching fields
- Your content is highly varied
- You want maximum flexibility in link recommendations
If you want to use only field matching without AI sorting:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--match-fields "category" \
--enable-ai-sorting=falseLinks will be created based on field matches only, without considering semantic similarity.
To see detailed information about what JOIL is doing:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--debugThis shows:
- Which pages are being processed
- Matching results
- AI distance calculations
- Link creation decisions
| Flag | Description | Example |
|---|---|---|
--acceptors-file |
CSV file with pages that receive links | --acceptors-file acceptors.csv |
--donors-file |
CSV file with pages that give links | --donors-file donors.csv |
| Flag | Description | Default | Example |
|---|---|---|---|
--output-file |
Where to save recommendations | interlinking.csv in acceptors directory |
--output-file results.csv |
| Flag | Description | Default | Example |
|---|---|---|---|
--match-fields |
Comma-separated field names for matching | None | --match-fields "category,city" |
--min-fields-required |
Minimum matching fields needed | All fields | --min-fields-required 2 |
--fields-separator |
Character separating field values | , |
--fields-separator ";" |
| Flag | Description | Default | Example |
|---|---|---|---|
--max-acceptor-links |
Maximum links each acceptor receives | 5 |
--max-acceptor-links 10 |
--max-donor-links |
Maximum links each donor gives | 5 |
--max-donor-links 3 |
| Flag | Description | Default | Example |
|---|---|---|---|
--algorithm |
Linking algorithm (greedy or equal) |
greedy |
--algorithm equal |
| Flag | Description | Default | Example |
|---|---|---|---|
--enable-ai-sorting |
Use AI to sort and filter links | true |
--enable-ai-sorting=false |
--min-ai-distance |
Minimum AI similarity (0.0-1.0) | 0.6 |
--min-ai-distance 0.4 |
--ai-field-name |
Column name with AI embeddings | Extract embeddings from page content |
--ai-field-name "embeddings" |
| Flag | Description | Default | Example |
|---|---|---|---|
--anchor-field |
Field to use for anchor text | title |
--anchor-field "h1" |
--attributes |
Comma-separated fallback fields | None | --attributes "title,h1" |
--anchor-clean-regexp |
Regex to remove from anchors | None | --anchor-clean-regexp '\[.*?\]' |
--custom-anchors-file |
CSV with custom anchor text | None | --custom-anchors-file anchors.csv |
--standard-anchor-percent |
Percent using standard anchors (0.0-1.0) | 0 |
--standard-anchor-percent 0.3 |
| Flag | Description | Default | Example |
|---|---|---|---|
--csv-delimiter |
CSV field delimiter | , |
--csv-delimiter ";" |
--url-field-name |
Column name with URLs | url or Address |
--url-field-name "page_url" |
| Flag | Description | Example |
|---|---|---|
--help |
Show help message | --help |
--version |
Show JOIL version | --version |
--debug |
Enable debug logging | --debug |
Problem: JOIL runs but doesn't create an output file.
Solutions:
- Check that your CSV files have the required columns
- Verify the URL column is named
url,URL, orAddress - Make sure you have write permissions in the output directory
- Check for error messages in the console
- Try running with
--debugflag to see detailed logs
Problem: Output file is empty or has very few links.
Possible causes and solutions:
AI distance too strict:
- Try decreasing
--min-ai-distanceto 0.5 - Check if your pages have AI embeddings in the CSV
Matching fields too restrictive:
- Reduce the number of matching fields
- Lower
--min-fields-required - Try
--match-fields ""to disable field matching
Link limits too low:
- Increase
--max-acceptor-linksand--max-donor-links
Pages already linked:
- Check if you provided an existing links file
- Remove the
--existing-links-fileflag to see all recommendations
Problem: Error about missing AI embeddings.
Solutions:
- Verify you configured OpenAI API in Screaming Frog
- Check the embeddings extraction ran successfully during crawl
- Look for the column
Extract embeddings from page contentin your CSV - If using a different column name, specify it with
--ai-field-name - Disable AI sorting with
--enable-ai-sorting=falseif you don't have embeddings
Problem: JOIL doesn't recognize your CSV format.
Solutions:
- Ensure CSV is UTF-8 encoded
- Check that the first row contains column headers
- Verify URL column is named
url,URL, orAddress - If using a custom delimiter, specify with
--csv-delimiter - Remove any BOM (Byte Order Mark) from the file
Problem: Generated anchor text is not what you expected.
Solutions:
- Check which field is being used with
--anchor-field - Try a different field:
--anchor-field "h1"or--anchor-field "meta_description" - Use custom anchors file for complete control
- Clean unwanted text with
--anchor-clean-regexp - Verify the acceptor CSV has the field you're referencing
Problem: All recommended links use the same anchor text.
Solutions:
- Use custom anchors with importance weighting for variation
- Mix custom and standard anchors with
--standard-anchor-percent - Check if your acceptor pages have diverse titles/headings
- Consider manually editing the output CSV for more variety
Problem: JOIL is running slowly on large datasets.
Tips:
- JOIL is fast, but very large datasets (100,000+ pages) take time
- AI embedding calculations are the slowest part
- Consider splitting into smaller batches
- Disable AI sorting if speed is critical:
--enable-ai-sorting=false - Use matching fields to reduce the number of comparisons
Problem: Can't read input files or write output file.
Solutions:
- Check file permissions:
ls -la *.csv - Ensure you have read access to input files
- Ensure you have write access to output directory
- Try saving output to a different location with
--output-file - On Mac/Linux, you may need to make JOIL executable:
chmod +x joil
Yes. JOIL accepts any CSV file with the required columns:
url- Page URLExtract embeddings from page content- AI embeddings (if using AI sorting)- Any custom fields you want to match on
You can generate these CSVs from:
- Custom scripts
- Database exports
- Other SEO tools
- Manual creation
No. You can disable AI sorting with --enable-ai-sorting=false and use only field matching. However, AI sorting provides better link quality by understanding semantic similarity.
Greedy (default):
- Prioritizes best matches first
- Some acceptors may get all their links quickly
- Others might get fewer links
- Best for quality-focused linking
Equal:
- Distributes links evenly across all acceptors
- All acceptors get similar number of links
- Best for balanced distribution
- Good for ensuring all pages receive attention
Yes. Use the same CSV file for both:
./joil --acceptors-file all_pages.csv --donors-file all_pages.csvJOIL automatically prevents pages from linking to themselves.
Recommended values:
0.3-0.4- Very strict, only highly similar pages (product variants, closely related topics)0.5-0.6- Moderate, reasonably related pages (default: 0.6)0.7-0.8- Loose, broadly related pages (more link opportunities)0.9+- Very loose, minimal relevance requirement
Start with the default (0.6) and adjust based on results. Review a sample of recommendations to determine if they're too strict or too loose.
You have two options:
-
Use AI-only matching:
--match-fields "" --enable-ai-sorting -
Add matching fields to your CSV:
- Export additional data from your CMS
- Add category/topic columns manually
- Use URL patterns as matching fields
General recommendations:
- Acceptor links: 3-10 per page (default: 5)
- Donor links: 3-8 per page (default: 5)
Consider:
- Content length (longer pages can give more links)
- Page importance (important acceptors can receive more)
- User experience (too many links can be overwhelming)
- Editorial style (some sites prefer fewer, more strategic links)
Yes. You can run JOIL multiple times with different settings:
-
Test different parameters:
- Try different AI distance thresholds
- Compare greedy vs equal algorithms
- Test various matching field combinations
-
Generate multiple batches:
- Create separate recommendations for different page types
- Use different donor sets for different acceptors
- Implement gradually over time
-
Avoid duplicates:
- Use
--existing-links-filewith previous recommendations - Combine previous output with existing links export
- Use
AI embeddings from OpenAI are highly accurate at understanding semantic similarity. However:
- Not perfect: Review a sample before implementing at scale
- Context matters: Some nuances may be missed
- Combine approaches: Use both AI and field matching for best results
- Test and iterate: Adjust parameters based on your specific content
Yes. OpenAI embeddings support multiple languages. JOIL works with any language that OpenAI's embedding models support, which includes most major languages.
JOIL streams CSV files, so it can handle very large datasets:
- Tested: Up to 100,000+ pages
- Practical limit: Depends on your computer's memory
- Performance: Larger datasets take longer to process
If you have performance issues, consider splitting into smaller batches.
For acceptors (pages receiving links):
- Use custom anchors with higher importance values
- Run JOIL multiple times focusing on priority pages
- Manually adjust the output CSV to prioritize certain pages
For donors (pages giving links):
- Increase
--max-donor-linksfor important donor pages - Create separate donor CSVs for different priority levels
- Use matching fields to focus on specific donor types
Yes:
- Filter your CSVs: Remove pages before running JOIL
- Use matching fields: Create a field that excludes unwanted pages
- Post-process output: Remove unwanted recommendations from the output CSV
- Use existing links: Add unwanted page pairs to the existing links file
This is fine and common. A page can both give and receive links. JOIL will:
- Include it in matching for both roles
- Prevent it from linking to itself
- Apply both donor and acceptor link limits
When to regenerate:
- After publishing new content (weekly or monthly)
- When site structure changes significantly
- After updating page content substantially
- When adding new product/service pages
- As part of regular SEO maintenance
Best practice: Set a regular schedule (monthly or quarterly) to update internal link recommendations.
Export all HTML pages from Screaming Frog and run JOIL:
./joil --acceptors-file all_pages.csv --donors-file all_pages.csvResult: interlinking.csv with balanced link recommendations
Link category pages to product pages with strict matching:
./joil --acceptors-file products.csv \
--donors-file categories.csv \
--match-fields "category,brand" \
--min-ai-distance 0.5 \
--max-acceptor-links 10Result: Product pages receive up to 10 links from category pages with matching category and brand
Link blog posts to service pages using AI similarity only:
./joil --acceptors-file services.csv \
--donors-file blog_posts.csv \
--match-fields "" \
--enable-ai-sorting \
--min-ai-distance 0.6 \
--max-donor-links 3Result: Each blog post gives up to 3 links to semantically related service pages
Ensure all pages get equal link attention on a new website:
./joil --acceptors-file pages.csv \
--donors-file pages.csv \
--algorithm equal \
--max-acceptor-links 5 \
--max-donor-links 5Result: Balanced link distribution across all pages
Use custom anchors with natural variation:
./joil --acceptors-file products.csv \
--donors-file blog.csv \
--custom-anchors-file product_anchors.csv \
--standard-anchor-percent 0.2 \
--anchor-clean-regexp '®|™'Result: 80% custom anchors, 20% standard anchors, trademark symbols removed
Link local content using city and state matching:
./joil --acceptors-file locations.csv \
--donors-file local_guides.csv \
--match-fields "city,state" \
--min-fields-required 1 \
--anchor-field "h1" \
--max-acceptor-links 8Result: Location pages receive up to 8 links from guides in matching cities or states
Generate new recommendations while respecting existing links:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--existing-links-file current_links.csv \
--output-file new_recommendations.csvResult: Only new link recommendations that don't already exist
Aggressive linking strategy for content-rich sites:
./joil --acceptors-file all_pages.csv \
--donors-file all_pages.csv \
--match-fields "topic,category" \
--min-fields-required 1 \
--min-ai-distance 0.7 \
--max-acceptor-links 15 \
--max-donor-links 10 \
--algorithm greedyResult: High-volume linking with quality controls via field matching and AI
View all available options:
./joil --helpVerify your JOIL version:
./joil --versionRun with detailed logging to troubleshoot issues:
./joil --acceptors-file acceptors.csv \
--donors-file donors.csv \
--debugFor additional help:
- Review this manual thoroughly
- Check the examples section for similar use cases
- Verify your CSV files have the correct format
- Test with a small dataset first
- Use debug mode to understand what JOIL is doing