Auto-extract extension taxonomy facts from XBRL linkbases

[dej-h]

Feature Request: Auto-extract extension taxonomy facts from XBRL linkbases

Feature Category

• New API functionality
• Performance improvement
• Developer experience improvement
• Documentation enhancement
• Tool/utility addition

---

Problem Statement

Is your feature request related to a problem? Please describe.

When fetching financials for companies that define custom XBRL extension concepts, Tesla (tsla_AutomotiveRevenue), Berkshire (brka_*), banks, utilities, and many others, all extension facts are silently dropped from the normalized output. These concepts live in the company's own XBRL namespace and are never matched against the standard us-gaap tag map. The result is that the most analytically interesting(from a fundamental perspective), company-specific data (segment revenue breakdowns, custom balance sheet line items, non-standard cash flow adjustments) is invisible.

Today, company_mappings/ ships hardcoded JSON for exactly 3 companies (TSLA, MSFT, BRKA). This cannot scale: there are thousands of SEC filers with extension namespaces, and the mappings go stale as companies rename concepts across filings.

The data to solve this is already in every filing. The XBRL submission package that edgartools already fetches contains three linkbase files .xsd, _lab.xml, _cal.xml that together provide the human-readable label and the structural GAAP parent relationship for every extension concept, with no hardcoding required.

Who would benefit from this feature?

• Beginner Python users working with SEC filings
• ✅ Financial analysts and researchers
• ✅ Advanced developers building financial applications
• ✅ Data scientists working with financial datasets

---

Proposed Solution

Describe the solution you'd like

When edgartools parses an XBRL filing package, add a linkbase extraction pass that reads the three files already present in the submission:

1. .xsd - identifies which extension elements are non-abstract (actual reportable values) vs. structural (axes, domains, members, table/line-item wrappers). The abstract="true" attribute is the authoritative signal.

2. _lab.xml (label linkbase) - resolves the arc chain (loc, labelArc, label) to get the company-authored human-readable label for each extension concept. This is exactly the label that would appear in the printed 10-K/10-Q.

3. _cal.xml (calculation linkbase) - for each extension concept that appears as a child of a us-gaap_ parent in a calculation arc, records the parent concept and weight (+1.0 additive, -1.0 subtractive). This is the company's own declaration of where the concept sits within the financial statement hierarchy.

The output would be a collection of ExtensionFact objects (or similar) exposed alongside the standard normalized facts, annotated with label and parent relationship so downstream consumers can place them correctly without guessing.

Describe alternatives you've considered

• Expanding company_mappings/ manually, not scalable, goes stale, requires maintenance per company per filing year.
• Using the companyfacts API endpoint, this endpoint pre-aggregates facts but strips the linkbase metadata. The label and parent relationship information is only available in the per-filing XML package, which edgartools already accesses for its XBRL parsing.
• Ignoring extension facts entirely, the status quo. For many companies this is a significant gap in coverage, particularly for segment-level analysis.

---

Use Case Example

How would you use this feature?

from edgar import Company

company = Company("TSLA")
financials = company.get_financials()

# Standard facts work as today
revenue = financials.income_statement["Revenue"]

# Extension facts — new
extension_facts = financials.extension_facts  # or similar

for fact in extension_facts:
    print(fact.concept)        # "tsla_RestructuringAndOtherExpenses"
    print(fact.label)          # "Restructuring And Other Expenses"
    print(fact.parent_concept) # "us-gaap_OperatingExpenses"
    print(fact.weight)         # 1.0  (additive component of parent)
    print(fact.value)          # 1_730_000_000
    print(fact.period)         # "Q3 2025"

For developers building applications that need segment-level data:

# Get all revenue sub-components Tesla reports that aren't standard GAAP
revenue_segments = [
    f for f in financials.extension_facts
    if f.parent_concept == "us-gaap_Revenues"
]
# → tsla_AutomotiveRevenues, tsla_EnergyGenerationAndStorageRevenues, tsla_ServicesAndOtherRevenues

---

Implementation Considerations

Proof of concept

Verified against Tesla's Q3 2025 10-Q using only stdlib + requests 17 non-abstract extension concepts with full parent relationships extracted from a single filing with zero hardcoding. Happy to share the extraction script if useful.

The GAAP parent relationship is precisely what's needed to slot each extension fact into the statement hierarchy and it's declared by the company itself in the filing.

Complexity Level:

• Simple (minor API addition)
• ✅ Moderate (new functionality with existing patterns)
• Complex (significant architectural changes)

The linkbase parsing code already exists in edgartools' xbrl/ module for other purposes. The main work is wiring the extraction into the normalization pipeline and defining the output type for extension facts.

Backwards Compatibility:

• ✅ This feature maintains backwards compatibility

Extension facts would be surfaced as an additive property or collection existing .income_statement, .balance_sheet, .cash_flow_statement access patterns are unchanged. The company_mappings/ JSON files could remain supported as an override layer for cases where manual curation is preferred.

---

Additional Context

Why the companyfacts API can't solve this

The SEC's data.sec.gov/api/xbrl/companyfacts/CIK.json endpoint, the most convenient EDGAR data source, pre-aggregates facts but strips all linkbase metadata. The label and parent relationship for extension concepts is only available in the per-filing XML submission package. Edgartools already accesses these packages for its XBRL instance parsing; this feature would read the linkbases that are fetched alongside them.

Filtering structural elements

Not all extension namespace elements are facts. XBRL filings use extension namespaces for structural scaffolding: *Axis, *Domain, *Member, *Abstract, *Table, *LineItems. These should be excluded. The reliable filter is abstract="true" in the .xsd, structural elements declare themselves abstract; reportable value elements do not. In the Tesla example above, 37 of 83 extension concepts are abstract and filtered out, leaving 46 actual reportable elements.

Statement role filtering

Some extension concepts appear in calculation links only under disclosure schedule roles (note tables, supplemental detail schedules) rather than primary financial statement roles (ConsolidatedBalanceSheets, ConsolidatedStatementsofOperations, etc.). These could optionally be flagged with lower prominence, since primary statement facts are what most consumers want by default.

Related Issues/Features:

• The existing company_mappings/ system in edgar/xbrl/standardization/ addresses the same problem via hardcoding this feature would make that approach unnecessary for the common case.
• The unmapped_logger.py in the same directory already tracks unrecognised concepts, suggesting this gap is known.

---

Feature requests are evaluated based on EdgarTools' core principles: Simple yet powerful, accurate financials, beginner-friendly, and joyful UX.

[dgunning]

This is an excellent, well-researched proposal. You're right on every count: extension facts are a real coverage gap, the company_mappings/ approach doesn't scale, and the linkbase data to solve it is already present in every filing package.

Classification: P2 feature, tracked as edgartools-ixk9.

A few thoughts on the design:

What we already have: The linkbase parsing infrastructure exists — we parse presentation and definition linkbases for statement rendering and dimension validation. The calculation linkbase (_cal.xml) is read but underutilized. Label linkbase (_lab.xml) parsing is in place. So the foundation is there.

Where the complexity lives: The main question isn't extraction (you've proven that works) — it's how extension facts integrate into the existing output. Today statement_of_equity(), income_statement(), etc. return Statement objects with line items driven by the presentation tree. Extension concepts that appear in the presentation tree already show up (with their company-authored labels). The gap is concepts that appear in calculation links but aren't in the presentation tree for a given role, plus the standard_concept mapping.

Your proof-of-concept: Would love to see the extraction script you mentioned. It would help us validate the approach against a broader set of filers beyond Tesla — particularly banks and utilities which tend to have the heaviest extension usage.

Timeline: This is post-v5.29.0 work. It fits naturally alongside the standardization mapping overhaul we're actively working on, which is expanding from 96 to 253 standard concepts and adding industry-aware disambiguation.

Thank you for the thorough write-up — this is exactly the kind of proposal that moves the library forward.

[amcamc92]

Great issue — this addresses a real gap that affects many downstream applications.

I want to share a specific use case and two concrete feature requests that would maximize the value of this feature beyond extension-only facts.

My use case

I run a stock screener pipeline across ~5,000 SEC filers. For each ticker, I build a rawFacts.csv from two sources merged: company.get_facts().query().to_dataframe() (EntityFacts API, primary — full history) and filing.xbrl().facts.to_dataframe() per 10-K/10-Q (per-filing XBRL, structural metadata). I then run a 9-stage processing pipeline that filters rawFacts to the XBRL concepts configured per fact type (Revenues, EPS, Net Income, etc.).

The problem I run into — in two layers

Layer 1 — extension concepts silently dropped: Extension namespace concepts (e.g. tsla:AutomotiveRevenue, brka:*, ares:*) already arrive in my rawFacts via the EntityFacts API with their taxonomy prefix intact. But my concept filter discards them because there's no way to know whether they're relevant — I'd need to know their parent in the GAAP hierarchy. Today I hardcode per-company exceptions manually. This does not scale.

Layer 2 — standard us-gaap concept prioritization by entity type: Banks don't tag a single us-gaap:Revenues — they use InterestAndDividendIncomeOperating + NoninterestIncome. BDCs use NetInvestmentIncome. REITs use rental income concepts. Insurance uses PremiumsEarnedNet. I currently maintain a hand-curated registry of per-category concept priorities. That registry was built through manual research across 130+ fixture tickers — work that the _cal.xml would have generated automatically, since it already declares which concepts sum to which parent line item for each specific filer.

Both problems have the same root cause: I don't have access to the calculation linkbase hierarchy at processing time.

Why the feature as scoped (extension-only) has limited impact on my pipeline

The EntityFacts API (company.get_facts()) is my primary source because it covers full filing history in a single call. But as your issue correctly notes, that endpoint strips all linkbase metadata. My per-filing XBRL source does have access to _cal.xml, but after the two sources are merged and deduplicated on (concept, period_start, period_end, filing_date), the API row can win — losing any metadata added to the XBRL row. Adding parent_concept only to extension facts in filing.xbrl().facts.to_dataframe() doesn't solve this dedup problem.

What would actually maximize the value

Request 1: Expose filing.xbrl().calculation_linkbase as a standalone property returning a DataFrame with columns:
concept | taxonomy | parent_concept | parent_taxonomy | weight | statement_role | is_abstract
for all concepts in _cal.xml (not just extension namespace), filtered to is_abstract=False by default. The statement_role column distinguishes primary financial statements (ConsolidatedStatementsOfIncome, etc.) from disclosure schedules and note tables.

This would let me build a lightweight conceptHierarchy.csv per ticker at download time — completely separate from rawFacts, immune to the dedup problem — and use it in my concept filter stage to dynamically include any concept (us-gaap or extension) whose parent_concept maps to a target line item.

Request 2: Add parent_concept, calculation_weight, and statement_role as optional columns to filing.xbrl().facts.to_dataframe(), populated from _cal.xml when available, None otherwise.

Why the calculation linkbase for standard us-gaap facts matters

Once the _cal.xml parser is built for extension facts (as this issue proposes), exposing the full tree costs little additional work. But the value is disproportionately larger: for my Layer 2 problem, knowing that BAC's _cal.xml declares InterestAndDividendIncomeOperating as a child of the revenues line — for that specific filer — would auto-generate the entity-type concept maps I currently maintain by hand. Extension-only exposes perhaps 10-20% of the analytical value of the linkbase; the full tree exposes the rest.

Happy to share more detail on the pipeline architecture or test against specific tickers if useful.

[dej-h]

This is an excellent, well-researched proposal. You're right on every count: extension facts are a real coverage gap, the company_mappings/ approach doesn't scale, and the linkbase data to solve it is already present in every filing package.

Classification: P2 feature, tracked as edgartools-ixk9.

A few thoughts on the design:

What we already have: The linkbase parsing infrastructure exists — we parse presentation and definition linkbases for statement rendering and dimension validation. The calculation linkbase (_cal.xml) is read but underutilized. Label linkbase (_lab.xml) parsing is in place. So the foundation is there.

Where the complexity lives: The main question isn't extraction (you've proven that works) — it's how extension facts integrate into the existing output. Today statement_of_equity(), income_statement(), etc. return Statement objects with line items driven by the presentation tree. Extension concepts that appear in the presentation tree already show up (with their company-authored labels). The gap is concepts that appear in calculation links but aren't in the presentation tree for a given role, plus the standard_concept mapping.

Your proof-of-concept: Would love to see the extraction script you mentioned. It would help us validate the approach against a broader set of filers beyond Tesla — particularly banks and utilities which tend to have the heaviest extension usage.

Timeline: This is post-v5.29.0 work. It fits naturally alongside the standardization mapping overhaul we're actively working on, which is expanding from 96 to 253 standard concepts and adding industry-aware disambiguation.

Thank you for the thorough write-up — this is exactly the kind of proposal that moves the library forward.

Thanks for the context on what's already parsed, good to know _cal.xml is read but underutilised, that's exactly where this hooks in.

Extraction script, tested against Tesla Q3 2025 (tsla-20250930):

import requests
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from typing import Optional

LINK  = "http://www.xbrl.org/2003/linkbase"
XLINK = "http://www.w3.org/1999/xlink"
XS    = "http://www.w3.org/2001/XMLSchema"

@dataclass
class ExtensionConcept:
    concept: str
    label: str
    parent_concept: Optional[str]
    weight: Optional[float]
    statement_role: Optional[str]
    is_abstract: bool

def extract_extension_concepts(filing_base_url: str, ticker: str, headers: dict) -> list[ExtensionConcept]:
    # XSD — abstract vs. reportable
    xsd = ET.fromstring(requests.get(f"{filing_base_url}.xsd", headers=headers, timeout=15).content)
    extension_elements = {
        f"{ticker}_{el.get('name')}": el.get("abstract", "false") == "true"
        for el in xsd.findall(f"{{{XS}}}element")
        if el.get("name")
    }

    # _lab.xml — concept → label
    lab_root = ET.fromstring(requests.get(f"{filing_base_url}_lab.xml", headers=headers, timeout=15).content)
    label_link = lab_root.find(f"{{{LINK}}}labelLink")
    locs   = {l.get(f"{{{XLINK}}}label"): l.get(f"{{{XLINK}}}href", "").split("#")[-1]
               for l in label_link.findall(f"{{{LINK}}}loc")}
    labels = {l.get(f"{{{XLINK}}}label"): l.text or ""
               for l in label_link.findall(f"{{{LINK}}}label")
               if l.get(f"{{{XLINK}}}role", "").endswith("/label")}
    arcs   = {a.get(f"{{{XLINK}}}from"): a.get(f"{{{XLINK}}}to")
               for a in label_link.findall(f"{{{LINK}}}labelArc")}
    concept_label = {
        concept: labels.get(arcs.get(loc_lbl, ""), "")
        for loc_lbl, concept in locs.items()
        if concept.startswith(f"{ticker}_")
    }

    # _cal.xml — extension concept → (gaap parent, weight, statement role)
    cal_root = ET.fromstring(requests.get(f"{filing_base_url}_cal.xml", headers=headers, timeout=15).content)
    parents: dict[str, tuple[str, float, str]] = {}
    for calc_link in cal_root.findall(f"{{{LINK}}}calculationLink"):
        role  = calc_link.get(f"{{{XLINK}}}role", "").split("/")[-1]
        clocs = {l.get(f"{{{XLINK}}}label"): l.get(f"{{{XLINK}}}href", "").split("#")[-1]
                  for l in calc_link.findall(f"{{{LINK}}}loc")}
        for arc in calc_link.findall(f"{{{LINK}}}calculationArc"):
            parent = clocs.get(arc.get(f"{{{XLINK}}}from", ""), "")
            child  = clocs.get(arc.get(f"{{{XLINK}}}to", ""), "")
            if child.startswith(f"{ticker}_") and "us-gaap_" in parent:
                parents[child] = (parent, float(arc.get("weight", 1)), role)

    return [
        ExtensionConcept(
            concept=concept,
            label=concept_label.get(concept, ""),
            parent_concept=parents[concept][0] if concept in parents else None,
            weight=parents[concept][1] if concept in parents else None,
            statement_role=parents[concept][2] if concept in parents else None,
            is_abstract=is_abstract,
        )
        for concept, is_abstract in extension_elements.items()
    ]


if __name__ == "__main__":
    headers = {"User-Agent": "your-name/0.1 (you@example.com)"}
    concepts = extract_extension_concepts(
        "https://www.sec.gov/Archives/edgar/data/1318605/000162828025045968/tsla-20250930",
        "tsla",
        headers,
    )
    for c in concepts:
        if not c.is_abstract and c.parent_concept:
            print(f"{c.concept}")
            print(f"  label:  {c.label}")
            print(f"  parent: {c.parent_concept}  weight={c.weight:+.1f}")
            print(f"  role:   {c.statement_role}")