Document Smart Auto-Categorization

[adriandarian]

Description

Implement intelligent auto-categorization that analyzes document content, filenames, and context to automatically suggest appropriate folders and tags for new documents.

---

Requirements

• Analyze document content (text extraction)
• Parse filename for clues
• Analyze linked application data
• Suggest folder based on document type
• Suggest tags based on content keywords
• One-click to accept suggestions
• Learn from user corrections
• Batch auto-categorize existing documents

---

Auto-Categorization Rules

1. Document Type Detection

Resumes/CVs:

• Keywords: "experience", "education", "skills", "work history"
• Filename patterns: "resume", "cv", "resume", "curriculum"
• Suggested folder: "Resumes"
• Suggested tags: "resume", "cv", role level (junior/senior/lead)

Cover Letters:

• Keywords: "dear hiring manager", "i am writing to", "position"
• Filename patterns: "cover_letter", "cl_", "_cover"
• Suggested folder: "Cover Letters"
• Suggested tags: "cover-letter", company name, position

Certificates:

• Keywords: "certificate", "certification", "completion", "licensed"
• Filename patterns: "cert", "certificate", "license"
• Suggested folder: "Certifications"
• Suggested tags: "certification", skill name, issuing org

Portfolio Items:

• Keywords: "project", "portfolio", "sample work"
• Filename patterns: "portfolio", "project_"
• Suggested folder: "Portfolio"
• Suggested tags: "portfolio", "project", tech stack

References:

• Keywords: "reference", "recommendation", "reference letter"
• Filename patterns: "reference", "rec_letter"
• Suggested folder: "References"
• Suggested tags: "reference", person name

---

Content Analysis

Text Extraction

const extractText = async (file: File) => {
  if (file.type === 'application/pdf') {
    return await extractPDFText(file);
  } else if (file.type.includes('word')) {
    return await extractDOCXText(file);
  } else if (file.type === 'text/plain') {
    return await file.text();
  }
  return '';
};

Keyword Extraction

const extractKeywords = (text: string) => {
  // Remove stop words
  const cleaned = removeStopWords(text);
  
  // Extract n-grams (1-3 words)
  const ngrams = extractNGrams(cleaned, 1, 3);
  
  // Score by frequency and importance
  const scored = scoreKeywords(ngrams);
  
  // Return top 10 keywords
  return scored.slice(0, 10);
};

Company Detection

const detectCompanies = (text: string, filename: string) => {
  const companies = [];
  
  // Check against known companies list
  KNOWN_COMPANIES.forEach(company => {
    if (text.includes(company) || filename.includes(company)) {
      companies.push(company);
    }
  });
  
  // Check against user's application companies
  const userCompanies = getUserApplicationCompanies();
  userCompanies.forEach(company => {
    if (text.includes(company) || filename.includes(company)) {
      companies.push(company);
    }
  });
  
  return [...new Set(companies)];
};

---

Filename Parsing

Pattern Matching

const parseFilename = (filename: string) => {
  const suggestions = {
    documentType: null,
    company: null,
    role: null,
    version: null,
    date: null
  };
  
  // Document type
  if (/resume|cv/i.test(filename)) {
    suggestions.documentType = 'resume';
  } else if (/cover[_-]?letter|cl/i.test(filename)) {
    suggestions.documentType = 'cover-letter';
  }
  
  // Company name
  const companyMatch = filename.match(/(?:_|-)([A-Z][a-z]+(?:[A-Z][a-z]+)*)/);
  if (companyMatch) {
    suggestions.company = companyMatch[1];
  }
  
  // Version
  const versionMatch = filename.match(/v?(\d+)/);
  if (versionMatch) {
    suggestions.version = versionMatch[1];
  }
  
  // Date
  const dateMatch = filename.match(/(\d{4}[-_]?\d{2}[-_]?\d{2})/);
  if (dateMatch) {
    suggestions.date = parseDate(dateMatch[1]);
  }
  
  return suggestions;
};

---

Learning from User Corrections

Feedback Loop

const learnFromCorrection = (
  document: Document,
  suggestedFolder: string,
  actualFolder: string,
  suggestedTags: string[],
  actualTags: string[]
) => {
  // Store correction
  const correction = {
    documentType: document.type,
    keywords: document.keywords,
    suggestedFolder,
    actualFolder,
    suggestedTags,
    actualTags,
    timestamp: new Date()
  };
  
  // Update learning model
  updateCategorizationModel(correction);
  
  // Apply to similar pending documents
  applyCorrectionToSimilar(document, actualFolder, actualTags);
};

---

UI Design

Auto-Categorization Suggestions

┌────────────────────────────────────────────┐
│ 📄 John_Doe_Resume_2025.pdf uploaded       │
├────────────────────────────────────────────┤
│ 🤖 Smart Suggestions:                      │
│                                            │
│ Folder: 📁 Resumes                         │
│ [Change]                                   │
│                                            │
│ Tags: 🏷️ resume, senior-engineer, backend │
│ [+ Add] [× Remove]                         │
│                                            │
│ Detected:                                  │
│ • Document type: Resume                    │
│ • Experience level: Senior (8+ years)      │
│ • Skills: React, TypeScript, Node.js       │
│ • Location: San Francisco                  │
│                                            │
│ [Reject All] [Accept Suggestions]          │
└────────────────────────────────────────────┘

Batch Auto-Categorization

┌────────────────────────────────────────────┐
│ 🔄 Auto-Categorize Existing Documents      │
├────────────────────────────────────────────┤
│ 47 uncategorized documents found           │
│                                            │
│ Review Suggestions:                        │
│                                            │
│ ☑ Senior_Dev_Resume.pdf                   │
│   → Resumes / #resume #senior             │
│                                            │
│ ☑ Google_Cover_Letter.pdf                 │
│   → Cover Letters / #cover-letter #google │
│                                            │
│ ☑ AWS_Certification.pdf                   │
│   → Certifications / #aws #cert           │
│                                            │
│ ... 44 more                                │
│                                            │
│ [Select All] [Deselect All]                │
│ [Cancel]         [Apply to Selected (47)]  │
└────────────────────────────────────────────┘

---

Technical Implementation

Files

src/features/documents/auto-categorization/
  ├── AutoCategorization.tsx
  ├── SuggestionCard.tsx
  ├── BatchCategorization.tsx
  ├── utils/
  │   ├── textExtraction.ts
  │   ├── keywordExtraction.ts
  │   ├── filenameParser.ts
  │   ├── companyDetection.ts
  │   └── learningModel.ts
  └── hooks/
      ├── useAutoCategorization.ts
      └── useLearningModel.ts

Data Structure

interface AutoCategorizationSuggestion {
  documentId: string;
  suggestedFolder: string;
  confidence: number; // 0-1
  suggestedTags: Array<{
    tag: string;
    confidence: number;
    reason: string;
  }>;
  detectedInfo: {
    documentType: string;
    companies: string[];
    skills: string[];
    keywords: string[];
  };
}

---

Acceptance Criteria

• Content analysis extracts text correctly
• Keyword extraction identifies relevant terms
• Filename parsing works with common patterns
• Company detection finds company names
• Folder suggestions appropriate
• Tag suggestions relevant
• Can accept/reject suggestions
• Learning from corrections works
• Batch categorization functional
• Confidence scores reasonable
• Performance good with large files

---

Machine Learning Approach

Use simple rules-based system first, then enhance with:

• TF-IDF for keyword importance
• Naive Bayes for classification
• Pattern matching with regex
• User feedback loop for improvement

---

Testing Requirements

• Test text extraction from PDF/DOCX
• Test keyword extraction accuracy
• Test filename parsing patterns
• Test company detection
• Test suggestion quality
• Test learning from corrections
• Test batch operations
• Test with various document types

---

Related Tasks

• task-200: Custom Folder System
• task-201: Multi-Tag System
• task-064: Document Linking