Urdu Wikipedia Text Categorization System

A comprehensive Python-based system for extracting, processing, and categorizing Urdu Wikipedia articles into balanced datasets with intelligent content filtering. This project provides multiple processing modes including file generation, Excel output, and Jupyter notebook support for interactive analysis.

Features

Core Features

• Balanced Dataset Creation: Smart algorithm that creates roughly equal numbers of Short, Medium, and Large chunks
• Intelligent Content Filtering: Rejects repetitive content and enforces minimum quality standards
• Multi-mode Processing: Supports disk output, Excel generation, print mode, and Jupyter notebook execution
• Real-time Progress Tracking: Visual feedback with distribution monitoring during processing
• Incremental Saves: Excel and CSV files updated every 10 processed files

Text Processing

• Smart Text Splitting: Intelligent text chunking respecting Urdu punctuation (۔) boundaries
• Size Categorization: Automatically categorizes text into Short (1-50 words), Medium (51-100 words), and Large (101-300 words)
• Content Validation: Minimum 6 words for short content, repetitive pattern detection
• Duplicate Detection: Prevents duplicate URLs and sentences from being processed

Data Management

• Content Categorization: Classifies articles into 5 categories: Islam, History, Science, Personality, and General
• Template-based Output: Generates structured XML output files with metadata
• Excel Export: Creates organized Excel files for data analysis
• Natural Sorting: Proper numerical sorting of input files (article1 before article100)

Quality Assurance

• Repetition Detection: Analyzes word frequency to reject repetitive content (threshold: 60%)
• Minimum Word Count: Enforces at least 6 words for short content
• API Integration: Wikipedia API with proper headers and error handling
• Robust Error Handling: Comprehensive error handling for network requests and data processing

Project Structure

├── app.py                    # Main processing script (command-line) with balanced distribution
├── app.ipynb                 # Jupyter notebook version with interactive widgets
├── Sample Data set/          # Input Wikipedia articles (article1.txt, article2.txt, etc.)
├── txtOutput/               # Generated text files by size and type
│   ├── Short/               # Short articles (1-50 words)
│   │   ├── NC/              # News/Current Affairs
│   │   ├── LR/              # Literature/Research
│   │   ├── HR/              # Historical Records
│   │   └── NP/              # Notable Personalities
│   ├── Medium/              # Medium articles (51-100 words)
│   │   ├── NC/
│   │   ├── LR/
│   │   ├── HR/
│   │   └── NP/
│   └── Large/               # Large articles (101-300 words)
│       ├── NC/
│       ├── LR/
│       ├── HR/
│       └── NP/
├── ExcelOutput/             # Excel files for analysis
│   ├── short.xlsx           # Short articles data
│   ├── medium.xlsx          # Medium articles data
│   ├── large.xlsx           # Large articles data
│   └── combined.xlsx        # All articles combined
├── README.md
└── requirements.txt

Installation

1. Clone or download this repository
2. Create a virtual environment (recommended):

   python -m venv venv
   

3. Activate the virtual environment:
   • Windows: venv\Scripts\activate
   • macOS/Linux: source venv/bin/activate
4. Install required packages:

   pip install -r requirements.txt
   

Required Dependencies

• pandas - Data manipulation and Excel export
• requests - Wikipedia API calls
• openpyxl - Excel file generation
• ipywidgets - Jupyter notebook progress bars (for notebook version only)

Usage

Method 1: Command Line Processing (Recommended)

The app.py script provides a unified approach with multiple output modes and balanced distribution:

Command Line Usage

# Generate files to disk
python app.py --mode disk

# Generate Excel files only
python app.py --mode excel

# Print results to console
python app.py --mode print

# Generate both files and Excel (recommended)
python app.py --mode both

# Allow duplicate URLs (process all documents even with same URL)
python app.py --mode disk --allow-duplicate-urls

# Run content validation tests
python app.py test

Processing Modes

• disk: Creates structured text files in txtOutput/ folder organized by size and subcategory
• excel: Generates Excel files in ExcelOutput/ folder for analysis
• print: Displays results in console for debugging
• both: Combines disk and excel modes (recommended for complete output)

Features During Processing

• Real-time Progress: Shows current file being processed and completion percentage
• Distribution Tracking: Displays balanced distribution (S:X M:Y L:Z) during processing
• Incremental Saves: Excel files saved every 10 files to prevent data loss
• Graceful Interruption: Press Ctrl+C to stop processing and save current progress

Method 2: Jupyter Notebook (Interactive)

For interactive analysis and experimentation with visual progress bars:

1. Install Jupyter if not already installed:

   pip install jupyter ipywidgets
   

2. Launch Jupyter notebook:

   jupyter notebook
   

3. Open app.ipynb and run the cells interactively

Notebook Features

• Interactive Progress Bars: Visual progress tracking with ipywidgets
• Real-time Distribution Display: See balanced distribution as processing happens
• Cell-by-cell Execution: Run individual components for testing
• Data Analysis Tools: Built-in summary statistics and sample viewing
• Export Options: Flexible export to Excel with custom filtering

Configuration

Size Categories with Balanced Distribution

• Short: 1-50 words (minimum 6 words, non-repetitive)
• Medium: 51-100 words
• Large: 101-300 words

The system uses an intelligent balancing algorithm that:

• Tracks the count of each size category in real-time
• Prioritizes underrepresented categories to maintain balance
• Adds 30% randomness to prevent overly deterministic patterns
• Aims for roughly equal distribution across all three sizes

Content Validation Rules

Short Content Requirements

• Minimum Words: At least 6 words
• Repetition Threshold: Less than 60% repetitive content
• Unique Word Ratio: Must have sufficient vocabulary diversity

Repetition Detection

The system analyzes word frequency to detect repetitive patterns:

repetition_ratio = 1 - (unique_words / total_words)
# Content rejected if repetition_ratio > 0.6 (60%)

Examples of rejected content:

• "وصول کنندگان." (too short, only 2 words)
• "تاریخ. تاریخ." (too repetitive)
• Single word or phrase repeated multiple times

Content Categories (Keyword-based Classification)

• Islam: Religious content, Quran, Hadith, Islamic practices
  • Keywords: اسلام، مسلم، قرآن، حدیث، نماز، روزہ، زکوة، حج، دعا، سیرت، صحابہ، فقہ، تفسیر، شریعت، اللہ، ایمان، تصوف، عقیدہ، جہاد، سنت
• History: Historical events, civilizations, wars, empires
  • Keywords: تاریخ، قدیم، قرون وسطی، ثقافت، جنگ، سلطنت، بادشاہ، استعمار، نیا دور، آثار قدیمہ، تحریک، انقلاب، فوجی، تمدن، تہذیب، دور غلامی، عالمی جنگ، سلطنت عثمانیہ، یورپی نشاۃ ثانیہ، قدیم یونان
• Science: Physics, chemistry, biology, technology, research
  • Keywords: سائنس، طبیعیات، کیمیا، حیاتیات، ریاضیات، فلکیات، جینیات، ٹیکنالوجی، ایولوجی، مصنوعی ذہانت، میکانکس، روبوٹکس، پیتھولوجی، جغرافیہ، آب و ہوا، ارتقاء، کوانٹم، کائنات، توانائی، تحقیق
• Personality: Biographies, character traits, leadership
  • Keywords: سوانح عمری، شخصیت، زندگی، انسانی خصوصیات، کردار، قائد، لیڈر، ترقی، خواب، ہنر، عادات، معروف افراد، رہنما، حوصلہ افزائی، ذہانت، ذاتی ترقی، اثر و رسوخ، ماہر، ذمہ داری، نظریہ
• General: Geography, sports, culture, literature, economics
  • Keywords: معلومات، ریاضی، شہر، جزیرہ، صوبہ، کھیل، کھیلوں کے اصول، فٹبال، کرکٹ، ایشیائی ممالک، دریا، پہاڑ، آبادی، زبانیں، موسیقی، فنون، کتب، ادب، معیشت، تعلیم، ثقافت، ریاضیاتی مساوات

Subcategories (Random Assignment)

• NC: News/Current Affairs
• LR: Literature/Research
• HR: Historical Records
• NP: Notable Personalities

Output Formats

Text Files (XML Structure)

Generated files follow this template structure:

<features domain="general" document_size="medium" type="HR" url="https://ur.wikipedia.org/wiki?curid=12345">
<source>
[Urdu text content here]
</source>
</features>

Excel Files

Excel output includes the following columns:

• source: Original filename
• title: Wikipedia article title
• id: Wikipedia article ID (curid)
• url: Wikipedia article URL
• document_size: Size category (short/medium/large)
• domain: Content category (Islam/History/Science/Personality/General)
• sentence: Processed text content

Key Features

Balanced Distribution Algorithm

The system implements an intelligent balancing algorithm to ensure equal representation:

1. Real-time Tracking: Monitors count of Short, Medium, and Large chunks
2. Dynamic Prioritization: Calculates ratios and prioritizes underrepresented sizes
3. Smart Ordering: Sorts sizes by current ratio (ascending) to favor underrepresented
4. Controlled Randomness: 30% chance to shuffle for variety while maintaining balance
5. Iterative Processing: Continuously adjusts priorities as new chunks are created

Example output:

Distribution: S:639 M:638 L:637  (nearly perfect balance!)

Smart Text Processing

Iterative Chunk Extraction

The system processes text iteratively:

1. Gets balanced size order based on current distribution
2. Tries to extract chunk for each size in priority order
3. Validates content (word count, repetition check)
4. Updates global counters and adjusts priorities
5. Repeats until all text is processed or remaining text is invalid

Sentence Boundary Detection

• Urdu Punctuation Awareness: Respects Urdu sentence endings (۔) for natural text splitting
• Smart Cutting: Finds last sentence boundary within word limit range
• Fallback Mechanism: Cuts at max words if no good boundary found
• Minimum Word Enforcement: Ensures chunks meet minimum word requirements

Content Quality Validation

Repetition Detection Algorithm

def is_repetitive_content(text, threshold=0.6):
    words = extract_words(text)
    unique_words = count_unique(words)
    repetition_ratio = 1 - (unique_words / total_words)
    return repetition_ratio > threshold  # 60% threshold

Short Content Validation

def is_valid_short_content(text):
    if word_count < 6:
        return False  # Too short
    if is_repetitive_content(text):
        return False  # Too repetitive
    return True

Duplicate Prevention

• URL Deduplication: Tracks seen URLs to prevent duplicate processing
• Sentence Deduplication: Tracks seen sentences to prevent duplicate chunks
• Optional Override: --allow-duplicate-urls flag for special cases

API Integration

The system uses the Wikipedia API with proper headers:

• User-Agent header for identification (UnifiedApp/1.0)
• Timeout handling (10 seconds)
• Error handling for HTTP errors and JSON parsing
• Lightweight API calls for better performance
• Extracts article title and content for categorization

Progress Tracking

Command Line

• File-by-file progress with completion percentage
• Real-time distribution display (S:X M:Y L:Z)
• Progress bar updates every 10 files
• Incremental save notifications

Jupyter Notebook

• Interactive progress bars with ipywidgets
• Visual percentage completion
• Real-time status updates
• Distribution summary in widget

Error Handling

The system includes comprehensive error handling for:

• Network Issues: Graceful handling of connection failures
• Wikipedia API Errors: Rate limiting (429), forbidden (403), not found (404)
• JSON Parsing: Handles malformed or empty API responses
• File I/O: Encoding fallbacks (UTF-8 with error replacement)
• Keyboard Interrupts: Ctrl+C saves progress before exiting
• Invalid Content: Warns about unprocessable text (too short/repetitive)

Graceful Degradation

try:
    process_all(args.mode, allow_duplicate_urls=args.allow_duplicate_urls)
except KeyboardInterrupt:
    print("\n\n⚠️ Ctrl+C detected. Saving progress...")
finally:
    if args.mode in ["excel", "both"]:
        save_excels()  # Always save what we have

Performance Considerations

Memory Efficiency

• Streaming Processing: Processes files one at a time
• Incremental Saves: Writes to disk every 10 files to free memory
• Lazy Loading: Only loads current file content into memory
• Efficient Data Structures: Uses sets for O(1) duplicate detection

Processing Speed

• Natural Sorting: Pre-sorts files once for correct order
• Regex Optimization: Compiled patterns for faster matching
• Batch Operations: Groups related operations (e.g., Excel saves)
• Progress Tracking: Minimal overhead with efficient string formatting

Scalability

• Large Datasets: Can handle thousands of files
• Interrupt Recovery: Save progress allows resuming from interruption
• Configurable Batch Size: Adjust save frequency (currently every 10 files)
• Resource Management: Proper file handle cleanup

Testing

The system includes built-in test functionality:

python app.py test

This runs comprehensive tests for:

• Repetitive Content Detection: Tests various repetitive patterns
• Valid Content Validation: Ensures quality content passes filters
• Balanced Processing: Verifies distribution algorithm works correctly
• Sample Output: Shows example chunks with word counts

Test Output Example

🧪 Testing Content Validation Logic
==================================================

❌ Testing Repetitive/Invalid Content (should be rejected):
  1. 'وصول کنندگان.' (2 words)
     Repetitive: True, Valid for short: False

✅ Testing Valid Content (should be accepted):
  1. 'اصلی امریکی. یورپی نوآبادیوں کی ابتدا...' (35 words)
     Repetitive: False, Valid for short: True

📊 Balanced Distribution:
  Short: 3 chunks (33.3%)
  Medium: 3 chunks (33.3%)
  Large: 3 chunks (33.3%)

Troubleshooting

Common Issues

1. Unbalanced Distribution

   • The algorithm should create roughly equal numbers of each size
   • If imbalanced, check if source text has sufficient variety
   • Verify content validation isn't rejecting too many chunks

2. Too Many Warnings About Unprocessed Words

   • Usually indicates repetitive or very short content
   • Check source data quality
   • Adjust repetition threshold if needed (currently 60%)

3. JSONDecodeError: Empty API responses

   • Check internet connection
   • Verify Wikipedia article IDs are valid
   • API may be temporarily unavailable

4. 403 Forbidden Errors: Wikipedia blocking requests (FIXED)

   • System includes proper User-Agent headers
   • Should not occur with current implementation

5. File Not Found Errors: Missing input files

   • Ensure Sample Data set folder exists
   • Check for article*.txt files in the folder
   • Verify file paths and permissions

6. Unicode/Encoding Errors: Text encoding issues

   • System automatically handles encoding with fallbacks
   • Uses UTF-8 with error replacement as fallback

Debug Mode

For troubleshooting, you can modify the code to add debug prints:

# In process_all function
print(f"Processing file: {fname}")
print(f"Found {len(matches)} documents in file")
print(f"Current distribution: {global_size_counts}")
print(f"Chunk created: {size} with {count_words(chunk)} words")

Monitoring Distribution

Watch the real-time distribution during processing:

✅ Completed article1.txt: 5 chunks | Distribution: S:2 M:2 L:1
✅ Completed article2.txt: 8 chunks | Distribution: S:5 M:5 L:3

If distribution becomes heavily skewed, it may indicate:

• Source text predominantly contains one size range
• Content validation rejecting specific sizes
• Need to adjust word count ranges

Example Workflow

Complete Processing Pipeline

1. Prepare Input Data

   # Ensure Sample Data set folder contains article files
   # Files should be in format: <doc url="...">content</doc>
   

2. Run Processing

   # Process with both disk and Excel output
   python app.py --mode both
   

3. Monitor Progress

   📄 Processing article1.txt (1/100)
   ✅ Completed article1.txt: 5 chunks | Distribution: S:2 M:2 L:1
   Files |████████████████████████████████████████| 10/100 (10.0%) processed
   💾 Saving progress... | Distribution: S:15 M:14 L:13
   

4. Review Output

   # Check text files
   ls txtOutput/Short/NC/
   ls txtOutput/Medium/LR/
   ls txtOutput/Large/HR/
   
   # Check Excel files
   ls ExcelOutput/
   # short.xlsx, medium.xlsx, large.xlsx, combined.xlsx
   

5. Analyze Results

   • Open Excel files for data analysis
   • Check distribution balance in summary
   • Review sample content for quality

Jupyter Notebook Workflow

1. Launch Notebook

   jupyter notebook app.ipynb
   

2. Run Cells Sequentially

   • Configuration cell
   • Helper functions
   • Processing function
   • Execute processing with progress bar

3. View Results

   • Summary statistics automatically displayed
   • Sample content from each size category
   • Export filtered datasets

Algorithm Details

Balanced Distribution Algorithm

def get_balanced_size_order():
    total = sum(global_size_counts.values())
    if total == 0:
        # First time: random order
        return shuffle(["Short", "Medium", "Large"])

    # Calculate ratios
    ratios = {size: count/total for size, count in global_size_counts.items()}

    # Sort by ratio (ascending) - prioritize underrepresented
    sorted_sizes = sorted(ratios.keys(), key=lambda x: ratios[x])

    # 30% chance to shuffle for variety
    if random.random() < 0.3:
        random.shuffle(sorted_sizes)

    return sorted_sizes

Iterative Processing Flow

Input Text
    ↓
Get Balanced Size Order (based on current distribution)
    ↓
Try Each Size in Priority Order
    ↓
Extract Chunk (respect sentence boundaries)
    ↓
Validate Content (word count, repetition)
    ↓
If Valid: Save Chunk, Update Counters
    ↓
Repeat with Remaining Text
    ↓
Output: Balanced Dataset

Data Quality Metrics

Expected Output Quality

• Balanced Distribution: ±5% variance between sizes
• Minimum Word Count: All short content ≥6 words
• Repetition Ratio: All content <60% repetitive
• Unique Sentences: No duplicate sentences across dataset
• Valid URLs: All chunks linked to valid Wikipedia articles

Sample Statistics (from test run)

📊 Total records created: 1914

📈 Breakdown by size:
  Short: 639 (33.4%)
  Medium: 638 (33.3%)
  Large: 637 (33.3%)

🏷️ Breakdown by domain:
  History: 920
  General: 366
  Islam: 256
  Personality: 187
  Science: 110
  Uncategorized: 75

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test with python app.py test
5. Ensure balanced distribution is maintained
6. Submit a pull request

License

This project is for educational and research purposes. Please respect Wikipedia's terms of service and API usage guidelines.

Acknowledgments

• Wikipedia API for providing access to Urdu content
• Pandas library for efficient data manipulation
• ipywidgets for interactive notebook progress tracking

Contact

For questions or issues, please create an issue in the repository or contact the development team.