OCR (Optical Character Recognition): How It Works

March 31, 2026 · 12 min read

OCR (Optical Character Recognition) converts images of text—scanned documents, photos of signs, screenshots, handwritten notes—into machine-readable text you can search, edit, and process. From digitizing century-old archives to extracting receipt data for expense reports, OCR has become an essential technology in our increasingly digital world.

Whether you're building a document management system, creating a mobile scanning app, or simply trying to extract text from a PDF, understanding how OCR works will help you achieve better results and avoid common pitfalls.

What Is OCR?

Optical Character Recognition is the electronic conversion of images containing typed, printed, or handwritten text into machine-encoded text. At its core, OCR analyzes the visual patterns in an image to identify individual characters, words, and text structure.

Early OCR systems from the 1970s and 1980s relied on template matching—comparing each character shape against a database of known patterns. These systems were rigid, requiring specific fonts and high-quality inputs. Modern OCR uses deep learning neural networks that can recognize characters across vast ranges of fonts, sizes, orientations, and quality levels.

Today's OCR technology powers countless applications:

• Document digitization: Converting paper archives into searchable digital databases
• Mobile scanning: Turning smartphone photos into editable text
• Automated data entry: Extracting information from invoices, receipts, and forms
• License plate recognition: Identifying vehicles for parking and toll systems
• Check processing: Reading account numbers and amounts on bank checks
• Book digitization: Creating searchable e-books from printed volumes
• Real-time translation: Translating signs and menus through camera apps
• Accessibility tools: Reading printed text aloud for visually impaired users

How OCR Works

Modern OCR is a multi-stage pipeline that transforms raw image pixels into structured text. Understanding each stage helps you optimize inputs and troubleshoot problems.

Stage 1: Image Acquisition

The process begins with capturing or loading the image. This might be a photo from a smartphone camera, a scan from a flatbed scanner, or a screenshot. The quality of this initial image significantly impacts final accuracy.

Key considerations during acquisition:

• Resolution should be at least 300 DPI for printed text
• Color depth can be 24-bit color, 8-bit grayscale, or 1-bit black-and-white
• File format matters less than image quality (JPEG, PNG, TIFF all work)
• Lighting should be even without shadows or glare

Stage 2: Preprocessing

Raw images rarely provide optimal input for character recognition. Preprocessing enhances the image and removes noise that could confuse the OCR engine.

Common preprocessing operations include:

1. Deskewing: Rotating the image to align text horizontally
2. Despeckling: Removing small dots and artifacts from scanning
3. Binarization: Converting to pure black text on white background
4. Border removal: Eliminating page edges and margins
5. Layout analysis: Identifying text regions, columns, and reading order
6. Line detection: Segmenting text into individual lines
7. Word segmentation: Separating lines into words
8. Character segmentation: Isolating individual characters (for some engines)

Stage 3: Character Recognition

This is where the actual "reading" happens. Modern OCR engines use LSTM (Long Short-Term Memory) neural networks that process text line-by-line, considering context to disambiguate similar-looking characters.

For example, the network learns that "l" (lowercase L) and "1" (number one) look similar but appear in different contexts—"l" appears in words while "1" appears in numbers. Similarly, "O" (letter) versus "0" (zero), "S" versus "5", and "B" versus "8" are distinguished by surrounding characters.

The recognition engine outputs not just characters but confidence scores for each recognition. A character recognized with 99% confidence is more reliable than one at 60% confidence.

Stage 4: Post-Processing

Raw OCR output often contains errors. Post-processing applies linguistic knowledge to correct likely mistakes:

• Dictionary lookup: Checking if recognized words exist in the language
• Spell checking: Correcting "rnedicine" to "medicine" (common rn/m confusion)
• Language models: Using context to fix errors ("the cat" not "the c@t")
• Format validation: Ensuring dates, phone numbers, and emails match expected patterns
• Confidence filtering: Flagging low-confidence recognitions for manual review

Stage 5: Output Generation

Finally, the recognized text is formatted for output. This might be:

• Plain text with all formatting removed
• Structured data (JSON, XML) with position coordinates
• Searchable PDF with invisible text layer over original image
• HTML preserving layout, fonts, and formatting
• Word or Excel documents with editable content

OCR Accuracy Factors

OCR accuracy varies dramatically based on input quality. Understanding what affects accuracy helps you prepare better inputs and set realistic expectations.

Practical Accuracy Tips

For scanning documents:

• Use 300 DPI for standard documents, 400-600 DPI for small text
• Flatten wrinkled pages before scanning (use a book or heavy object)
• Clean the scanner glass to remove dust and smudges
• Use grayscale mode for black-and-white documents (better than color)
• Enable automatic deskew in scanner software if available

For smartphone photos:

• Hold the phone parallel to the document (not at an angle)
• Use natural daylight or bright indoor lighting
• Avoid flash—it creates glare and harsh shadows
• Tap to focus on the text before capturing
• Fill the frame with the document (get close)
• Use document scanning apps that auto-crop and enhance

For screenshots:

• Capture at native resolution (don't resize before OCR)
• Avoid compression artifacts (use PNG instead of JPEG)
• Ensure text is rendered clearly (zoom in if needed)
• Disable font smoothing/anti-aliasing if possible

Preprocessing Techniques

Preprocessing can dramatically improve OCR accuracy. Here are the most effective techniques and when to use them.

Binarization (Thresholding)

Converting grayscale images to pure black-and-white simplifies recognition. The challenge is choosing the right threshold value.

Global thresholding uses a single threshold for the entire image. Works well for evenly-lit documents but fails when lighting varies across the page.

Adaptive thresholding calculates different thresholds for different regions. Essential for photos with uneven lighting or shadows. Otsu's method is a popular automatic approach.

Noise Reduction

Scanned documents often contain speckles, dust marks, and scanning artifacts. Noise reduction removes these without damaging text.

Common techniques:

• Median filtering: Removes salt-and-pepper noise
• Morphological operations: Opening removes small white spots, closing removes small black spots
• Connected component analysis: Removes objects too small to be text

Deskewing

Text must be horizontal for optimal recognition. Deskewing detects the text angle and rotates the image to correct it.

Most OCR engines include automatic deskewing, but manual correction may be needed for severely rotated images (more than 10-15 degrees).

Border Removal

Page edges, scanner borders, and margins can confuse layout analysis. Detecting and removing these improves results, especially for multi-column documents.

Contrast Enhancement

Faded documents benefit from contrast enhancement. Histogram equalization spreads out intensity values to maximize contrast. Be careful not to over-enhance, which can create artifacts.

Language Support

Modern OCR engines support 100+ languages, but accuracy varies significantly based on script type, character complexity, and training data availability.

Latin Script Languages

Languages using the Latin alphabet (English, French, German, Spanish, Italian, Portuguese, etc.) achieve the highest accuracy—often 99%+ on clean printed text. These languages have:

• Limited character sets (26 letters plus diacritics)
• Extensive training data
• Decades of OCR research and optimization
• Strong language models for post-processing

CJK Languages

Chinese, Japanese, and Korean present unique challenges with thousands of characters. Despite this complexity, modern neural networks handle them well:

• Chinese: 3,000-5,000 common characters, both simplified and traditional variants
• Japanese: Mix of kanji, hiragana, and katakana scripts
• Korean: Hangul syllable blocks (simpler than Chinese characters)

Accuracy for CJK languages on printed text typically reaches 95-98%, slightly lower than Latin scripts but still highly usable.

Right-to-Left Languages

Arabic, Hebrew, Persian, and Urdu read right-to-left and include contextual letter forms (characters change shape based on position in word). These require specialized handling:

• Bidirectional text support (mixing RTL and LTR text)
• Contextual form recognition
• Diacritic mark handling
• Ligature detection

Always specify the expected language to the OCR engine. This enables appropriate language models and character sets, significantly improving accuracy.

Multilingual Documents

Documents mixing multiple languages (like English with Chinese) require engines that can detect language changes and switch recognition models accordingly. Most modern engines support this, but accuracy may be lower at language boundaries.

Handwriting Recognition

Handwriting recognition (also called ICR - Intelligent Character Recognition) is significantly harder than printed text OCR. Human handwriting varies enormously in style, size, slant, and legibility.

What Works Well

Modern AI-based handwriting recognition achieves good results for:

• Block letters: Printed-style handwriting with separated characters
• Constrained forms: Single characters in boxes (like postal codes)
• Numeric digits: Numbers are easier than letters (fewer variations)
• Short text fields: Names, addresses, dates in structured forms

Accuracy for block letters can reach 90-95% on clear handwriting.

What Remains Challenging

Cursive handwriting remains the hardest problem in OCR:

• Connected letters make segmentation difficult
• Individual writing styles vary dramatically
• Letter shapes change based on surrounding letters
• Ambiguous characters (a/o, n/u, r/v) are common

Even state-of-the-art systems struggle with cursive, achieving only 70-80% accuracy on average handwriting and much lower on poor handwriting.

Improving Handwriting Recognition

To get better results with handwritten text:

1. Use constrained input: Boxes for individual characters work better than free-form text
2. Provide context: If the engine knows it's reading a date or phone number, accuracy improves
3. Train custom models: For specific handwriting styles (like a particular person's writing), custom training helps significantly
4. Combine with forms: Structured forms with labeled fields provide context clues
5. Use multiple recognizers: Combining results from different engines can improve accuracy
6. Enable manual review: Flag low-confidence recognitions for human verification

Signature Recognition

Signatures are a special case—they're not meant to be read as text but verified as authentic. Signature verification uses different techniques than OCR, focusing on stroke patterns, pressure, and timing rather than character recognition.

OCR Engines Comparison

Choosing the right OCR engine depends on your requirements: accuracy, speed, cost, language support, and deployment options.

Tesseract OCR

Originally developed by HP in the 1980s, now maintained by Google. Tesseract is the most popular open-source OCR engine.

Pros: Free, supports 100+ languages, runs locally (no API costs), actively maintained, good documentation.

Cons: Requires preprocessing for best results, lower accuracy than commercial engines on challenging documents, limited handwriting support.

Best practices: Use Tesseract 4.0+ with LSTM neural networks. Specify language with -l eng parameter. Preprocess images for better results. Consider page segmentation modes (--psm) for different layouts.

Cloud OCR Services

Google Cloud Vision, AWS Textract, and Azure Computer Vision offer state-of-the-art accuracy with minimal setup. They handle preprocessing automatically and provide structured output with confidence scores.

Pros: Highest accuracy, no infrastructure to manage, automatic updates, handle complex layouts, support handwriting.

Cons: Ongoing API costs, require internet connection, data leaves your infrastructure, rate limits apply.

Cost considerations: Most cloud services charge per 1,000 images. Prices range from $1.50-$3.00 per 1,000 pages. Free tiers typically include 1,000 pages/month.

Real-World Use Cases

OCR powers diverse applications across industries. Here are practical examples with implementation considerations.

Document Digitization

Converting paper archives to searchable digital databases. Libraries, government agencies, and corporations digitize millions of pages annually.

Requirements: High accuracy (99%+), layout preservation, batch processing, quality control workflow.

Implementation tips: Use commercial OCR for critical documents. Implement human review for low-confidence pages. Store both original images and OCR text. Create searchable PDFs with invisible text layer.

Invoice Processing

Automatically extracting vendor names, dates, amounts, and line items from invoices for accounts payable automation.

Requirements: Structured data extraction, table detection, multi-format support (PDF, images), integration with accounting systems.

Implementation tips: Use specialized document AI services (AWS Textract, Azure Form Recognizer). Train custom models for your specific invoice formats. Validate extracted amounts against expected ranges. Flag anomalies for manual review.

Receipt Scanning

Mobile apps that photograph receipts and extract merchant, date, total, and tax for expense tracking.

Requirements: Fast processing, works on smartphone photos, handles crumpled receipts, extracts key fields.

Implementation tips: Use cloud OCR APIs for best accuracy. Implement client-side image enhancement (crop, rotate, contrast). Extract structured data with regex patterns. Store original images for audit trail.

License Plate Recognition (ALPR)

Identifying vehicle license plates for parking enforcement, toll collection, and security systems.

Requirements: Real-time processing, works on moving vehicles, handles various plate formats, high accuracy (99.5%+).

Implementation tips: Use specialized ALPR engines (not general OCR). Implement vehicle detection before plate recognition. Handle multiple plates per image. Validate against known plate formats.

Business Card Scanning

Extracting contact information from business cards into address books and CRM systems.

Requirements: Field extraction (name, title, company, phone, email), handles various layouts, mobile-friendly.

Implementation tips: Use OCR with named entity recognition. Parse extracted text into structured fields. Validate email addresses and phone numbers. Handle international formats.

Real-Time Translation

Camera apps that translate signs, menus, and documents in real-time by overlaying translated text.

Requirements: Low latency (<1 second), works on video frames, handles perspective distortion, multiple languages.

Implementation tips: Use mobile-optimized OCR (on-device when possible). Implement text tracking across frames. Cache translations for repeated text. Handle mixed-language content.

Accessibility Tools

Reading printed text aloud for visually impaired users, converting textbooks to audio, and enabling screen readers for scanned documents.

Requirements: High accuracy, preserves reading order, handles complex layouts, integrates with text-to-speech.

Implementation tips: Prioritize reading order detection. Describe non-text elements (images, charts). Provide navigation by headings and sections. Support multiple output formats (audio, braille).