Introduction

Manual data entry is one of the most persistent inefficiencies in government service delivery. In programs like SNAP (Supplemental Nutrition Assistance Program), state caseworkers still rely heavily on handwritten applications — many of which must be manually transcribed into digital systems.

Recently I attended a conference where this was a hot topic among frontline workers. Their frustrations sparked a thought: What if we could use AI to eliminate manual entry altogether?

The Problem: A System Still Built on Paper

Many states still accept or even require paper-based applications for SNAP and similar programs. These forms are often:
- Handwritten by applicants
- Incomplete or illegible
- Manually entered by staff into eligibility or case management systems

“It’s 2025, and we’re still typing what someone else already wrote down on paper.”

Experimenting With AI and Handwriting Recognition

Curious whether recent advances in AI could solve this, I ran an experiment:
- I downloaded Missouri’s SNAP application
- Filled it out by hand (badly, on purpose)
- Ran the scanned image through a vision model

The results were surprisingly good.

Example Input: A Handwritten SNAP Form

Here’s the actual image used as the test input:

This wasn't a generic prompt — I designed a form-specific instruction set that told the AI model how to interpret the layout and labeling of this particular SNAP application. That targeted prompting (often referred to as few-shot instruction engineering) made all the difference.

By aligning the AI’s expectations with the exact structure of the form — including field locations, checkbox patterns, and handwriting placement — the model was able to extract data far more reliably than it would with generic OCR tools.

The Output: Structured, Accurate Data

Below is the full structured output the model returned:

{
"Full Name": "Marge B. Simpson",
"Date Signed": "08-01-2025",
"Home Address": "742 Evergreen Terrace, Springfield, MO 65807",
"County": "Greene",
"Mailing Address": "Same as home",
"Phone Number": "(417) 555-1234",
"Email": "marge.simpson@email.com",
"Best Contact Method": "Call and Text",
"Identity Verifier": "Ned Flanders – (417) 555-6789",
"Expedited Screening": {
    "Income < $150 and assets < $100": true,
    "Rent/utilities > income + assets": true,
    "Seasonal/Migrant Worker": false
},
"Household Members": [
    {
      "Name": "Marge Simpson",
      "Relationship": "Self",
      "DOB": "03-19-1980",
      "SSN": "123-45-6789",
      "Sex": "F",
      "Race": "White",
      "Hispanic": false,
      "Buy & Cook": true
    },
    {
      "Name": "Bart Simpson",
      "Relationship": "Child",
      "DOB": "04-01-2010",
      "SSN": "123-45-1111",
      "Sex": "M",
      "Race": "White",
      "Hispanic": false,
      "Buy & Cook": true
    },
    {
      "Name": "Lisa Simpson",
      "Relationship": "Child",
      "DOB": "05-09-2012",
      "SSN": "123-45-2222",
      "Sex": "F",
      "Race": "White",
      "Hispanic": false,
      "Buy & Cook": true
    },
    {
      "Name": "Maggie Simpson",
      "Relationship": "Child",
      "DOB": "11-12-2020",
      "SSN": "123-45-3333",
      "Sex": "F",
      "Race": "White",
      "Hispanic": false,
      "Buy & Cook": true
    }
],
"Non-citizens in Household": true,
"New Missouri EBT Card Needed": true,
"Preferred Language": "English",
"Legal Declarations": {
    "Convicted of SNAP fraud > $500": false,
    "Fleeing prosecution/felony": false,
    "Violating parole/probation": false,
    "False identity for benefits": false,
    "Felony for illegal drugs after 8/22/96": false,
    "Duplicate SNAP benefits": false,
    "Traded SNAP for guns/ammunition": false,
    "Traded SNAP for drugs": false,
    "Convicted of serious crimes since 2/7/14": false
},
"Household Details": {
    "Received SNAP in another state": false,
    "Disabled household members": false,
    "Adult in school": false
},
"Resources": [
    {
      "Type": "Savings Account",
      "Bank": "Springfield First National",
      "Owner": "Marge Simpson",
      "Balance": "$57.12"
    }
],
"Employment Income": [
    {
      "Name": "Marge Simpson",
      "Employer": "Kwik-E-Mart",
      "Gross Monthly Income": "$850",
      "Frequency": "Monthly"
    }
],
"Other Income": [
    {
      "Source": "Child Support",
      "Recipient": "Marge Simpson",
      "Amount/Month": "$300"
    }
],
"Recent Income Change": {
    "Who": "Marge Simpson",
    "Last Paycheck Date": "07/15/2025",
    "Gross Amount": "$540.00"
},
"Monthly Expenses": [
    {"Type": "Rent", "Amount": "$650", "Paid By": "Marge Simpson", "Heats/Cools Home": true},
    {"Type": "Electric", "Amount": "$75", "Paid By": "Marge Simpson", "Heats/Cools Home": true},
    {"Type": "Gas", "Amount": "$45", "Paid By": "Marge Simpson", "Heats/Cools Home": true},
    {"Type": "Water", "Amount": "$25", "Paid By": "Marge Simpson"},
    {"Type": "Trash", "Amount": "$15", "Paid By": "Marge Simpson"}
],
"Other Deductions": {
    "Court-Ordered Support": false,
    "Medical Expenses 60+/Disabled": false
},
"Legal Agreements & Signature": {
    "Understood Fraud Penalties": true,
    "Info Verifiable by Authorities": true,
    "Work Registration Accepted": true,
    "ABAWD Rules Understood": true,
    "Signature Provided": true,
    "Opted Out of Calls/Texts": false,
    "Witness Signature Required": false
}
}

Process Visualization: Manual vs AI Workflow

To help illustrate the difference between legacy and modern workflows, here’s a side-by-side visual:

Why It Works: AI + Computer Vision

Modern AI vision models combine OCR (optical character recognition) with deep learning-based document understanding. Unlike traditional OCR tools, AI Vision can:
- Understand poor handwriting
- Interpret checkboxes and structured tables
- Maintain context across mixed content

Technical Note: Performance was better with high-resolution images than with raw PDFs. Processing PDFs required a recursive approach to convert to image preprocessing.

Why This Matters

Time Savings: Manual entry of 10–20 fields per application adds up quickly.
- Accuracy: Eliminates transcription errors from misreading handwriting.
- Scalability: Forms from multiple programs can be trained with similar templates.
- Interoperability: Structured output can be fed directly into state systems like eligibility engines or ERPs.

What’s Next: Scaling This for Real Use

Here’s how states could start:
- Define a set of standard, scanned form templates
- Train/test models on real handwriting samples
- Pilot in counties or districts with high SNAP volume
- Embed image upload into public-facing portals or kiosks

Closing Thoughts and Future Use

Handwriting solutions have existed for several years, often as components of document management systems. The current approach demonstrates AI as a tool that can be applied in various ways. With advancements in technical capabilities, implementing AI augmentation may help governments decrease clerical workloads, increase service delivery speed, and enhance citizen interactions.

This is part one of a three-part post. Next we will look at the document intake portion of the workflow using only AI to process the content. (Sneak Peak)