Protobuf Schema Generation from Sample Data

What is Protocol Buffers?

Protocol Buffers (protobuf) is Google's language-neutral, platform-neutral mechanism for serializing structured data. Think of it as a faster, smaller, more efficient alternative to XML or JSON for data interchange. Originally developed internally at Google, protobuf has become the backbone of gRPC — the high-performance RPC framework used by companies like Netflix, Square, Lyft, and countless microservice architectures worldwide.

Unlike JSON or XML, protobuf uses a schema definition (a .proto file) to describe the structure of your data. This schema is then compiled into language-specific code (C++, Java, Python, Go, JavaScript, etc.) that handles serialization and deserialization. The result is data that's 3-10x smaller than JSON and 20-100x faster to parse — critical advantages in high-throughput systems.

In this guide, we'll focus on a common practical challenge: generating protobuf schemas from existing data. Whether you're migrating from JSON to protobuf, bootstrapping a new gRPC service, or documenting an existing data structure, auto-generating schemas from sample data can save significant development time.

Understanding Proto3 Syntax

Before we generate schemas, let's understand the proto3 syntax — the current version of the protobuf language:

syntax = "proto3";

package myapp.data;

// Import other proto files
import "google/protobuf/timestamp.proto";

// Define a message (like a struct/class)
message User {
    string id = 1;
    string name = 2;
    string email = 3;
    int32 age = 4;
    bool is_active = 5;
    repeated string tags = 6;
    Address address = 7;
    google.protobuf.Timestamp created_at = 8;
}

message Address {
    string street = 1;
    string city = 2;
    string state = 3;
    string zip_code = 4;
    string country = 5;
}

// Define a service (for gRPC)
service UserService {
    rpc GetUser(GetUserRequest) returns (User);
    rpc ListUsers(ListUsersRequest) returns (stream User);
}

Key Proto3 Concepts

• Messages — The primary data structure, similar to a class or struct
• Fields — Each field has a type, name, and unique field number
• Field numbers — Must be unique within a message and should never be reused (even if a field is removed)
• Scalar types — string, int32, int64, float, double, bool, bytes
• Repeated fields — Arrays/lists using the repeated keyword
• Nested messages — Messages can contain other messages
• Enums — Enumerated types for fixed sets of values
• Oneofs — Fields where only one can be set at a time

Generating Schemas from JSON Data

The most common scenario is generating a protobuf schema from existing JSON data. Here's a Python script that analyzes JSON and produces a .proto file:

import json
import sys
from collections import defaultdict

def infer_proto_type(value):
    """Infer protobuf type from a Python value."""
    if isinstance(value, bool):
        return 'bool'
    elif isinstance(value, int):
        if -2147483648 <= value <= 2147483647:
            return 'int32'
        return 'int64'
    elif isinstance(value, float):
        return 'double'
    elif isinstance(value, str):
        return 'string'
    elif isinstance(value, list):
        if len(value) > 0:
            return f'repeated {infer_proto_type(value[0])}'
        return 'repeated string'
    elif isinstance(value, dict):
        return 'message'
    elif value is None:
        return 'string'  # Default to string for null values
    return 'string'

def json_to_proto(data, message_name='Record', package='data'):
    """Generate a .proto file from JSON data."""
    messages = {}
    
    def process_object(obj, msg_name):
        fields = []
        for i, (key, value) in enumerate(obj.items(), 1):
            field_name = key.replace(' ', '_').replace('-', '_').lower()
            
            if isinstance(value, dict):
                sub_msg = msg_name + '_' + field_name.title()
                process_object(value, sub_msg)
                fields.append((sub_msg, field_name, i, False))
            elif isinstance(value, list) and len(value) > 0 and isinstance(value[0], dict):
                sub_msg = msg_name + '_' + field_name.title()
                process_object(value[0], sub_msg)
                fields.append((sub_msg, field_name, i, True))
            else:
                proto_type = infer_proto_type(value)
                is_repeated = proto_type.startswith('repeated ')
                if is_repeated:
                    proto_type = proto_type.replace('repeated ', '')
                fields.append((proto_type, field_name, i, is_repeated))
        
        messages[msg_name] = fields
    
    # Handle array of objects
    if isinstance(data, list):
        if len(data) > 0 and isinstance(data[0], dict):
            # Merge keys from all objects
            merged = {}
            for obj in data:
                for k, v in obj.items():
                    if k not in merged or merged[k] is None:
                        merged[k] = v
            process_object(merged, message_name)
        else:
            messages[message_name] = [('string', 'value', 1, True)]
    elif isinstance(data, dict):
        process_object(data, message_name)
    
    # Generate .proto file
    output = f'syntax = "proto3";\n\npackage {package};\n\n'
    
    for msg_name, fields in messages.items():
        output += f'message {msg_name} {{\n'
        for proto_type, field_name, number, is_repeated in fields:
            prefix = 'repeated ' if is_repeated else ''
            output += f'  {prefix}{proto_type} {field_name} = {number};\n'
        output += '}\n\n'
    
    return output.strip()

# Usage
if __name__ == '__main__':
    sample = [
        {"id": 1, "name": "Alice", "email": "[email protected]", "age": 30},
        {"id": 2, "name": "Bob", "email": "[email protected]", "age": 25}
    ]
    print(json_to_proto(sample, 'User', 'myapp'))

Generating Schemas from CSV Data

CSV files present a unique challenge because all values are strings. Type inference must be more sophisticated:

import csv
import re
from io import StringIO

def infer_csv_type(values):
    """Infer the best protobuf type from a column of CSV values."""
    non_empty = [v for v in values if v.strip()]
    if not non_empty:
        return 'string'
    
    # Try bool
    if all(v.lower() in ('true', 'false', 'yes', 'no', '1', '0') for v in non_empty):
        return 'bool'
    
    # Try int
    try:
        parsed = [int(v) for v in non_empty]
        if all(-2147483648 <= v <= 2147483647 for v in parsed):
            return 'int32'
        return 'int64'
    except ValueError:
        pass
    
    # Try float
    try:
        [float(v) for v in non_empty]
        return 'double'
    except ValueError:
        pass
    
    # Try date/timestamp
    date_patterns = [
        r'\d{4}-\d{2}-\d{2}',
        r'\d{2}/\d{2}/\d{4}',
    ]
    if any(all(re.match(p, v) for v in non_empty) for p in date_patterns):
        return 'string'  # Dates as strings, or use google.protobuf.Timestamp
    
    return 'string'

def csv_to_proto(csv_text, message_name='Record', package='data'):
    """Generate a .proto file from CSV data."""
    reader = csv.DictReader(StringIO(csv_text))
    rows = list(reader)
    
    if not rows:
        return 'syntax = "proto3";\n\n// No data found'
    
    headers = list(rows[0].keys())
    
    # Collect all values per column for type inference
    columns = {h: [row.get(h, '') for row in rows] for h in headers}
    
    output = f'syntax = "proto3";\n\npackage {package};\n\n'
    output += f'message {message_name} {{\n'
    
    for i, header in enumerate(headers, 1):
        field_name = header.strip().replace(' ', '_').replace('-', '_').lower()
        field_name = re.sub(r'[^a-zA-Z0-9_]', '', field_name)
        proto_type = infer_csv_type(columns[header])
        output += f'  {proto_type} {field_name} = {i};\n'
    
    output += '}\n'
    return output

Using ConvertMatrix for Quick Schema Generation

For quick protobuf schema generation without writing any code, ConvertMatrix offers instant conversion from multiple formats:

• CSV to Protobuf — Generate schemas from CSV files
• JSON to Protobuf — Infer schemas from JSON data
• Excel to Protobuf — Convert spreadsheets to .proto schemas
• SQL to Protobuf — Generate schemas from SQL table definitions

Simply paste your data or upload a file, and the converter generates a valid .proto file with properly inferred types and field numbers.

Schema Evolution Best Practices

One of protobuf's greatest strengths is backward and forward compatibility. Follow these rules to keep your schemas evolvable:

Rules for Schema Evolution

1. Never reuse field numbers — Once a field number is assigned, it should never be reused, even if the field is removed
2. Use reserved for removed fields — Mark deprecated field numbers and names as reserved
3. Add new fields with new numbers — New fields should always use the next available field number
4. Don't change field types — Changing a field from int32 to string breaks compatibility
5. Use optional for new fields — In proto3, all fields are implicitly optional, but be explicit about it

message User {
    string id = 1;
    string name = 2;
    // Field 3 was "age" - removed in v2
    reserved 3;
    reserved "age";
    
    string email = 4;
    // Added in v2
    string phone = 5;
    // Added in v3
    repeated string roles = 6;
}

Type Mapping Reference

Validating Generated Schemas

After generating a schema, always validate it by compiling with protoc:

# Install protoc (Protocol Buffer Compiler)
# macOS: brew install protobuf
# Ubuntu: apt install protobuf-compiler
# Windows: Download from github.com/protocolbuffers/protobuf/releases

# Validate the schema
protoc --proto_path=. --descriptor_set_out=/dev/null my_schema.proto

# Generate Python code
protoc --python_out=./gen my_schema.proto

# Generate Go code
protoc --go_out=./gen --go_opt=paths=source_relative my_schema.proto

# Generate JavaScript code
protoc --js_out=import_style=commonjs,binary:./gen my_schema.proto

Performance Comparison: Protobuf vs JSON vs XML

Conclusion

Generating protobuf schemas from sample data is a practical shortcut that accelerates API development and data migration projects. Whether you use Python scripts for programmatic generation, ConvertMatrix for quick browser-based conversion, or custom tooling tailored to your data pipeline, the key is to start with a well-structured schema and evolve it carefully using protobuf's built-in versioning support. Always validate generated schemas with protoc, follow schema evolution best practices, and leverage protobuf's performance advantages in high-throughput systems.