Python Logging Best Practices — From basicConfig to Structured Logging

Overview

Configure Python logging properly with handlers, formatters, log levels, and structured JSON logging for production. This guide covers the essential concepts, practical examples, and production-ready patterns you need to get started.

Getting Started

Before diving in, make sure you have the necessary prerequisites installed. This guide assumes basic familiarity with the underlying technology stack.

Core Concepts

Understanding the fundamentals is essential before applying advanced patterns. The following sections break down the key ideas with working code examples.

Best Practices

• Start simple and add complexity only when needed
• Write tests alongside your implementation
• Document decisions that aren't obvious from the code
• Monitor in production with appropriate logging and metrics

Common Pitfalls

Developers commonly run into a few specific issues when first implementing these patterns. Understanding them upfront saves significant debugging time later.

Production Checklist

• Error handling and graceful degradation
• Logging and observability
• Performance testing under realistic load
• Security review

Why Proper Logging Matters

Logging is essential for debugging, monitoring, and understanding application behavior in production. Python's built-in logging module is powerful but often underutilized.

Logging Levels Explained

import logging

logging.debug("Debug: Detailed information for debugging")
logging.info("Info: General information about operation")
logging.warning("Warning: Something unexpected happened")
logging.error("Error: A serious problem occurred")
logging.critical("Critical: A very serious error")

Level	Numeric Value	When to Use
DEBUG	10	Detailed diagnostic information
INFO	20	General operational messages
WARNING	30	Unexpected but handled events
ERROR	40	Errors that need attention
CRITICAL	50	Serious errors requiring immediate action

Basic Configuration

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    handlers=[
        logging.FileHandler('app.log'),
        logging.StreamHandler()
    ]
)

Using Loggers

logger = logging.getLogger(__name__)

def process_data(data):
    logger.debug(f"Processing: {data}")
    try:
        result = transform(data)
        logger.info(f"Transformed successfully")
        return result
    except Exception as e:
        logger.error(f"Transformation failed: {e}", exc_info=True)
        raise

Handlers, Formatters, and Filters

# Create custom formatter
formatter = logging.Formatter(
    '%(asctime)s | %(name)s | %(levelname)-8s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

# Console handler
console = logging.StreamHandler()
console.setLevel(logging.INFO)
console.setFormatter(formatter)

# File handler with rotation
from logging.handlers import RotatingFileHandler
file_handler = RotatingFileHandler(
    'app.log', maxBytes=10*1024*1024, backupCount=5
)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)

# Add to logger
logger.addHandler(console)
logger.addHandler(file_handler)

Structured Logging with JSON

import json
import logging

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            'timestamp': self.formatTime(record),
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
            'module': record.module,
            'function': record.funcName,
            'line': record.lineno
        }
        if record.exc_info:
            log_entry['exception'] = self.formatException(record.exc_info)
        return json.dumps(log_entry)

json_handler = logging.StreamHandler()
json_handler.setFormatter(JSONFormatter())
logger.addHandler(json_handler)

Best Practices

• Use appropriate log levels: Don't log everything as INFO
• Include context: Add relevant IDs, parameters, and state
• Log exceptions properly: Use exc_info=True or logger.exception()
• Avoid logging sensitive data: Never log passwords, tokens, PII
• Use lazy evaluation: logger.debug("Value: %s", expensive_func())

Frequently Asked Questions

What's the difference between print() and logging?

Logging provides levels, timestamps, filtering, and multiple outputs. Print goes only to stdout with no filtering.

How do I log to both file and console?

Add multiple handlers to your logger - one StreamHandler for console, one FileHandler for file.

Why aren't my logs appearing?

Check that your logger level is set correctly and handlers are properly configured. Remember: both logger AND handler must allow the level.

How do I rotate log files?

Use RotatingFileHandler or TimedRotatingFileHandler from logging.handlers.