StackLog #

Table of Contents #

• 🚀 Features
• 📦 Installation
• ⚡ Quick Start
• 📁 Input Format
• 🛠️ Usage
• 📊 Output
• 🏗️ Building from Source
• 👨‍💻 Author
• 📜 License
• 🚧 TODO

🚀 Features #

• 📊 Google Cloud Log Processing: Parse and analyze structured JSON logs from Google Cloud Platform (GCP)
• 🎨 Syntax Highlighting: Beautiful color-coded output for enhanced log readability
• 🔄 Dual Payload Support: Handles both textPayload and jsonPayload fields with intelligent preference
• 📁 Organized Output: Automatically creates structured output directory with categorized files

📦 Installation #

Prerequisites #

• Dart SDK 3.0+ - Required to run or compile the project

Method 1: From Source (Recommended for Development) #

Clone the repository:

git clone https://github.com/LazyLazyMeat/stack-log.git
cd stack-log

Install dependencies:

dart pub get

Verify installation:

dart run bin/parser.dart --help

Method 2: Using Pre-compiled Binaries #

Will be added soon

Method 3: Global Installation (After Building) #

Will be added soon

Verifying Installation #

Test that everything is working correctly:

dart run bin/parser.dart --help
dart run bin/highlighter.dart --help

Or with compiled versions

Will be added later

Platform-Specific Notes #

Windows:

• Use Command Prompt or PowerShell
• Ensure Dart SDK is in your PATH
• Build produces .exe files

Linux:

• Works with most common distributions
• Build produces executable binaries
• May need to set execute permissions: chmod +x build/*

⚡ Quick Start #

Step 1: Prepare raw log file #

You can find valid example of input.json in example/input.json

Step 2: Run the Parser #

• Basic usage (uses input.json in current directory): dart run bin/parser.dart
• Or specify your log file: dart run bin/parser.dart my_logs.json
• With custom output directory: dart run bin/parser.dart --input my_logs.json --output my_results

Step 3: View the Results #

• View the default output (output/output.txt): dart run bin/highlighter.dart
• Or specify a file: dart run bin/highlighter.dart output/output.txt

Expected Output:

• You should see color-coded logs like this:

(05.10.23 12:00:00:123) [INFO] <Mozilla/5.0...> User logged in successfully
(05.10.23 12:01:30:456) [ERROR] Database connection failed

Step 4: Explore Generated Files #

Check the output/ directory for all processed data:

ls output/
# output.txt            - Formatted logs (used by highlighter)
# output.json           - Structured JSON data
# json_payloads.json    - Parsed JSON payloads
# errors.txt            - Any processing errors
# run-log.txt           - Execution log

📁 Input Format #

The parser expects logs in the standard Google Cloud Logging JSON format. The input file must be a JSON array containing log entry objects.

Basic Structure #

[
  {
    // Log entry 1
  },
  {
    // Log entry 2
  }
]

Supported Fields #

Legend: ❌ Optional, ⚠️ At least one payload required

Complete example #

Minimal valid entry:

[
  {
    "severity": "INFO",
    "timestamp": "2025-11-01T12:00:00.000Z",
    "httpRequest": {
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    },
    "textPayload": "User session created for user@example.com"
  },
  {
    "severity": "ERROR",
    "timestamp": "2025-11-01T12:01:00.000Z",
    "jsonPayload": {
      "message": "Database connection timeout",
      "errorCode": "DB_CONN_001"
    }
  }
]

Field Priority #

If both textPayload and jsonPayload are present, jsonPayload takes precedence. Missing fields are not included in output, invalid fields are ignored.

Filtered Messages #

The following log messages are automatically filtered out and won't appear in output files:

• Messages starting with "Starting new instance"
• Messages starting with "Default STARTUP"
• Messages starting with "Ready condition"
• Messages starting with "audit_log"

Validation #

The parser will:

• ✅ Process valid log entries
• ⚠️ Skip entries with missing required structure (with error messages)
• ❌ Stop processing if the root is not a JSON array

🛠️ Usage #

Parser #

The main parser processes Google Cloud log files and generates multiple output formats.

Basic Syntax

dart run bin/parser.dart [OPTIONS] [INPUT_FILE]

Arguments

Examples

Basic Usage:

# Process default input.json, create output/ directory
dart run bin/parser.dart

# Process specific file
dart run bin/parser.dart my_logs.json

# Using positional argument (legacy style)
dart run bin/parser.dart logs/application.json

Advanced Usage:

# Custom input and output paths
dart run bin/parser.dart --input logs/production.json --output analysis/results

# Using short options
dart run bin/parser.dart -i cloud_logs.json -o processed_logs

# Show help
dart run bin/parser.dart --help

With Compiled Binary:

Will be added soon

Output Files

After processing, the following files are created in the output directory:

• output.txt - Formatted text logs (human readable)
• output.json - Structured JSON data
• json_payloads.json - Parsed JSON payloads
• errors.txt - Processing errors with details
• run-log.txt - Execution log and statistics

Highlighter #

Basic Syntax

dart run bin/highlighter.dart [FILE_PATH]

Arguments

Examples

Basic Usage:

# Highlight default output file
dart run bin/highlighter.dart

# Highlight specific file
dart run bin/highlighter.dart output/output.txt

# Highlight custom log file
dart run bin/highlighter.dart my_formatted_logs.txt

Advanced Usage:

# Highlight file from different directory
dart run bin/highlighter.dart ../other_project/output.txt

# Show help
dart run bin/highlighter.dart --help

With Compiled Binary:

Will be added soon

Color Scheme The highlighter uses the following color scheme:

• Blue [INFO] - Severity levels
• Green (05.10.25 12:00:00:000) - Timestamps
• Red <Mozilla/5.0...> - User agents
• Default - Log message content

Integration with Other Tools #

# Pipe to grep for filtering
dart run bin/highlighter.dart output/output.txt | grep "WARNING"

# Count occurrences of specific messages
dart run bin/highlighter.dart output/output.txt | grep -c "user"

📊 Output #

The parser generates multiple output files organized in the specified output directory. Each file serves a different purpose and format for various use cases.

Output Directory Structure #

output/
├── output.txt           # Formatted text logs
├── output.json          # Structured JSON data
├── json_payloads.json   # Parsed JSON payloads
├── errors.txt           # Processing errors
└── run-log.txt          # Execution log

File Details #

output.txt - Formatted Text Logs

Human-readable formatted logs with consistent structure. Format:

(TIMESTAMP) [SEVERITY] <USER_AGENT> LOG_MESSAGE

Example:

(05.10.23 12:00:00:123) [INFO] <Mozilla/5.0...> User authentication successful
(05.10.23 12:01:30:456) [ERROR] <curl/7.68.0> Database connection timeout
(05.10.23 12:02:15:789) [WARNING] <N/A> High memory usage: 85%

Features:

• Timestamps formatted as DD.MM.YY HH:MM:SS:mmm
• User agents shortened to 12 characters with ellipsis
• Consistent spacing and alignment

Ready for terminal display or logging systems

output.json - Structured JSON Data

Complete structured data preserving all original fields in a normalized format.

Format:

[
  {
    "Severity": "INFO",
    "Agent": "APIs-Google; (+https://developers.google.cn/webmasters/APIs-Google.html)",
    "Time": "05.10.23 12:00:00:123",
    "log": "User authentication successful"
  },
  {
    "Severity": "ERROR",
    "Time": "05.10.23 12:01:30:456",
    "log": "Database connection timeout"
  }
]

Fields:

• severity - Log level (INFO, ERROR, WARNING, etc.)
• agent - Full user agent string (not shortened)
• time - Formatted timestamp
• log - Log message content

json_payloads.json - Parsed JSON Payloads

Specialized output for logs that contain JSON payloads, with the JSON properly parsed and formatted.

Format:

[
  {
    "Severity": "ERROR",
    "Agent": "curl/7.68.0",
    "Time": "05.10.23 12:01:30:456",
    "log": {
      "message": "Database connection timeout",
      "errorCode": "DB_CONN_001",
      "details": {
        "host": "db-server-1",
        "port": 5432
      }
    }
  }
]

Features:

• JSON payloads are parsed into proper objects
• Maintains original JSON structure
• Ideal for data analysis and processing
• Only includes entries with JSON payloads

errors.txt - Processing Errors

Detailed error information for log entries that couldn't be processed.

Format:

Ошибка обработки элемента с индексом 0:
Тип ошибки: FormatException
Сообщение: Invalid JSON format
Элемент: { malformed json }
Stack trace: ...
------------------------

Contains:

• Error index (position in input file)
• Error type and message
• Problematic log entry
• Stack trace for debugging
• Separators between different errors

run-log.txt - Execution Log

Comprehensive log of the parsing execution with statistics and timing.

Format:

Google Cloud Log Parser
=======================
Входной файл: input.json
Выходная директория: output

Начата обработка 150 записей...
Ошибка в элементе 42: FormatException: Invalid timestamp
✓ Текстовый результат сохранен в output/output.txt
✓ JSON результат сохранен в output/output.json
✓ Логи с JSON-подобным payload сохранены в output/json_payloads.json
Найдено 23 логов с JSON-подобным payload.
⚠ Найдено 1 ошибок. Подробности в output/errors.txt

Обработка завершена!
✓ Успешно обработано: 149 из 150 элементов
✓ Отфильтровано: 5 элементов
⚠ С ошибками: 1 элементов

Includes:

• Start and end timestamps
• Input/output file paths
• Processing statistics
• Success/failure messages
• Error summaries

🏗️ Building from Source #

Will be added soon

👨‍💻 Author #

Oleg Uvarov

• Email: uv.ol.al@gmail.com
• GitHub: @LazyLazyMeat

📜 License #

This project is licensed under the GNU General Public License v3.0.
See the LICENSE file for the full text.

What this means: #

• ✅ You can use, modify, and distribute this software
• ✅ You can use this in personal projects
• 📝 If you modify and distribute this software, you must:
  • Make the source code available under GPL v3
  • Include the original copyright notice
  • State the changes you made

🚧 TO-DO #

• Add scripts for copying binaries to system bin directory or adding to PATH for Linux
• Develop equivalent solution for Windows
• Upload compiled binaries to GitHub releases
• Consider adding macOS support
• Translate all strings in source code to English
• Consider localization implementation
• Create Russian-language README
• Move configuration to separate config file
• Rename execution log to run.log (possibly move to logs folder and use timestamps for naming)